magpie

https://dl.dropboxusercontent.com/u/402325/dontdelete/magpie.jpg

magpie: [ma]rkdown, [g]it, [pie]thon

Git-backed Evernote replacement

Quickstart

Pre-requisites

(outside of PyPI):

  • Python 2.7
  • Pip/Easy Install
  • Git
  • libmagic (OS X specific issue; it’s already installed on most Linux distros)
  • cygwin (Windows only requirement; see issue #4)

Installing magpie

If you’re a Python person, and you are using virtual environments, just:

pip install magpie

If you don’t know what Python virtual environments are:

sudo pip install magpie

Setup Git Repo

git init

Locally:

mkdir -p /path/to/notes/repo/
cd /path/to/notes/repo/
git init

On Github

On Bitbucket

git config

Depending on what your environment is like, you may get a nasty error the first time you try to do something useful. This happened to me when I setup magpie to work with supervisor. In order to resolve this, I had to set the user.name and user.email fields in git config. The error message on the magpie page will tell you the syntax.

Launch magpie

In a Python virtualenv:

magpie

If you installed magpie with sudo:

sudo magpie

Connect

In browser, go to http://localhost:8080

Configure

On the main page, there should be a link to configure magpie. (Alternatively, http://localhost:8080/config)

Enter as much info here as you want. Username / Password are not required, but recommended. The only required field is repo. The path from “Init Git Repo” above should be entered here.

Home Dir Config

As of version 0.0.4, if you have a ”.magpie” directory in your home directory, you can store config files there, instead of the default path. You’ll need to make this directory after installing magpie and before running it. Alternatively, you can create the directory and move the config files there manually.

What is magpie?

I love Evernote, but I no longer trust my data to cloud providers. Magpie is an attempt to make a reasonably sufficient Evernote replacement where the users control their data.

Basically, magpie is just a web tool for managing text files in a git repo. In it, you can create notebooks (which are just folders); create, edit, and delete notes (which are just files). That’s pretty much it. However, when you make any of these changes, they are automatically committed to git.

Demo

A demo of magpie is available here

(If you make changes and they disappear, it’s because the demo resets itself every 15 minutes.)

What isn’t magpie?

  • Complete (see contributing)
  • Secure (magpie is only as safe as you make it. If your git repo is hosted on a public server, people will be able to read your notes. If you run it on an open network, people may be able to access your notes. Etc.)
  • Shiny (This is a side project written and maintained - so far - by a single person. It’s never going to be as good or as useable as Evernote.)

Features

  • Markdown & HTML notes are rendered on the page

  • Uses git as a backend

    • Easy backups (if you know git): clone once, then push/pull to backup notes
    • track history, etc, just like with git (using git, not via magpie’s web interface ... yet?)

  • Render “[ ]” and “[x]” as check boxes. Clicking them changes and saves the note.

  • Email yourself notes (see emailing notes)

  • Scrape PDFs to make them searchable in magpie (see pdf_scraper)

Image Attribution

Logo/favicon courtesy of Kieran Palmer, as licensed under CC BY-SA 2.0 Generic.

Contributing

magpie Mailing List

If you’re interested in contributing, consider signing up for magpie’s mailing list.

Thanks!

Thanks to:

  • erickr for fixing searching for notes w/unicode (#40) and helping with other unicode issues (#42).
  • matthewi for making the listen address more configurable (#37).
  • tony-o for beginning the auto-save feature (#20), fixing the .git search problem (#21), cleaning up the base template (#22), and adding ünicode support.
  • Erwyn for fixing the write/reload race condition (#15)
  • tomleo for fixing the home dir config bug (#10)
  • looper for adding the home directory config functionality (#5)

Bug Fixes / Minor Changes

magpie uses todo.md to track TODO items in its code. If you’re looking to fix bugs, magpie’s todo.md is a good place to start.

New Features

There are a few features that magpie could use to make it really great, and even better competition for Evernote.

Git Setup

In order to use magpie right now, you have to know enough about git to init the repo, and set the user name and email. It would be nice if magpie was smart enough to do this through the web application.

Note history

It should be possible to show the history of a note, and diff the changes between notes versions, something like how redmine handles it on file versions.

Web Scraper

Evernote has a Javascript bookmarklet that will scrape a webpage and add it as a note. If I was better with Javascript, I’d try to implement this myself.

OCR

Evernote’s OCR is really good. I looked into Python projects that would allow me to do OCR in magpie, and I didn’t find anything that seemed both feature complete and easy to use.

A good option might be to use tesseract, but that will require that as an external dependency, rather than some pure python-implementation.

Other document scrapers

It would be great if a user could upload docs of various types to magpie and have them scraped into plaintext. I’ve already added a rudimentary PDF scraper, and I started building functionality into magpie already that will handle viewing the plaintext version of a non-plaintext document, by looking at files with identical names, only the plaintext filename starts with a dot. For example:

file.pdf  <-- PDF
.file.pdf <-- scraped plaintext

In the above example, magpie will render the .file.pdf plaintext file when you click on file.pdf in the left-hand side notes menu.

OCR should be done on parts of the PDF which does not have text in it, such as image parts or scanned PDFs.

Utils Config Tool

The pdf scraper and the script for turning emails into notes are called magpie utils (and exist in the utils dir of the code base). Like the web application, the utils each have their own config file, all located in magpie/config. There is functionality in the application to edit its own config file, but not to edit the config files for any of the utils. It would be nice if that was added by someone.

Testing

Some rudimentary tests exist on travis-ci, but more tests are always better. Submitting pull requests will automatically trigger travis to run magpie’s tests against your pull request.

Documentation

Hopefully this documentation is good enough to get people using magpie, but documentation can always be more thorough.

Magpie Utils

In addition to the web application, magpie comes with utils, which allow the creation of notes outside of the web interface. These live in “magpie/utils” in the code base, and are installed as executables in the same location as magpie when you pip install magpie.

Emailing Notes

One of the most convenient features of Evernote is the ability to email notes to yourself. I’ve attempted to duplicate this functionality for magpie. This is a totally separate process from the magpie web server, but is also installed when you pip install magpie. In addition to magpie being installed as an executable, you should also get an executable called email_notes.py.

The email_notes.py script does not delete email, and it only operates on unread emails.

Configuration

email_notes.py is configured via a config file in magpie/config. Assuming you’re using virtualenv, and you’ve named the magpie virtualenv “magpie”, then the full path would be something like /home/you/.virtualenvs/magpie/lib/python-verison/site-packages/magpie/config/email_notes.cfg. In that file, you can set the following options:

  • imap_server: This is the address of your IMAP server (non-IMAP email isn’t currently supported)
  • username: The username you use to authenticate to the server (likely, but not necessarily your email address)
  • password: The password you use to authenticate to the server
  • folder: You’ll probably want to have notes filtered into a special folder (gmail calls these labels) so they don’t clutter up your inbox. specify that here.
  • repo: This is the git repo where your notes live (almost certainly should be the same path you specified when configuring the web application)
  • use_ssl: Do you want to connect securely to your IMAP server (recommended)
  • default_notebook: You can specify which notebook an email will be saved to in the subject line of the email (see “Subject Syntax” below). If you don’t specify that in the subject, your notes will default to this notebook.

With the exception of use_ssl, all fields should be wrapped in single or double quotes. use_ssl should either be True (capital T) or False (capital F).

Subject Syntax

The body of the email you sent will be the contents of your note in magpie. The title of your note, which notebook it should be stored in, and more will be controlled via the subject line of the email. If you have not set the folder field in the config file, or have set it to anything other than “inbox” (case-in**sensitive), then your notes ***MUST* start with “*Note* ”. If you have specified a folder (other than “inbox”, then do not include “*Note*”. If you want to specifiy a notebook other than the default from the config file, use “@notebook name” towards the end of the subject line. If you want an existing note to be appended to, rather than overwritten, add ” +” as the very last thing in the subject line. Everything from the beginning of the subject line (excluding *Note* if it’s required), and before either @ if you’ve specified the notebook name or + if you haven’t will be considered the title of the note.

Examples
This is a plain note title

This note will be called “This is a plain note title” and will be stored in the default_notebook specified in the config file. If a note with that name already exists, its contents will be overwritten by the contents of the email.

*Note* This is also plain

If you have not specified a folder in the config file, then “*Note* ” is required, and this note will be called “This is also plain” and stored in the default notebook. If you *have* specified a folder in the config file, this note will be titled “*Note* This is also plain.”

This is an appended note +

This note will be called “This is an appended note” It will be placed in the default_notebook and if a note with that title exists, the contents of the email will be appended to the end of the existing text.

This will be stored in a different notebook @another notebook

This note will be stored in the notebook “another notebook”

Appended note in @another notebook +

This note will be stored in “another notebook” it will be titled “Appended note in” and will be appended to the end of the existing note, if one already exists with that name.

Filtering

You’ll probably want to create an email filter for these notes so they don’t clog your inbox, and so you can specify that in the config file so you don’t have to add “*Note* ” to the beginning of every note you send yourself. Many email servers allow you to alter your email address in order to setup special filters. For example, in gmail you can add “+anyTextYouWant” to the end of your username and you can add or remove periods to your heart’s content. So if your email address was “magpie@gmail.com” you could set up a filter to send notes sent from yourself to “m.agpie@gmail.com” or “magpie+note@gmail.com” to be moved to your “notes” folder/label, and then configure “notes” as the “folder” value in the config file.

Scheduling

email_notes.py runs once and then exits; it does not run as a daemon. You probably want to configure cron or some other scheduler to run it at some interval so you don’t have to remember to run it manually.

Scraping PDFs

A nice feature of Evernote is that it scrapes PDFs you upload in order to make them searchable. Magpie has that ability as well, if you use the magpie util pdf_scraper.py.

Configuration

pdf_scraper.py has a config file, like magpie and email_notes.py, and it lives in the same place as the other config files. It only has two options to configure:

  • repo: This is (presumably) the same value you specified in the magpie config.
  • default_notebook: If you run pdf_scraper.py on a PDF outside of your magpie repo, the plaintext output will be written to this notebook inside the magpie repo, rather than wherever the PDF lives
Scraping

Using pdf_scraper.py should be fairly straightforward. After configuration, simply run pdf_scraper.py /path/to/pdf1.pdf /another/path/pdf2.pdf etc.pdf. The scraper will run against each of the files passed as command line arguments. If the PDFs were inside the configured magpie repo, then the output files will be stored in the same location as the original PDF, and then name will be identical, except the filename will have a leading period. For example, if the PDF was /path/to/file.pdf then the plaintext output from the scraper would be stored in /path/to/.file.pdf. If the PDF is not already in the configured repo, then the file will still start with ”.”, and will be saved to the default_notebook from the config file.

Ugly Output

The odds are pretty good that the output of pdf_scraper.py will be ugly. That’s the best I could do for now. The purpose of this functionality is primarily to allow for searching the PDFs, not necessarily to read their contents in the web application. However, once the plaintext version exists in magpie, you can edit it in the web application just like any other note, and it will not impact the PDF. This means if you want to make the note readable and clean it up, you can.

Release Notes

magpie 0.1.0

Thanks!

First and foremost, thanks to everyone who helped build this release. You can see the list of contributors here.

New Features
Star / Unstar Notes

Starring a note will put it at the top of the list of notes in the right side menu. Starred notes are sorted alphabetically. Notes that haven’t been starred are also sorted alphabetically, but appear underneath the sorted list of starred notes.

Starred notes are stored in cookies, so if you use more than one device to access magpie, and you don’t sync your browser data, starred notes won’t move with you.

Choose Listening Address

In magpie’s config, you can now specify what address magpie listens on, rather than just localhost only, or all addresses. To listen on all address now, set the IP to 0.0.0.0.

Unicode Notebook / Note Names

You can now create notes and notebooks with unicode in their names. Unfortunately, there appears to be a bug with colons in filenames.

Config Locations

In addition to the location inside the magpie install, and ~/.magpie, if you have a config file in the directory you launch magpie from, magpie will read from that config file, instead of the other two. The order of preference is:

  • . (current working directory when you launched magpie)
  • ~/.magpie
  • /<path to magpie install>/config/
Bug Fixes
Can’t launch magpie / Sphinx error

Sphinx was causing problems for some people. Since it is only a requirement for building the documentation, it was removed from requirements.txt, so it should no longer be an issue (magpie will no longer try to install it when you pip install magpie).

Potential / Known Bugs
  • As noted above, magpie doesn’t seem to like colons in notebook names or note names.
  • Neither of the utils has been updated with unicode support. They may or may not continue to work. Please open an issue if you are using either of them, and 0.1.0 breaks something for you.

Latest

magpie 0.1.0

Thanks!

First and foremost, thanks to everyone who helped build this release. You can see the list of contributors here.

New Features

Star / Unstar Notes

Starring a note will put it at the top of the list of notes in the right side menu. Starred notes are sorted alphabetically. Notes that haven’t been starred are also sorted alphabetically, but appear underneath the sorted list of starred notes.

Starred notes are stored in cookies, so if you use more than one device to access magpie, and you don’t sync your browser data, starred notes won’t move with you.

Choose Listening Address

In magpie’s config, you can now specify what address magpie listens on, rather than just localhost only, or all addresses. To listen on all address now, set the IP to 0.0.0.0.

Unicode Notebook / Note Names

You can now create notes and notebooks with unicode in their names. Unfortunately, there appears to be a bug with colons in filenames.

Config Locations

In addition to the location inside the magpie install, and ~/.magpie, if you have a config file in the directory you launch magpie from, magpie will read from that config file, instead of the other two. The order of preference is:

  • . (current working directory when you launched magpie)
  • ~/.magpie
  • /<path to magpie install>/config/

Bug Fixes

Can’t launch magpie / Sphinx error

Sphinx was causing problems for some people. Since it is only a requirement for building the documentation, it was removed from requirements.txt, so it should no longer be an issue (magpie will no longer try to install it when you pip install magpie).

Potential / Known Bugs
  • As noted above, magpie doesn’t seem to like colons in notebook names or note names.
  • Neither of the utils has been updated with unicode support. They may or may not continue to work. Please open an issue if you are using either of them, and 0.1.0 breaks something for you.