PyLookup Django: Django Documentation Lookup in Emacs

Edit: I’ve submitted a pull request that will render step 3 obsolete. I will update this article when it’s merged. These instructions can also be used for NumPy.

Edit edit: The pull request has been merged, rendering step 3 obsolete. These instructions are now effectively the same as those in the pylookup README.

(This also works for NumPy, SciPy, etc.)

I’ve been using Taesoo Kim’s pylookup for a while now to lookup Python documentation offline in Emacs. It’s a wonderful tool, especially combined with Emacs’ w3m browser, allowing Python programmers to lookup documentation without ever leaving the Emacs frame.

pylookup django

Here’s how to lookup offline Django documentation with pylookup:

I have pylookup installed through el-get, so it’s in ~/.emacs.d/el-get/pylookup. These instructions are modeled on Taesoo’s “Create Database” instructions in the Github README, but with a slight modification.

  1. Download and unzip the Django HTML documentation from https://www.djangoproject.com/m/docs/django-docs-dev-en.zip
  2. Move the resulting folder to the pylookup folder
  3. pylookup’s database creation expects a genindex-all.html file in the documentation folder. Let’s create that file by copying Django’s genindex.html:
    cp django-docs-dev-en/genindex.html django-docs-dev-en/genindex-all.html

  4. Add the Django documentation to pylookup’s documentation index (I already have the python standard library documentation indexed as per Taesoo’s instructions):
    ./pylookup.py -a -u django-docs-dev-en

    (Note the -a option that appends the newly indexed Django docs to the existing standard library docs rather than replacing them)

I can add more complete documentation for setting up pylookup if necessary, but between the Github README and this extra, you should be able to work it out.

Happy coding!

Advertisement
Posted in Emacs, Python | Tagged , , , | Leave a comment

Emacs Send to Email Command

I’ve been using Emacs for nearly all my editing for a couple years now, and I’m getting more hungry for highly personalized customizations. I often need to email files I’m working on, so I wrote up this little command (Throw it in your .emacs):

(defun mail-current-buffer ()
  "Send the current buffer to email (for Mac)"
  (interactive)
  (shell-command (format "open -a Sparrow %s" (buffer-file-name))))
(define-key global-map "\C-cm" 'mail-current-buffer)

This opens a new message with the file in the current buffer attached. If you use another mail app, replace Sparrow with its name. The last line binds the function to C-c C-m. Really, this will open the current buffer in any application you choose to put in that command besides Sparrow, so perhaps you have some other handy ideas?

I guess the next step is just doing my email in Emacs.

Posted in Emacs | Tagged | Leave a comment

BlogTuner: Download + Add + Tag MP3s

Discovering music through blogs is one of my favorite internet pastimes. I’ve built a tool for Mac OSX that makes it easier to bring music from blogs into your iTunes library.

BlogTuner is a super quick and easy way to add songs found online to iTunes, as well as to keep track of where those songs came from. Suggested workflow is as follows:

0. Install by double clicking the .zip file to unzip. Drag the application to your “Applications” folder, and then to your Dock so that the icon is easily accessible.

1. Go out, browse for music, and find a song you love.

2. Click and drag the link to that song into the BlogTuner icon in your Dock:
Click and Drag the Link
(Example from The Music Ninja)
Drop on the BlogTuner Icon

3. The song is automatically downloaded, added to iTunes, and the site you retrieved it from is added to the “Comments” metadata. A Growl notification pops up when the import is complete.

4. If you’d like, you can use the metadata to create Smart Playlists for any blog. This is how the setup would look for The Music Ninja:
BlogTuner Smart Playlist Setup

Click here to download BlogTuner from my Dropbox.

(Need I say it? This software runs well to the best of my knowledge but I offer absolutely NO warranty.)

Is this useful? Have you found any bugs? Would you be willing to make a Dock icon for BlogTuner? Drop me a line in the comments below.

Posted in Music, Software | Leave a comment

Installing Emacs MMM-Mako Mode

For my job at Hulu this summer, I’ve been developing a RESTful web service and web based frontend using CherryPy and Mako Templates. Since Mako uses a mix of its own syntax, HTML and Python, syntax highlighting won’t work out of the box in Emacs. Enter MMM (Multiple Major Mode) Mode, which allows for multiple Emacs major modes within one buffer, and MMM-Mako Mode which provides support for Mako files using MMM. Getting these set up was somewhat problematic, so I’m going to document the process for anyone else trying to get these working.

(I assume python-mode is already working)

  1. Download and untar MMM Mode from http://sourceforge.net/projects/mmm-mode/files/. I’m using version 0.4.8.
  2. Make and install MMM Mode. You need to specify where your Emacs is installed. I use Gnu Emacs 23, which I have installed in the /Applications directory.
    EMACS='/Applications/Emacs.app/Contents/MacOS/Emacs'
    ./configure
    make
    sudo make install

    The elisp files were installed to /usr/local/share/emacs/site-lisp in my case. You may need to add this directory to the Emacs load path. You will also need to tell Emacs to load the module. I added this to my .emacs:

    (add-to-list 'load-path "/usr/local/share/emacs/site-lisp")
    (require 'mmm-auto)
    (setq mmm-global-mode 'maybe)

    Make sure it installed correctly by opening up Emacs and typing M-x mmm-mode to turn on MMM-Mode.

  3. Now you’ll want to download MMM-Mako from https://bitbucket.org/pjenvey/mmm-mako/downloads. Follow instructions as in the README, pointing Emacs to the mmm-mako.el file. I added to my .emacs:
    (add-to-list 'auto-mode-alist '("\\.mako\\'" . html-mode))
    (mmm-add-mode-ext-class 'html-mode "\\.mako\\'" 'mako)
  4. At this point, everything is configured for MMM-Mako, and opening a .makofile should enable MMM-Mode. If you open a file with both Python and HTML, however, you will probably see an error:
    mmm-format-string: Wrong type argument: stringp, (sgml-xml-mode "XHTML" "HTML")

    A bug report explains why this is happening in Emacs 23 and offers a fix. Find the mmm-utils.el file from your MMM-Mode installation and apply the patch from the bug report, or edit it by hand.

    sudo patch /usr/local/share/emacs/site-lisp/mmm-utils.el fix_mmm-format-string.diff

    You’ll want to byte-compile the modified file, so open up Emacs again and type M-x byte-compile-file and provide it with the location of your mmm-utils.el file. You may need to do this with root privileges depending on where you installed the MMM-Mode files.

At this point you should have a fully functioning MMM-Mako Mode.

All the additions to my .emacs file:

(add-to-list 'load-path "/usr/local/share/emacs/site-lisp")
(require 'mmm-auto)
(setq mmm-global-mode 'maybe)
(load "src/mmm-mako.el")
(add-to-list 'auto-mode-alist '("\\.mako\\'" . html-mode))
(mmm-add-mode-ext-class 'html-mode "\\.mako\\'" 'mako)
Posted in Emacs | Tagged , , , | Leave a comment

Why Turntable.fm Rocks the Decks

The 21st century’s response to the mixtape, Turntable.fm brings music to our virtual social lives. Didn’t Myspace already do this? Well, no. At least not nearly as well as Turntable is poised to.

So what’s so great about Turntable? In an age in which our social proof is displayed in a virtual world, Turntable.fm provides a truly social outlet for sharing musical interests. Though not required, most users actively engage in the scene of the (usually themed) “room” they are in. Hate the song that’s playing? Hit the “Lame” button and if others agree with you, the song will be skipped. Love it? “Awesome” will set your avatar’s head bobbing and give points to the DJ.

I had a blast just bouncing around the rooms meeting new people, but the true value of Turntable.fm will be leveraged in our existing social connections. This is currently implemented by requiring that one of your Facebook friends is using Turntable before you can make an account. Though surely helpful in avoiding the problems of scale that plague pivoting startups, this requirement encourages an organic growth in user base. When you bring up the room list, friends’ Facebook profile photos show up indicating their presence. “Fan” them and Turntable will email you whenever they step up to DJ. What better way to judge someone’s musical tastes than to see what they play to the public, and how it’s received? A recent night showed a “Google NYC” room ostensibly filled with Googlers DJing for each other in the office.

Turntable.fm would do well to exploit the social and game mechanics they have integrated to increase the quality of the musical programming. Currently the only benefit of points seems to be new avatars, but I could see the point system extending throughout the experience. Imagine when high scoring DJs score spots in busier rooms and those who receive bad scores are kicked out of the DJ booth. The current system provides little consistency as an empty seat goes to the quickest clicker.

There is a distinct lack of diversity in the musical tastes on offer. Of the current top 15 rooms by population, one third are “indie” themed and another third focus on electronic music. Some songs (RJD2’s excellent Ghostwriter, for instance) play over and over. The musical diversity will surely increase as Turntable’s population does, but it will be essential that the company helps expose that diversity through the interface. As the number of rooms explodes, Turntable’s developers must deliver an effective interface for browsing the choices. With only 30 rooms holding 10 people or more about a month after launch, finding a good place to hang out is easy. How will we navigate when this number is in the thousands?

I expect Turntable to be one of the most successful startups of 2011. Find me there DJing as PastaMasta.

Posted in Music, Startups | Tagged , , | 1 Comment

Exploring Caches and Spatial Locality with C

Could a simple change in the layout of a data structure in memory result in a 20% reduction in performance? Recently in my Digital Systems Organization and Design class, we have been discussing data caches in modern processors. As a toy example of the ability to explicitly prefetch data into the cache, Professor Milo Martin presented this C code which performs a depth-first traversal of a binary tree, adding the values at every node:
(Full source available at https://gist.github.com/946002 — Fork it! Modify it! Let me know!)

int tree_add(treenode* t){
  if(t == NULL) return 0;
  __builtin_prefetch(t->left);
  __builtin_prefetch(t->right);
  return t->value + tree_add(t->right) + tree_add(t->left);
}

Needing an exercise in C, and curious about the effects of explicit prefetching, I decided to test this out. First I generated a tree in a depth-first fashion to try the algorithm on:

treenode* buildtree(int height){                                                        
  if(h < 0) return NULL;                                                           
  treenode* t = malloc(sizeof(treenode));                                          
  if(t == NULL){                                                                   
    printf("Could not allocate memory");                                           
    exit(1);                                                                       
  }
  t->value = rand() % 10;                                                        
  --height;
  t->right = buildtree(height);
  t->left = buildtree(height);
  return t;
}

When I tested 10,000 runs of the tree_add algorithm on a tree of height 20 (2,097,151 nodes) with and without the explicit __builtin_prefetch function, I found nearly zero difference (372.257728 seconds without prefetching vs. 372.901536 seconds with prefetching). Professor Martin suggested this might be an artifact of the fact that I was generating the tree in depth-first fashion. So I generated a breadth-first tree:

treenode* newnode(){
  treenode* t = malloc(sizeof(treenode));
  if(t == NULL){
    printf("Could not allocate memory");
    exit(1);
  }
  t->value = rand() % 10;
  t->left = NULL;
  t->right = NULL;
  return t;
}

treenode* buildtreeBFS(int height){
  if(height < 0) return NULL;
   treenode* t = newnode();
   list* l = malloc(sizeof(list));
   if(l == NULL){
     printf("Could not allocate memory");
     exit(1);
   }
   addfront(l, t);
   int nodes = pow(2, height+1) - 2;
   printf("%d\n", nodes);
   while(nodes > 0){
    treenode* node = removeback(l);
    node->left = newnode();
    node->right = newnode();
    addfront(l, node->left);
    addfront(l, node->right);
    nodes -= 2;
  }

  while(removeback(l) != NULL); //clear out the list
  free(l);

  return t;
}

This is where things get surprising. Running the inorder traversal on the breadth-first generated tree takes significantly longer than on the depth-first tree! Again, the explicit prefetching seems to have little effect (446.019808 seconds without prefetching vs. 447.025984 seconds with prefetching). We take 20% longer to perform the tree_add function on the breadth-first tree than we did with the depth-first tree. Though these are functionally the same data structure, I hypothesize that their layout in memory, along with the effects of spatial locality on caching data, accounts for the performance difference.

Since modern computers use cache blocks that can hold many of our treenode structs, if we are accessing nodes that are lined up sequentially in memory, we will have far fewer cache misses as we traverse the tree. When we generated the tree in the same depth-first order as we later traversed it in, we exploited this locality by accessing sequentially allocated heap locations. Using the breadth-first tree, we jumped around through addresses in the heap, likely incurring a much higher cost for memory accesses.

I find it fascinating that in a realistic scenario we see such a large performance difference using data structures with so many apparent similarities. It really proves the usefulness of hardware classes even for someone who never intends to design computer hardware. Thank you Professor Martin!

Posted in C Programming, Hardware | Leave a comment

iPhone Tracking: Who Cares? You Should.

The tech community has sounded the alarm in the last couple days that our iPhones have been storing years worth of information on our whereabouts, down to an alarming level of detail. Surely this has been known by some for quite a while, but a beautifully presented application by Alasdair Allen and Pete Warden brought the issue to the public in the last week (A hacker friend of mine suggested that this and other non-technical presentations of known exploits — see Firesheep — only appear once the exploits are no longer economically viable).

I downloaded the application to try it out, and sure enough, without even touching my Unix shell, I had a precise mapping of my whereabouts over the last year. Certainly it isn’t a huge shock that in the 21st century our locations are being tracked, but we shouldn’t take these revelations too lightly. David Pogue, a New York Times blogger, ridicules those who suggest that we should be concerned by this news. But I think Mr. Pogue is missing some of the broader implications of the release.

Cell phone privacy has become a major issue in 21st century privacy law. We are far from having clear definitions of how information stored on our phones can and cannot be used by the government. Just this year, the California Supreme Court ruled warrantless searches of cell phones to be legal. This is where things get scary. In their decision, the court grants “carte blanche” for an officer to look through any data in a cell phone confiscated during an arrest, according to the dissenting opinion of Justice Kathryn Werdegar. We now know that this data can hold years worth of information about our whereabouts. If that is not a unreasonable search, I don’t know what is.

Given these circumstances, the public must be as informed as possible as to the information we may be carrying, and how it can be used against us. To opponents of privacy like Mr. Pogue I ask: Why shouldn’t we hold our devices to higher standards? I hope our courts can recognize the increasing importance of cell phone privacy, and support those rights as the Ohio Supreme Court did in 2009. Beyond that, Apple, RIM, Google: You owe us all an explanation as to why our devices store our location data indefinitely.

Posted in iPhone, Privacy | Leave a comment

The Go Command

Alexey Komissarouk requested a modification to the cl command that opens the argument in the default text editor if it’s a file. The go command will open the file in the editor defined by the EDITOR environment variable.
Place this in your .bashrc, .bash_profile or analogous file:

go()
{
if [ -f "$1" ]
then
    $EDITOR "$1"
else
    cd "$1" && ls
fi
}

If you have not defined the default editor, perhaps you’d like to try emacs, my chosen editor. Add this to the .bashrc or .bash_profile as well:

export EDITOR=emacs
Posted in Shell Scripting | 11 Comments

Combining cd and ls Into One Command

Andrew Braunstein and I were discussing how one might add a shell command to change a directory and list it using one command. I’m sure anyone who uses the shell is familiar with cd followed immediately by ls.

A search found this discussion on the Unix.com forums. The suggestion was to add the following shell function to the .bashrc or .bash_profile:

cl()
{
if [ -d "$1" ]; then
cd "$1"
ls
fi
}

This works reasonably well, but I had a couple of complaints:

  1. Because the function does nothing if the argument is not a directory, when we try to cl aDirectoryThatDoesNotExist, there is no indication of the error. The shell just prints a new prompt.
  2. Also due to the check that the argument is a directory, we cannot use the default functionality of cd with no argument that returns us to the home directory.

My proposed solution is this:

cl()
{
cd $1 && ls
}

Here we take advantage of the exit code produced by the cd command to ensure that the directory change was successful before we ls. If cd exits with an error, the boolean “and” is already false, and the ls is not executed. All we see is the error message, and we remain in the same directory. This behavior seems ideal. Additionally, calling cl with no arguments changes to, and lists the home directory as expected.

cl in action

cl in action

Posted in Shell Scripting | Tagged , , | 2 Comments

Automatic SOCKS Proxy for a Domestic IP Address

Update 2013-03-31: In Mac OS 10.8 (and maybe 10.7), the Airport name has been changed to Wi-Fi. The script should look like:

trap "{ networksetup -setsocksfirewallproxystate Wi-Fi off; }" INT
networksetup -setsocksfirewallproxy Wi-Fi localhost 3333
ssh -ND 3333 USERNAME@HOSTNAME

While enjoying a relaxing spring break in Jamaica, I was attempting to read through the monster that is Gödel, Escher, Bach: The Eternal Golden Braid. Hofstadter inspired me to check out the music of Bach, so I logged into my Rhapsody account. All good. But when I attempted to play the songs I received the message, “This content not available in your country.” That’s interesting, since I’m a US based subscriber to Rhapsody. The problem was that the request was coming from a Jamaican IP Address.

Of course, there is a way around this. SSH into a machine in the US and create a SOCKS proxy so that the web requests are coming from the US-based IP. But this was inconvenient to do every time I wanted to listen, so I made a shell script to automatically create the SSH connection, change my Mac’s system proxy to this, and undo all the changes when quit.

Running this script will require an SSH public key pair be created so that the remote machine can be connected to without a password. Edit the script to include the correct user and host names. This will only work on a Mac. It also assumes that you are getting internet through the Airport. If you are getting internet through ethernet change Airport on line 2 to Ethernet. Any suggestions on how to find which device is providing internet in the script to do this automatically?

trap "{ networksetup -setsocksfirewallproxystate Airport off; }" INT
networksetup -setsocksfirewallproxy Airport localhost 3333
ssh -ND 3333 USERNAME@HOSTNAME

This can be run directly from the command line, but I enclosed it in an Automator workflow so that I could launch it through LaunchBar as though it were an application. When the script is closed it should turn off the proxy and restore the previous IP.

I suspect this method should work for any site that restricts content by IP address. Netflix, Hulu and Youtube users will likely find this useful.

Posted in Shell Scripting | Tagged , , , , , , , , | 1 Comment