Terminally Incoherent

There are few things Vim and Emacs users have in common. One of these things is the fact they can talk together about programatically extending their work environment. You can’t really expect to talk to Eclipse of Visual studio about such things, because only a small fraction of them have ever even attempted to write a plugin for their IDE. Many probably installed plugins, but scant few even entertain the notion of writing one? Why? Because it is usually a pain in the ass.

Vim and Emacs users are different – take one at random from each camp, put them in a room and sooner or later you will find them talking about that plugin/extension they wrote for their editor. Why is it that majority of people who use these two tools always eventually end up writing custom plugins for their editor, whereas most other programers never do? Perhaps Vim and Emacs simply attract certain kind of minds to them – inquisitive tinkerers and hackers who simply like to tweak and customize things. Or perhaps it is that both these editors are just so easy to tweak and modify that there is simply no barrier to entry. If you have written your .vimrc from scratch (as every Vim user should – don’t let anyone tell you otherwise) then you already know how to write plugins. In fact, you have probably already written a few without realizing it – all you would need to do now would be to dis-entangle them from your config file and generalize them a wee bit so they can be installed as a self contained module. Same goes for Emacs.

Of course, whenever Emacs user and Vim user start talking about plugin developments, the former is bound by law to smugly condescend about the quality of the respective domain specific languages employed by each editor. Emacs uses Elisp which is a Lisp flavor and therefore superior.

You see, the ancient legend says that before the universe was compiled, the creator deity pondered for a spell, and then on the first day said Let there be parentheses! and he saw they were good. Then on the second day he said Let code be data and data be code and there was much rejoicing. Then on the third day he hacked up a meta-programming macro, and fucked off for the rest of the week as the universe generated itself. Then, millions of years later man invented haskell and was like “What now god?” and god was like “Shiiiiieeet, son!” because in his omnipotence he saw it was also pretty fucking awesome, though mostly in an incomparable to lisp in an apples to barnacles kind of way.

The point is, Emacs users have a pretty cool language they can play around with. Us Vim users, on the other hand, have VimL which is… Neither pretty nor elegant. It has no OOP support, weird scoping, too many overloaded syntax level operators that behave differently based on context, no JIT, and it allows (or even encourages) you to write terrible, unreadable spaghetti code that is worse in terms of readability than Perl. You could probably argue about Elisp’s aesthetic flavor with all the parentheses and what not, but at the end of the day it is rather elegant and consistent environment. VimL is quite the opposite: it is quirky, inconsistent mess of a language that grew along with the editor developing strange warts and tentacles along the way. It is much like a giraffe developed by a committee: while it is supposed to be a giraffe, it is actually an aardvark wearing a pineapple fruit hat and a monocle. It is neither glamorous, nor particularly fun to work with – but it will get the job done if you clench your teeth and try to resist the overwhelming urge to slap it in the face for being so goofy looking.

Fortunately for us, we have options. Modern versions of Vim tend to come pre-baked with support for alternate languages such as Python or Ruby for example. So if we wish, we can get away from VimL and write code that looks and reads nicely. Of course such a thing comes at a trade-off. The general rule of plugin development is that the further you go from the native language the editor “speaks” the worse off you are. In case of Emacs, this doesn’t matter because it was largely written in Elisp (except the few small parts that were coded in C). Vim is mostly implemented in C with VimL already being second class DSL so putting another layer of abstraction between us and the editor can become problematic. So the rule of thumb is that for simple stuff you should stick to VimL and fall back on external language only when you have to do some heavy and/or convoluted processing. VimL is closely tied to Vim but not as expressive as it could be when it comes to representing complex program logic.

Python, on the other hand is. While, perhaps not as elegant as Lisp, Python is nice, strong, opinionated and beautiful language. Opinionated, because among other things, it doesn’t give a shit about your preconceived notions regarding the use of white space. Beautiful, because it is the kind of language that will force you to write code that doesn’t look like shit or die trying. It is the language you need, but perhaps not one you deserve.

Also, because it is older and more mature than Ruby and therefore you have a better chance of it being compiled in your copy of Vim. Mature only in software terms of course – after all making Monty Python references in your code is the accepted development pattern and recommended best practice after all.

How do you know if your Vim can handle Python code? Well, it is easy:

• If you are outside of Vim you just run vim –version and look for +python which indicates you’ve got it or -python which indicates that whomever compiled your vim package is a major douchenozzle.

• If you are inside of Vim you can just issue the :py 1 command. If nothing happens, you are good. If you get an error message complaining that py is not an editor command, then… Douchenozzle.

Once you have confirmed that your vim will consume python code, you go about writing a plugin as you would with a VimL one. First, you create a file: say helloworld.vim or something like that. Then, at the top of that file you place some contingency management code:

if !has('python')
	finish
endif

This checks if the user’s editor has the Python support baked in, and silently quits if it does not. This is a nice thing to do, since it prevents someone’s editor from blowing up. Chances are the person who will be installing your plugin doe not read your README file and thus does not realize he needs python support, and therefore his Vim will blow up with a barrage of error messages next time he launches it.

Of course we could try being more explicit, and display an error message along the lines of “Y U has no +python!??” but I personally prefer the silent failure. Don’t get me wrong, explicit is almost always good, but I hate when plugins spam me with error messages on start-up. I keep my .vim/ directory under source control and I clone it onto every machine I work on. Sometimes I don’t have the luxury of using the latest and greatest Vim with all the fancy options. I don’t like to build specialized configs for those special cases – I’d rather use my usual setup, and just have the fancier stuff fail silently and simply be unavailable, rather than vomit stream of warning messages at me every time I do something. But that’s just me.

Next we want to expose some interface to the user in VimL so they can, for example bind it to a command or a key. That part has to be done in VimL. Think of it as bridging the divide between the native language and the external one. Usually, the best way to expose some functionality to the user in an unobtrusive way is to create a publicly accessible function:

function! HeloWorld()
 
endfunc

In general, you can access the Python environment in Vim by issuing the :python command (short version :py). You could technically write an entire plugin by simply prefacing every line with that command, but that would be extremely ugly and kinda crazy. Instead we can do something like this:

function! HeloWorld()
 
python <<< EOF
 
print "Hello World!"
 
EOF
endfunc

Now you can do :source % to load it up, and call HelloWorld() to test it. If all went well, you should see the text Hello World in the status line.

Granted, this is still kinda ugly. Mixing VimL and Python in a single file is definitely non-kosher, and it completely defeats the purpose of this entire exercise: to write plugins in a more elegant language, and make them more readable. Fortunately, there is another command we can use to load up an external python file, and execute it in Vim context: :pyfile (or :pyf). So our ugly function can be reduced to something like:

function! HeloWorld()
    pyfile helloworld.py
endfunc

“That is all well and good,” I can hear you say “but what does this do for us?” To actually do anything interesting as a plugin, your Python code needs access to internal Vim stuff – like the buffer list, or the contents of said buffers. How do we get that?

Well, we get it via the magical vim module. It is magical because it isn’t really a module that you can go and download or install in the usual way. It doesn’t exist on the outside, and can only be imported when the script is called from within Vim. This is both good and bad. It is good, because it makes it completely plug-and-play. If you have Vim with +python there is nothing else you need to do: it just works. On the other hand, it is bad because it requires Vim to work. This means you can’t run your code outside the context of your editor. For example, you may not be able use your favorite Python unit-testing suite to run test on your plugin, unless you mock up the Vim module functionality first. Which in retrospect is what you probably ought to do anyway so I guess it works out.

The vim module exposes bunch of useful objects such as vim.buffers or vim.current.buffer that you can use to manipulate your buffers, files and settings. The current buffer for example is actually a zero indexed array, each element representing a line as a string. Any changes to the elements in said array are immediately reflected on the screen. I will let that sit with you, so you can just imagine the possibilities for fun and abuse here.

If you want to know all the available objects and their details, I suggest you RTFM. Just as an exercise however, lets write a quick and dirty python plugin together.

I work with Markdown files a lot, so I always run them through something like Pandoc to check how will they look in HTML format. Let’s say I wanted to create a plugin that takes current buffer and dumps it into HTML format without the need for Pandoc or any other converter to be installed on the system. Because I’m lazy and I don’t feel like writing actual Markdown converter myself I will just use this Markdown package.

So, first let’s create a VimL plugin file that will load our Python code and call it, say markdown2html.vim:

if !has('python')
	finish
endif
 
function! Markdown2HTML()
	pyfile markdown2html.py
endfunc
 
command! MKD2HTML call Markdown2HTML()

The last line actually binds my function call to a Vim command providing an easy way for the user to run it. Next let’s write our python code:

try:
    import markdown
    import vim
    import string
 
    html = string.join(vim.current.buffer, "\n")
    html = markdown.markdown(html, output_format='html5')
 
    filename = vim.current.buffer.name+".html"
 
    f = open(filename, 'w')
    f.write(html)
    f.close()
 
    print "HTML output written to "+filename
 
except ImportError, e:
    print "Markdown package not installed, please run: pip install markdown"

Bam! Instant plugin. How awesome is that?

Of course the downside of using Python to write your plugins is that the users will need to have their editor compiled with +python in order to use them. Also, every time you use a third party Python package (like I just did above) your users will need to actually install that package on the system. So if you actually try to implement the above, don’t forget to pip install markdown at some point beforehand.

Still, the ability to bring the power of Python and all of it’s packages and modules into your Vim plugins is not something one should scoff at. If you would like to see a real life example of an excellent and popular plugin written in Python, go check out Gundo. It’s actually a plugin I use all the time as I work.

Lets talk about Vim tabs today, because they seem to be a source of perpetual confusion for new users. Most conventional text editors use tabbed interfaces the same way as web browsers do. You take a file, and load it into a tab and that’s all there is to it. There is a direct, unbreakable, strict, one-to-one relationship between a tab and an edited file and it cannot be broken at run-time. You can’t for example load another file into an existing tab without either overwriting its contents. This is sort of the behavior we came to expect from tabbed interfaces.

Vim novices usually come with these preconceived notion of how the tab metaphor ought to work in a text editing context, and become confused when Vim refuses to play by these rules. To put it simply, tabs in Vim are different. They won’t work the way you expect them to, because they were not designed for that. The one file per tab convention is not the Vim way.

Let me preempt the obligatory self-professed “usability expert” snark comments about vim breaking established UI conventions: please kindly go fuck a hedgehog. Bill Joy was establishing his own conventions when he wrote vi while doing your mom long before she spawned you and then became so fat she developed her own event horizon. Complaining that Vim doesn’t implement tabs the same exact way Sublime Text does makes very little sense when you consider the huge architectural differences between these two editors.

The original vi editor, and earlier versions of Vim did not really support tabbed interface. They did however support editing multiple files with a single instance of the program using buffers. If you have ever progamatically manipulated text in some way, you should be familiar with the concept of a buffer. To put it plainly a buffer is the copy of your document that is held in memory as you manipulate it. When you open a file, you load it into a memory buffer, and that’s where your changes are applied as you work on it. When you start a new document, your text is held in a memory buffer until you decide to write it to disk. Vim maintains a list of buffers that it has currently open, and you can view it by issuing the :buffers command (or alternatively the shorter :ls command alias).

You can typically only edit a single buffer at a time, so the current (or active) buffer is the one displayed on your screen, while all the other buffers are hidden away. When you open a new file, a new buffer is created and immediately displayed. The buffer that was being shown before it is not gone, but simply hidden away. Buffers are “like” tabs in modern editors, only they don’t have a graphical UI representation. They work the same though – you activate one of them, and it’s contents become visible while all the other ones are hidden away.

Vim Buffers

You can load any open buffer into your view-port by issuing the :b command followed by a number or partial file name which can be tab completed. The term “view-port” is a key concept here because it helps you understand what is really going on. It is best to think of the display area as a virtual port-hole through which you cane interact with your buffers one at a time.

When you create a split (for example using the :vsplit command) you effectively create two (or more) independent view-ports. Each one has access to Vim’s internal buffer list. So now you can have multiple buffers on the screen at the same time. Or you can load the same buffer in two splits, and watch changes you make on one side be automatically reflected on the other if that’s what floats your boat.

In essence, buffers do everything the tabs in say Sublime Text or whichever other Notepad replacement editor you happen to fancy. They just don’t have the clickable UI warts you might be tempted to fondle with your mouse. Lets face it – using a mouse to operate Vim is kinda like trying to eat tomato soup with a crossbow: it can probably be done, but it doesn’t seem at all practical. In fact, Vim’s buffer list is more flexible than most tabbed interfaces. For example, have you ever tried recursively loading your entire project tree into your editor? Let’s say you have 600 odd source files: what happens if you try to load all of them into your editor of choice? Firstly, chances are your app will crash and burn trying to prerender all these files or hitting some internal memory limits. It will also likely slow down to a crawl trying to create and display tabs in rapid succession (likely refreshing the screen for each one in the sequence). Assuming none of that will happen, your tab bar will likely become completely unusable: tabs will likely be about 1 pixel wide, or will force you to scroll around to find what you need.

Vim on the other hand will slurp up the entire project into buffers without even breaking a sweat because it doesn’t really have to render them unless you actually bring them into focus. Navigating a huge buffer list is actually easier than messing around with hundreds of tabs because you can use tab completion to match partial file names and get to where you need to go very quickly. Vim will also allow you to do bulk-edits using the :buffdo command which lets you run arbitrary vim commands (be it normal mode actions, build-in commands or function calls) on all the open buffers.

So the question is: could we simply force Vim to treat each buffer as a tab for the sake of following the UI convention followed by most of the relatively immature and untested (as compared to vim) editors on the market? The answer is no, we couldn’t. Why? Not because snobbery or entrenched traditionalism. It wouldn’t work because it would break the split screen functionality which is an established feature that predates the tabbed interface by a few decades.

Right now Vim lets you keep splitting your screen into independent view-ports, each of which can operate on a different buffer. How do you reconcile that with tabbed interface? Let’s say you have 7 files open, and you load 4 of them into 2×2 split (split horizontally, then split each half again vertically): what happens to the tabs that were loaded into splits of another tab? Do they disappear from the tab bar? What if you want to swap the contents of the splits? Do you re-shuffle the tab bar every time you split? If no, how do you deal with the tabs that are loaded in other tabs? If yes, how do you prevent tabs from moving around or getting lost? It seems like a very, very difficult problem to solve without altering existing functionality, with little to no benefit for existing users.

Tabs as implemented by Sublime Text and others simply make no sense in Vim world. Which brings us back to the original point of this post: what the fuck does Vim use the tabs for then? I can answer that with an example:

Imagine you are working on some code and you somehow ended up with a 2×2 split as above with 4 buffers on the screen at the same time. Suddenly you realize you need to open another file to tweak a thing or two related to your current task. You don’t want to split your screens any further, but at the same time you don’t want to blow that layout away since you are not done with it. To make matters worse said file is big, complex and ugly, so it ought to be viewed in full screen. In the olden days you would probably suspend (or minimize) your current session, and open another Vim instance to do so. But with tabs you don’t have to. You just open a new tab, which gives you a fresh view-port. One with access to the same buffer list, the same registers and recorded macros. One which you can split into multiple panes if you wish.

Vim Tabs

Vim tabs are essentially work-spaces or view-ports. They are more or less the virtual desktops for your text editor. They are not meant to be bound to buffers, but instead provide you with additional work surface where you can “spread out” the buffers you have opened.

You can of course try to force Vim to follow the “one-buffer-per-tab” convention, but it will always be an uphill battle. I’ve seen people write elaborate VimL scripts to accomplish this. I have seen countless plugins that claim they offer this functionality. But most of them break into thousand pieces as soon as you try to do something fancy with splits, hidden buffers or the :buffdo and :argdo commands. And if you are never going to try anything fancy, complex or weird then what’s the point of using Vim anyway?