Terminally Incoherent

I like to use this example to totally freak out Java people. Cool dynamic languages such as Python or Ruby allow you to modify the definition of any class on the fly. Javascript let’s you do that too, but then again Javascript does not really have classes right now. It’s a prototype based language - at least until ECMAScript 4 descends from heavens in a beam of heavenly light and will bring forth much awesomeness. Here is how you do it in Python:

Observe:

>>> class Person:
...     def __init__(self, fname, lname):
...             self.fname=fname
...             self.lname=lname
...
>>> john = Person("John", "Smith")
>>>
>>> Person.whatisyourname = lambda p: "My name is " \
...     + p.fname + " " + p.lname
>>>
>>> john.whatisyourname()
'My name is John Smith'

Every new instance of Person will now have a whatisyourname() method. C and Java people are probably sneering right now, at how insecure this is, how it breaks encapsulation and etc. I used to be like that too, but I reformed. What I see here is raw power.

I’m putting this here because I was trying to do closures in Python but found out you can’t really do them. Python has lambda functions instead which are really a lisp concept. The only difference between a Python lambda function and a closure is that lambda must evaluate to something. In other words, it’s body must be an expression rather than a statement. Which makes perfect sense in Lisp because it has no statements. In Lisp everything is an expression - and frankly, that is not a bad idea. Lets you chain stuff pretty nicely.

If you need to use Python statements however, you can’t use lambda. You just have to use regular assignment:

>>> def foo(self):
...     if(self.fname < self.lname):
...             print "foo"
...     else:
...             print "bar"
...
>>> Person.foo = foo
>>> john.foo()
foo
>>> zack = Person("Zack", "Abrams")
>>> zack.foo()
bar

Of course the downside here is that this is not a closure, and that the method definition exists outside the class on its own which can contribute to clutter in your code. Then on the other hand, perhaps this isn’t so bad as it allows you to do stuff like:

>>> foo(john)
foo
>>> foo(zack)
bar
>>> foo = Person.whatisyourname
>>> foo(john)
'My name is John Smith'

I guess that’s part of Python’s charm that standalone functions can easily become instance methods and instance methods can be easily used as standalone functions.

When the Google App Engine first launched I didn’t really have much to say about this. It was neat but I failed to fully comprehend what it represented. Only after I read a transcript of Steve Yegge’s recent talk in which he briefly mentioned it I realized the significance of this service. It was really an off-hand comment but something just popped in my head the moment he mentioned it. Which is not surprising at all:

Btw, I just wanted to mention that Google App Engine and Google Apps for Your Domain are really heading for a branding conflict. Not even the Google search engine itself can easily distinguish between the two. One ought to be renamed at some point to spare users the confusion.

But why is this service interesting? It gives us a brand new hosting model. You want to build small, throw away web applications? Look no further - all you need is a valid Gmail account and some knack for Python scripting. That’s all. No configuration, no maintenance, and no hassle - you even get a free subdomain at appspot.com.

In the past you really just had to buy yourself some hosting space. There was no way around it. In most cases you ended up with a heavy duty plan with your own domain name, lots of space, a full LAMP stack (or something equivalent that needs to be maintained), ssh access and various other benefits. Often an overkill and and a considerable investment both time and money wise for what you might have needed.

Google on the other hand provides you with a basic environment that’s relatively easy to pick up, and lets you start hacking immediately. It is a bit like what Geocities was to us in the early days of the web - a free hosting where anyone can put their personal web app. Especially if you don’t want to pay a monthly fee because you are broke, or because your parents won’t let you use their credit card to buy hosting. It’s a place for young and inspired teenagers, or lazy and bored developers with a cool idea but no inclination to host and maintain. The mundane details are abstracted - you don’t need source code repository, because each time you deploy Google stores a copy of your old code so that you can roll back.

You don’t need to worry about database specific stuff - like setting up users, connecting to the db and etc - it’s all built in, automated and behind the scenes. It’s actually even easier than in RoR where you still have to create a database configuration file, build migration files and then “rake” them. Google App Engine just requires that you import the db module, and write a Model class specifying data types for the fields you want to store - right there in your code. And the fun part is that you can modify your data scheme by just modifying the code. On top of that you get a neat web interface letting you maintain and browse through the stored data.

You don’t need to worry about authentication scheme since you can just drop in Goggle’s authentication instead. This way they store the passwords, and they are responsible for making things secure and efficient. Implementing these an authentication scheme yourself is always a hassle - so “outsourcing” it this way is generally a good idea - especially for small projects, or when you are feeling lazy.

Getting dugg, or DDOS’ed (aren’t the two the same though?) is no longer your problem, but Google’s problem - and they are pretty good at load balancing throttling and dealing with situations like that. The scalability of Google App Engine can actually be a major draw - for anyone. For example, Jaiku - a somewhat popular twitter clone is migrating to the Google App Engine platform for that very reason: scalability. This move will allow them to concentrate on delivering value to end users via new features and bug fixes rather than fighting a constant logistical battle trying to build up their infrastructure to handle ever increasing server load. It’s not an easy battle to win, as Twitter users can attest to. Scaling up is arguably most expensive, and most labor intensive element of running a popular web application. If Jaiku can offload this cost onto Google they gain an important advantage over Google.

So Google is offering us is a really powerful and versatile package, and it’s free. All you need to do is create an account and start hacking. If you look through their app library you will see all kinds of interesting ideas popping up. Some are big apps like Jaiku, and some are small hobby projects. Out of the latter my favorite must be the TweetWheel. Just try it and see that it is an interesting concept, but not really something you would actually want to pay for hosting.

This is a profound step toward increasingly more complex scriptable server side systems. It will never really replace traditional hosting. Since Google App Engine is a sandbox environment, a lot of apps would be simply impractical or inconvenient to run there. But it offers a new compelling new alternative for established applications, startups and small hobby projects alike. But it is more than that. This is an experiment in allowing users run and host their code on your servers.

We all know that moddable and scriptable systems are valuable because community that grows around them adds immense value. Look at Firefox for example - with millions of extensions and add-ons that do everything from checking your email to playing music. Some are actually incredible paradigm breaking ideas such as Greasemonkey which allows you to inject your own Javascript into a page while it is rendered allowing you to control it’s layout and even add features never considered by creators. This is the type of creativity that can be spurned only by a fully scriptable system. If you want another success story look at Emacs - a text editor which started as very basic text processing engine, and grew into a monster that we know and love today.

So scriptable and moddable applications on the desktop are good thing. And we pretty much figured out ways to make sure scripts and mods can’t totally break your app - at least not easily. We also know that moddable web applications are great too - everyone loves Twitter apps, Facebook apps and Google maps mashups. These are great examples how community can add to a great service. This is why every service which wants to be popular creates a public API. But, if you want to write a facebook app, or a twitter app or something like that you have to host it somewhere. And your hosting plan must be robust enough to survive a sudden spike in popularity. I have seen many a Facebook app go under or become unusable because their servers could not handle the load. And this is a problem - not everyone can afford to have a dedicated server just to host an add-on for a popular online service.

This is the problem that Google App Engine solves. They are the few companies out there which figured out how to safely let people run arbitrary code on their servers without getting pwned by script kiddies. They have found a way to safely, securely and efficiently sandbox user created web applications from the rest of their system. And this is huge. It is a proof of concept that this sort of thing can be done, and it can be done safely.

The implication of this experiment is that people can now start building extendable systems which not only expose a public API but will host and maintain user-submitted add-ons making them much more reliable responsive and flexible. After all, if you are hosting the app there is no reason to communicate via XML, JSON or whatever your public API is using. You can simply let users call a function and return an object they can manipulate eliminating the overhead of TCP/IP transmission and serializing/serializing data. I don’t think anyone is doing that yet, but Google App Engine is a first step in that direction. In the future we can see a brand new species of online services - ones that grow and transform themselves and adopt allowing the vibrant community to add new features and modules. Picture this: an emacs like flexibility and malleability, but for a web service. This could be huge…

Ever since I updated my Windows box to Firefox 3.0 I’ve been having issues with the Google Notebook plugin. I actually use it a lot to “clip” interesting quotes, links or snippets of code and have them accessible from any machine. The plugin makes this easy because I can just highlight something, right click and choose “Note This” to send a copy, complete with formatting and any included images to the notebook.

Lately however instead of the notebook mini-window I have been seeing this:

I tried googling this error message, but I found absolutely nothing other than some random thread full ow “LOL, I have the same problem” comments, and few forum threads in Chinese. Trying Google translate on these threads revealed that they were also full of pointless messages along the lines of “with great justice I have the sadness with such problem also”. It appears that this issue is extremely rare, and that I was on my own. So I started troubleshooting.

1. Removing the notebook plugin and reinstalling it did not work
2. Transfering the folder plugin from the Firefox profile on another computer where it worked did not work
3. Disabling adblock did not work
4. Creating a new Firefox profile, and installing the plugin there worked

This meant that the issue had something to do with my profile. Just for shits and giggles I copied all the plugin files from my old profile to the new profile, and restarted the browser. The Notebook plugin broke again. So the culprit was one of the plugins. But which one? Try a wild guess. I tried guessing but failed miserably, so I disabled all plugins and started switching them on one by one. Do you know which one broke Notebook?

It was TorButton - the one that actually ships with the Vidalia Bundle. Yep, it was literally the last plugin I expected to be responsible for the issue. But disabling it make the Notebook spring back to life for me. Go figure. That button was rather ugly anyway, and I don’t really use Tor that often so I think I’ll survive without it. Living without Notebook was more of a pain.

As usual, I’m putting it up here so that Google can pick it up and people running into this issue in the future can actually get some constructive advice. It is likely one of your plugins. Do what I did, disable them all, and then start enabling them one by one testing the Notebook after each. For me it was the TorButton, but one of the foreign language threads out there mentioned skype plugin a lot so I assume that it can cause this problem too. And if there are two conflicting plugins out there, chances are there are more. So I hope this post will be helpful, and will point you in the right direction.

Cue in comments about how Google is stealing my privacy.

I still have Gutsy on my work computer. I should really upgrade it to Hardy but unfortunately “dist upgrade in progress” is not really an acceptable excuse for not doing actual work so I keep putting it away. Ideally I could probably just run dist-upgrade during my lunch break, and then come back to a fully upgraded system. But every single dist-upgrade I did in the past required some babysitting so I’m not expecting this one to be any different. So for the time being I will stick with Gutsy - at least until I carve out a good chunk of time to devote to upgrading it, and potentially fixing anything that broke in the process.

Then again, maybe if I have upgraded earlier I wouldn’t have this VirtualBox issue today. At work I’m usually running a Windows XP instance inside VirtualBox so that I can access and use various Windows specific apps. Yesterday it just stopped working - it simply wouldn’t boot telling me that a wrong kernel module is installed. I wish I took a screenshot of the error message but I didn’t. Instead I started troubleshooting. First I needed to find out which version of kernel was installed on my machine:

lmaciak@malekith:~$ uname -a
Linux malekith 2.6.22-15-generic #1 SMP Tue Jun 10 09:21:34 UTC 2008 i686 GNU/Linux

It seems that my Kernel is 2.5.22-15. I vaguely remember pulling down some updates on Friday morning. A new kernel was among them, I believe. Next I took a quick peak on my VirtualBox installation:

lmaciak@malekith:~$ aptitude search virtualbox-ose-modules
i   virtualbox-ose-modules                   -
i   virtualbox-ose-modules-2.6.22-14-generic - virtualbox-ose modules for linux-image-2.6.22-14-generic
p   virtualbox-ose-modules-2.6.22-14-server  - virtualbox-ose modules for linux-image-2.6.22-14-server

It immediately became clear - there is a mismatch between my kernel and the installed modules. Unfortunately the only VirtualBox kernel modules in the repositories were the 2.6.22-14 ones. A 2.6.22-15 version simply doesn’t exist. Not for Gutsy at least and the Hardy release has to many unresolved independecies. So if you are still on Gutsy for some reason (like me) and you just updated your kernel (like me) you are now totally screwed (like me). The OSE version of VirtualBox just won’t work and there is no solution for this yet - and there may never be since Gutsy is on it’s way out.

There is a quick workaround though. Ditch the OSE version, and use the non-free binary under the PUEL License. I believe that my use of the software falls uder the “Personal use or Evaluation” clause since I’m not using it for hosting, and not sharing or distributing it. And I will likely only keep it only until I upgrade to Hardy.

If you are in the same boat, here is what you do. First get rid of your current VirtualBox installation:

sudo aptitude remove virtualbox-ose

Next, head over to the sun download page and grab yourself an appropriate deb file (pick Gutsy from the pull-down list). Install it via dpkg:

dpkg -i virtualbox_1.6.2-31466_Ubuntu_gutsy_i386.deb

It will go through the motions, and you may need to acknowledge one or two prompts as it configures appropriate kernel module. Once it is done, you should be all set. Please note that /usr/bin/virtualbox is unlinked replaced by /usr/bin/VirtualBox (note the camel case) which means any old aliases, shortcuts and key bindings will no longer work. You will have to manually adjust them. Other than that, the difference between the two versions in minimal. They use slightly different icons, and a different splash screen but for the most part it is the same software and you can easily use your regular VirtualBox images.

I hope this will help any potentially pissed off VirtualBox users who are still clinging to Gutsy for whatever reason.

Let’s talk backups. How do you do it? I mean, you do make backups, right? We talked about this before many times. Backups are not an option! They are not something that computer geeks do for fun.They are necessity, and if you neglect them or ignore them you will regret it. Don’t tell me you don’t have anything worth backing up on your computer. Just don’t! Everyone says that and then when they lose all their data they cry. When will you people realize that the 60 GB of mp3’s you have meticulously collected since they you were 13 can never be recovered, or recreated once your hard drive dies. It’s gone! You have to start from scratch! Do you even remember which songs you had there? How about your emails? Letters from your first girlfriend/boyfriend? Pictures from that crazy spring break that you never printed or uploaded anywhere cause they were to embarrassing? Don’t tell me things like that are not important to you. Cut the bullshit and go start backing up your data. Yes! Go! Right Now! I’ll wait.

Ok, so now that I made you think about backing up your data, here is a question: do you image the whole disk, so that you can roll back your system to a previous state, or do you just backup your data? Both methods have their benefits and flaws.

Right now, backup mostly my data by rsyncing up my home directory to an external drive. On Windows I use Unison instead of rsync but idea is the same - incremental sync of just my data files. I leave system files and applications alone.

For one it saves space. Believe it or not, the OS, games and software tend to be huge. By saving only the stuff I actually care about, I make better use out of my external media and cut down on backup time.

Secondly, file based backup can be less failure prone. I used to use NTBackup on my windows machine back in the day to take snapshots of the OS state and the results were less than impressive. The backup would run for 18+ hours and then would fail right near the end leaving me with an unusable 100+ GB file on the external media. That may be because NTBackup generally sucks but I believe that most applications that work this way would have similar flaws. I also tried ghosting, but that required a reboot into the ghost environment and had a simillar issue. After the backup was done, I had one big file, which could become easily corrupted. I hear that newer versions of Ghost do not require a reboot but from what I have seen they are terribly bloated which is sort of a trademark for Symantec products. Huge, slow and bloated. Bleh…

A file based backup seems like a more robust procedure to me at the moment. Even if the process fails in the middle of execution, I still usually get a partial backup. Furthermore the files are directly accessible if I need to do only a very selective restore. I do not need to go through some special application to unpack the data I need. I just mount the drive and grab the necessary files.

But that’s me, and I’m always open to new ideas. How do you do your backups? What software do you use? Do you think backing up just data is a bad idea? If I wanted to do full system backup, what should I use? Let me know in the comments.