Terminally Incoherent

Recently I wrote this lengthy on Google App Engine and new avenues of opportunity such ventures open up for us. It seems that I’m not alone in thinking about this stuff. The gist of that post was that Google App Engine and similar services which are bound to pop up sooner or later offer us what I call “a third option”.

Traditionally if you wanted to run a blog, a forum or some sort of dynamic service you essentially had two options. You could either buy yourself some form of hosting (or hot or you could go with a free service. The problem with free services is that you pretty much lock yourself into their proprietary software and are limited by it’s functionality. So while you don’t pay for a blogger, live journal or wordpress.com account you also have very little choice in how your blog will be configured, what “plugins” it is running, and etc. None of these services allows you to change the code of the application itself to extend it for your needs.

Private hosting on the other hand costs money and in most cases requires you to be your own sysadmin. You had to configure your web server and the database, install the forum software, make sure it works and finally protect it from being slashdotted or dugg to death.

Now we have a third option which is sort of a middle ground between the two. You build your app, and you upload it to Google App Engine. This way you have both access to the source code and flexibility which this entails, Google’s famous back end scalability, and it is completely free for you to use making it a an attractive offer.

Now I have a concrete example of someone taking this idea and running with it. Krzysztof Kowalczyk (btw, if you can pronounce that name you get a cookie) just recently released a minimalistic forum software he dubbed FoFou written specifically for the Google App Engine.

He essentially took Joel Spolsky’s forum design idea (side note: Joel can sometimes have those rare moments of brilliance when he is not busy being a fucking tool) took some of the ideas from FruitShow project (which implemented Joel’s forums in PHP) and ported them into Python with Google App Engine in mind. The result is a very light weight, minimalistic discussion board which is quite usable despite it’s spartan design.

But I’ll let Krzysztof explain the motives behind his project in his own words:

I believe that thanks to ease of development, ease of deployment and generous free quotas, App Engine has a potential to revolutionize the landscape of low-end web hosting. App Engine is plenty powerful to host your website, blog, wiki, forums and what not, for free, with better quality guarantees than most low-end web hosts.

It doesn’t make sense to pay even $5/month for low-end web hosting and mess with configuring web servers and databases and PHP software.

It doesn’t make sense to pay $5/month for TypePad blog, if you could install TypePad-like software on App Engine and run it for free.

It’s even attractive to use App Engine instead of free (read: ad supported) services like blogger or wordpress.org, because you get rid of ads, you can install any blog software you like and you get more control over the website.

It’s still early in App Engine lifetime so we’re not yet at a point where this is true. The biggest missing piece is availability of high-quality, free re-implementations of the most common types of software people use: blogs, wikis, forums etc. Fofou is here to give you one more choice when it comes to forum software.

This is precisely what I have been saying in that previous post. This guy totally gets it, and I’m with him on this 100%. We need more initiatives like this where we port popular open source web applications for Google App engine. In fact, if someone at Google is by some fluke reading this blog (which I doubt) this is something you guys could totally pounce on. Make a contest for the best port of a popular app, offer some attractive prizes and cash in on the positive PR, buzz and see a huge influx of users for your Google App Engine.

I must say that I really like Fofou. I grabbed it and set a test forum for myself just to see how easy it was to deploy and configure. It’s a breeze. You create a new app in App Engine web interface, dump the Fofou folder into your Google Apps dev directory, edit app.yaml to include your app’s unique name, run appcfg.py update and you have a working installation. All that is left is to log in and create a forum. Easy as pie - or even easier.

In the past my brother asked me for recommendation on a free forum hosting and I ended up sending him to some crappy PHPBB based service which turned out to be a complete disaster (they had more downtime than dreamhost and twitter combined which is actually something that is not very easy to achieve). This time around, I would be tempted to set him up with Fofou on Google App Engine… Then again Fofou might have been too bare bones for what he needed, since he was all into that secret-password-protected forums, user privilege stratification and etc. Still, it would be an attractive offer.

Let’s hope that Fofou is just simply an early indication of an upcoming trend of porting popular applications to the App Engine. And if Google at some point adds PHP as the second supported App Engine Language next to Python (which they probably won’t) we might see an avalanche of behemoths such as Wordpress, PHPBB and etc being ported en masse.

When the Google App Engine first launched I didn’t really have much to say about this. It was neat but I failed to fully comprehend what it represented. Only after I read a transcript of Steve Yegge’s recent talk in which he briefly mentioned it I realized the significance of this service. It was really an off-hand comment but something just popped in my head the moment he mentioned it. Which is not surprising at all:

Btw, I just wanted to mention that Google App Engine and Google Apps for Your Domain are really heading for a branding conflict. Not even the Google search engine itself can easily distinguish between the two. One ought to be renamed at some point to spare users the confusion.

But why is this service interesting? It gives us a brand new hosting model. You want to build small, throw away web applications? Look no further - all you need is a valid Gmail account and some knack for Python scripting. That’s all. No configuration, no maintenance, and no hassle - you even get a free subdomain at appspot.com.

In the past you really just had to buy yourself some hosting space. There was no way around it. In most cases you ended up with a heavy duty plan with your own domain name, lots of space, a full LAMP stack (or something equivalent that needs to be maintained), ssh access and various other benefits. Often an overkill and and a considerable investment both time and money wise for what you might have needed.

Google on the other hand provides you with a basic environment that’s relatively easy to pick up, and lets you start hacking immediately. It is a bit like what Geocities was to us in the early days of the web - a free hosting where anyone can put their personal web app. Especially if you don’t want to pay a monthly fee because you are broke, or because your parents won’t let you use their credit card to buy hosting. It’s a place for young and inspired teenagers, or lazy and bored developers with a cool idea but no inclination to host and maintain. The mundane details are abstracted - you don’t need source code repository, because each time you deploy Google stores a copy of your old code so that you can roll back.

You don’t need to worry about database specific stuff - like setting up users, connecting to the db and etc - it’s all built in, automated and behind the scenes. It’s actually even easier than in RoR where you still have to create a database configuration file, build migration files and then “rake” them. Google App Engine just requires that you import the db module, and write a Model class specifying data types for the fields you want to store - right there in your code. And the fun part is that you can modify your data scheme by just modifying the code. On top of that you get a neat web interface letting you maintain and browse through the stored data.

You don’t need to worry about authentication scheme since you can just drop in Goggle’s authentication instead. This way they store the passwords, and they are responsible for making things secure and efficient. Implementing these an authentication scheme yourself is always a hassle - so “outsourcing” it this way is generally a good idea - especially for small projects, or when you are feeling lazy.

Getting dugg, or DDOS’ed (aren’t the two the same though?) is no longer your problem, but Google’s problem - and they are pretty good at load balancing throttling and dealing with situations like that. The scalability of Google App Engine can actually be a major draw - for anyone. For example, Jaiku - a somewhat popular twitter clone is migrating to the Google App Engine platform for that very reason: scalability. This move will allow them to concentrate on delivering value to end users via new features and bug fixes rather than fighting a constant logistical battle trying to build up their infrastructure to handle ever increasing server load. It’s not an easy battle to win, as Twitter users can attest to. Scaling up is arguably most expensive, and most labor intensive element of running a popular web application. If Jaiku can offload this cost onto Google they gain an important advantage over Google.

So Google is offering us is a really powerful and versatile package, and it’s free. All you need to do is create an account and start hacking. If you look through their app library you will see all kinds of interesting ideas popping up. Some are big apps like Jaiku, and some are small hobby projects. Out of the latter my favorite must be the TweetWheel. Just try it and see that it is an interesting concept, but not really something you would actually want to pay for hosting.

This is a profound step toward increasingly more complex scriptable server side systems. It will never really replace traditional hosting. Since Google App Engine is a sandbox environment, a lot of apps would be simply impractical or inconvenient to run there. But it offers a new compelling new alternative for established applications, startups and small hobby projects alike. But it is more than that. This is an experiment in allowing users run and host their code on your servers.

We all know that moddable and scriptable systems are valuable because community that grows around them adds immense value. Look at Firefox for example - with millions of extensions and add-ons that do everything from checking your email to playing music. Some are actually incredible paradigm breaking ideas such as Greasemonkey which allows you to inject your own Javascript into a page while it is rendered allowing you to control it’s layout and even add features never considered by creators. This is the type of creativity that can be spurned only by a fully scriptable system. If you want another success story look at Emacs - a text editor which started as very basic text processing engine, and grew into a monster that we know and love today.

So scriptable and moddable applications on the desktop are good thing. And we pretty much figured out ways to make sure scripts and mods can’t totally break your app - at least not easily. We also know that moddable web applications are great too - everyone loves Twitter apps, Facebook apps and Google maps mashups. These are great examples how community can add to a great service. This is why every service which wants to be popular creates a public API. But, if you want to write a facebook app, or a twitter app or something like that you have to host it somewhere. And your hosting plan must be robust enough to survive a sudden spike in popularity. I have seen many a Facebook app go under or become unusable because their servers could not handle the load. And this is a problem - not everyone can afford to have a dedicated server just to host an add-on for a popular online service.

This is the problem that Google App Engine solves. They are the few companies out there which figured out how to safely let people run arbitrary code on their servers without getting pwned by script kiddies. They have found a way to safely, securely and efficiently sandbox user created web applications from the rest of their system. And this is huge. It is a proof of concept that this sort of thing can be done, and it can be done safely.

The implication of this experiment is that people can now start building extendable systems which not only expose a public API but will host and maintain user-submitted add-ons making them much more reliable responsive and flexible. After all, if you are hosting the app there is no reason to communicate via XML, JSON or whatever your public API is using. You can simply let users call a function and return an object they can manipulate eliminating the overhead of TCP/IP transmission and serializing/serializing data. I don’t think anyone is doing that yet, but Google App Engine is a first step in that direction. In the future we can see a brand new species of online services - ones that grow and transform themselves and adopt allowing the vibrant community to add new features and modules. Picture this: an emacs like flexibility and malleability, but for a web service. This could be huge…

I’m on JavaScript kick lately. Or rather, I have been rediscovering it ever since I read the series of Steve Yegge’s blog posts on JS and Rhino and his predictions for the next big language. I really think he is right - JS will be the next big thing that will trump Ruby from it’s pedestal or at least become its viable competitor as the next big dynamic scripting behemoth. Why? I think a better question is: why not? It almost seems like this language is destined for greatness.

For one, JavaScript or rather ECMAScript is really a kick-ass language if you look at it objectively. When I say JavaScript most people automatically think about the client-side, browser dependent, inconsistent scripting language which only works half of the time. But this is not really the fault of the language but inconsistent implementation of the DOM model across different browsers.

Internally the language is really consistent, flexible, readable and fun to use. Not only does it have most of the cool features that are all the rage now (ducktyping, closures, variadic functions, prototypes, metaprogramming end etc..) but reads a lot like C. The syntax is full of the familiar and lovely brackets and semicolons that we all know and love (or hate) and all the control sequences and language constructs follow the familiar C patterns. Coding in JavaScript is really a lot like coding in Java only 5 times less verbose which is a good thing.

And when it comes to shuffling around HTML or XML data there is no other language that is both so good at it, and so well documented. We have been using it for that purpose decades now and nothing else compares. No other language can be used int the online so effectively and seamlessly.

Also, Javascrip has something that neither Python nor Ruby possess - a real spec. So it is really a good language - a Language with capital L. It is just burdened with a bad stigma of being a browser specific toy language.

This stigma is slowly dissipating as the Web 2.0 craze is raging on. I don’t care who you are right now - if you are doing any web development right now, you have to deal with Javascript whether you want it or not. You can avoid it, you can claim you are server side guy but if you want to be competitive right now your web app needs to use AJAX and if it is, then Javascript is becoming a big internal part of your code base. You can hide behind a framework, and have it generate AJAX code for you the way RoR does, but sooner or later you will have to do something complex and will end up hacking on js files. Whether you love it or hate it - you will be exposed to it at some point.

So your code is now roughly half Javascript and Half something else which is a bit messy, but that’s Web 2.0 for you. Does it have to be that way though? Of course not. You don’t need a dedicated client side language ad another dedicated server side language. You could really do all your scripting on both sides with Javascript. Yes, Server Side Javascript is not a joke or some pipe dream - it is reality. Believe it or not, but there is a lot of development going on in this area - there are tons of projects out there which use JS on the server side.

These projects range from very simple to very complex full featured solutions. For example, mod_js is an Apache module loosely based on Perl’s CGI.pm which implements ECMAScript on the server side. It is astonishingly simple, and compact but also very immature project.

If you want something more robust and mature there is Jaxer - a self contained Java based server which supports server side scripting in Javascript. The whole idea behind this environment is unifying your code base. With Jaxer you can use the very same language to query the database, dynamically rearrange the HTML output, and make asynchronous calls back to the server. Added bonus is that it is bundled with the very popular Aptana IDE (aka. the Eclipse for web designers) which means it is already near ubiquitous.

If you are allergic to Eclipse and it’s forks and you like NetBeans then Phobos is probably what you will want to sink your teeth into. It uses Glassfish server and uses Javascript as it’s default server side language. Same concept, different approach. The project seems to be mature, well supported and have a big community.

Want more? Theres Helma which is a Rails like framework which leverages server side Javascript and Rhino. The whole thing runs in the JVM and uses Jetty as the default, self contained web server - but you can also put it behind Apache or IIS if you want. How come you can go wrong with a combination like that? You get a MVC framework with a self contained server and set of useful tools and utilities that only requires you to have a basic JVM installed. You get to script it with the dynamic and flexible language that requires no compilation, and no SDK. You get the entirety of the Java API at your disposal courtesy of Rhino. Not only that, but you can simply drop in your custom precompiled Java libraries and classes and seamlessly integrate them into your project.

But there is more - recently I spotted another interesting project called Junction. Like Helma it is a Rails like MVC framework that uses Rhino but in addition, it also has a built in support for Google Gears. This means that apps built on the Junction framework get to benefit from the offline persistence features offered by the Google technology.

There is all this development going on out there. It’s not very visible yet - most of it is somewhat behind the scenes. But it’s like that distant rumbling before the storm. You know something big is coming. Just look around.

You want to develop rich client interfaces? There’s XUL and Adobe Air for you - both using Javascript at their core.

You want dynamic web UI framework? There is JQuery, Prototype, Dojo, Google Web Toolkit and YUI just to name few.

Want to use Java API? Use Rhino. Want to tap into the richness of Microsoft CLR and interact with the rest of .NET ecosystem? Theres JScript.NET. You want to do some native scripting on MacOS? There is Javascript OSA.

You want to add oflline persistence to your web applications? You need Google Gears which is also using Javascript.

The language is everywhere. Everything that is new, that is hot and that relates to the web is at least touching or benefiting from it. Every developer will see it, and work with or alongside it sooner or later. It’s ubiquitous.

The pieces are slowly falling in place and setting up the stage for the upcoming explosion. It’s no longer question of whether we will reach that critical mass, but when will it happen. We have the tools, we have the technology, we have the hook and the need (unifying the client and server side under one language) - now we just need to wait for the hype. And for that, we need branding. Steve Yegge talked about this during his Ted talk - we need to re-brand the language to make it more sexy. Everyone thingks that Javascript is a toy language, and ECMA sounds like a some sort of acne prescription so we need a new, cool name. We also have a good reason for re branding - the upcoming release of Javascrip2 standard which brings about many new features to the language such as classes, interfaces, namespaces, packages, destructuring assignments, structs, type declarations and more.

It is a bucket load of new features so why not start fresh, with a new sexy name and identity. I mean look at AJAX - the acronym really means “using the asynchronous calls in Javascript” but if you say it like that, people immediately fall asleep and run away. But AJAX is cool and sexy. Javascript can be to - and I predict it will be, sooner or later.

For a while now, I have been searching for a good Javascript framework to mess around with. I think I finally found one that I really like. JQuery is a 15 Kb bundle of pure fucking magic. It is tiny compared to some of the other frameworks out there, and at the same time very powerful. What can it do? Tons of stuff. First thing you notice is of course eye candy things like sliding and hiding divs, easily manipulating tons of data on the page and etc.

The best thing about JQuery however are the plugins created by the very active community. For example, there is a small script out there that lets you sort HTML tables on the client side. Normally when I dumped data out of the database on the screen I would implement sorting by simply making another database query with a modified WHERE clause and then re-draw the whole table with the new dataset. But why should I use such a database intensive method when JQuery makes it possible to quickly sort a large table without even reloading the page using client side resources?

Go check out the demo on the tablesorter plugin website. It is really a great piece of work. Christian Bach deserves major props for putting this together. To tell you the truth, I never really considered sorting of tables in javascript mainly because the amount of effort it would take on my part. Implementing the sorting algorithm and the transformations would just be so much more time consuming than simply writing a quick SQL query. It somehow never seemed worth the effort to me, until now.

The way I see it, this method of sorting has two major benefits over the old fashioned server side method. First, as I mentioned earlier it takes some load of the database. Second, the sorting will seem much faster to the user because we are not reloading the page. It should even be faster than an asynchronous call as these things tend to have a visible delay. Tablesorter manipulates DOM objects which are already loaded in clients memory - there is no loading or waiting required.

It even comes with a neat pager functionality which could come in handy. Then again, I’m not absolutely convinced that paging should be done on client side for very large tables but it is a nice feature to have just in case you need it.

Just to illustrate how easy it is to apply it to an existing table here is what you need to do. First download both JQuery and Tablesorter and dump them somewhere. Then simply add following calls in the header:

<script type="text/javascript" src="/path/to/jquery.js"></script> 
<script type="text/javascript" src="/path/to/tablesorter/jquery.tablesorter.js"></script>
<script type="text/javascript">	$(document).ready(function() { $("#myTable").tablesorter(); } );</script>
<link rel="stylesheet" href="/path/to/tablesorter/themes/blue/style.css" type="text/css" media="screen, print" >

Then simply make your table id to be “myTable” and it’s class to be “tablesorter”. The first one is mandatory for the sorting logic to work. The “tablesorter” class is defined in the css stylesheet you link to and it will create nice headers and table layout for you:

<table id="myTable" class="tablesorter">
	<thead>
		<tr>
			<th>Column 1</th>
			<th>Column 2</th>
			<th>Column 3</th>
		</tr>
	</thead>
	<tbody>
		<tr>
			<td>1</td>
			<td>a</td>
			<td>foo</td>
		</tr>
		<tr>
			<td>2</td>
			<td>b</td>
			<td>bar</td>
		</tr>
 
	</tbody>
</table>

That’s it. You are done. Your table is completely sortable now, by every column, and is even nicely formated. It couldn’t be easier than this. And yes, you do have to use the thead and tbody tags.

There is one caveat though. When you download current release, be aware that the jquery.tablesorter.pack.js file is broken for some reason. It will work just fine in Firefox, and Opera but in IE6 it will throw some sort of cryptic bug. It might be some artifact introduced in the packing process or something like that. If you use jquery.tablesorter.min.js which is only marginally larger in size, this issue goes away. Go figure. It’s still a great tool despite this little glitch.

I think I’m sold on JQuery. It is tiny, and yet extremely powerful toolkit, and I highly recommend checking it out.

As you may or may not know, Facebook removed it’s course tracking application at the beginning of August. Apparently this was done in order to make more room for home-grown facebook apps. I didn’t notice it until yesterday when I decided I would stalk my prospective students. I guess I just started mentally filtering out all the news stories with the keyword “Facebook App” in them in the last few months.

For one, I do not understand this decision. This type of applications only work well if everyone is using the same one. When the functionality was built in, everyone was on the same page so to speak. If you entered in your courses, you were immediately connected with all the other students who did the same.

Currently, users have to pick between 6 or 7 popular course tracking apps (and new ones are popping up every day it seems) - all of which are essentially walled gardens. Where once there was unified user base, we now have fractured clusters spread over multiple domains. If you install one of them, and your friends installs another you will never know that you are in the same class. So which one do you join? Do you join them all?

I tried several, and the 3 top ones had major scalability problems. They were all slow, and one of them basically refused to display anything but an error message. It appears that these things are hosted on 3rd party web servers that are in no way shape or form capable of dealing with Facebook grade bandwidth onslaught. So the choice now seems to be reduced to either using a popular app which is consistently down due to a unintentional DDOS, or a more obscure one that has no active users. I’d rather have basic no-frills functionality than a feature rich environment that doesn’t work, or lacks users.

By liberating the course tracking functionality, Facebook essentially killed the whole idea of easily connecting with your classmates that made their social network so popular and quickly adopted. Yes, I do understand that the network is not just for college students anymore, but that is no reason to remove the functionality that attracted students to this network in the first place. Welcome to Myspace 2.0 kids. That’s where Facebook is heading.