Terminally Incoherent

Every time I try to post something on Twitter lately I see this:

Either that, or an internal server error. I have been twittering less and less in the last month or two - but at this point I hardly use it anymore. Every time I get around to post something it is down. I have an account on Jaiku that I hardly ever use. Maybe I should switch to that?

Anyone knows what the hell is going on with twitter lately? I know they were getting hammered hard with traffic, but for some time there seemed like they got the hand of it, and worked out how to load balance and cache everything.

I’m really interested in Twitter because it is a great case study in using Ruby to handle a high traffic social network. It seems that they are slipping now - I’m curious to see if this is because of problems scaling up Ruby on Rails, or is it something completely unrelated to the language and platform of their choice.

As you may or may not know, I’m working on a Java based Twitter client recently. Sending and getting data from Twitter is really, really simple. Here is how to do it in Java.

Let’s assume that we are trying to post an update to twitter with the following data:

String username = "someone@example.com";
String password = "password";
String status = "Look! I'm twittering using Java";

Twitter uses basic HTTP authentication which requires you to send a username:password pair encoded in base64:

String cridentials = new sun.misc.BASE64Encoder().encode((username + ":" + password).getBytes());

Now let’s create a HTTP connection:

URL url = new URL("http://twitter.com/statuses/update.xml");
URLConnection conn = url.openConnection();
conn.setDoOutput(true);

Authentication is activated like so:

conn.setRequestProperty ("Authorization", "Basic " + cridentials);

Now we can send the update by simply writing to conn’s output stream:

OutputStreamWriter wr = new OutputStreamWriter(conn.getOutputStream());
wr.write(data);
wr.flush();
wr.close();

Finally, if you wand you can grab and print out the XML response sent back to you by Twitter server:

BufferedReader rd = new BufferedReader(new InputStreamReader(conn.getInputStream()));
String line, output = "";
while ((line = rd.readLine()) != null)
{
System.out.println(output);
}

That’s it - that’s all you need to do.

Ruby is a really great language with some awesome features, and Rails make it even better. But it appears that it does not scale well yet. Twitter is a great example of using RoR to run a website with millions of users hitting your server few hundred times a day each. Amazingly enough, it seems to be running very smoothly lately, but they did have many rough spots in the last few months when the sites performance was really poor.

It seems that their issue is not the execution speed, but the Rails design itself. In a recent interview a Twitter developer Alex Payne said:

The common wisdom in the Rails community at this time is that scaling Rails is a matter of cost: just throw more CPUs at it. The problem is that more instances of Rails (running as part of a Mongrel cluster, in our case) means more requests to your database. At this point in time there’s no facility in Rails to talk to more than one database at a time. The solutions to this are caching the hell out of everything and setting up multiple read-only slave databases, neither of which are quick fixes to implement. So it’s not just cost, it’s time, and time is that much more precious when people can[’t]reach your site.

Emphasis was mine. The lack of support for multiple databases seems a huge bottleneck issue for high traffic sites. Most of us don’t even think about stuff like that when considering using a platform to build our applications with.

I still think RoR is a good platform. It just might not be ready for the enterprise scale apps like Twitter. Most of us however will never experience this issue - unless of course we get slashdotted or something. I’m confident that the Rails team will use the Twitter problems to improve scalability features in future releases.

Now that my thesis is done, I feel that my procrastination skills might get bit rusty. Or even worse, I might start using them at work. Oh wait, never mind - I already do that. Either way, I need more exercise in extreme procrastination.

So, I figured I need a project. Something that I could neglect, put away for later, and generally feel guilty about not doing. Yes, that is exactly what I need.

So for the lack of better ideas I created a Java Twitter Client project on Google Code. Note that I haven’t written a single line of code yet. I haven’t really did much research on this either, beyond scanning through the Twitter API.

Why twitter? Perhaps because twitter is cool. It is still very hot on teh internets, and it’s API seems relatively straightforward. I’m not really looking to compete with Twitterific, Twitteroo and similar clients. This is more of a fun project for me to mess with.

Any input would be greatly appreciated. If you want to get involved and help out with coding, testing or whatever, let me know and I’ll add you to the project. Note that you will need a google account. If this project ever goes anywhere, we can always move it over to Source Forge. Google Code seems sufficient for now.

Of course I could host it here. I have subversion running on this very server. But Google Code offers integrated Wiki, and a bug tracking system which is nice. The 100 MB quota is kinda lame. Come on Google, I get 3 gigs for email, but only a hundred megs for code? WTF? Still, my current Eclipse workspace with 17 different projects in it, is only 30 MB. It should be enough, at least for the time being.

So, what do you guys think? Suggestions?

If you are not on Twitter, or if you are not following any of the top 10 people on the twitterholic list you probably haven’t heard about Jaiku. A good part of twitts in the past week dealt with bashing this new service, praising it, discussing it’s merits and etc. Leo Laporte, a former top 5 contender on twitterholic actually completely jumped ship. I figured that if all the cool kids are doing it, it might be worth while to check it out. After playing around with it for few minutes I kinda realized that this is probably how Jaiku came to be:

Web 2.0 Startup Foundr: Did you see that Twinttor thing? That is fucking tits! I want that. Can you copy it?
Web Designr: Sure thing chief.
Web 2.0 Startup Foundr: But I want it better… More web 2.0-ish.
Web Designr: Twitter is pretty Web 2.0 already…
Web 2.0 Startup Foundr: No, I want more. I want like rounded corners, and like cutesy little icons all over!
Web Programr: And we could make it, like, import feeds like Tumbr and shit!
Web 2.0 Startup Foundr: YES! Do that!
Asshole from Maketing: And we need ads or some product tie-in.
Web 2.0 Startup Foundr: Goes without saying of course…

This is exactly what Jaiku seems to be. A Twitter clone with a more polished, professional look, a Tumblr like feed import feature, and some sort of advertising deal with Nokia. All in all it is not bad. I signed up and imported in all my feeds into the service. You can find my jaiku profile here.

I still prefer Twitter because it focuses on simplicity, usability and extendibility instead of a rich feature set, or graphical design. Besides, now Jaiku will pull in my Twitter feed so all my twitts will show up on both services.