You know, we didn’t even have the failwhale back then. We had fucking lolcats notifying us or error messages. If you don’t remember that fucking cat, you can consider yourself a newbie.

Over the years I watched the service grow, expand, become more stable and add new features. I wrote several twitter tools and clients, and tried just about every single third party client out there. In fact, Twitter even made my ISP suck less that one time. I love the service and I enjoy using it.

This is why I get annoyed when people who just heard about Twitter yesterday tell me that Twitter is the stupidest thing ever. I’m sorry, but you have no clue what you are talking about. I’ve been using it for years now, and I find it very useful and valuable. You just saw it on TV the other day and you are convinced it is useless. May I ask how did you decide this?

Usual answer I get is something among the lines of:

“Well, why would I care what someone ate for lunch or what TV show they are watching right now”

I don’t care abut that stuff either, but I use Twitter every day. I just don’t follow people who only tweet about their lunches or their TV shows.

“I also don’t care about what the celebrities are posting – it’s mostly stupid, useless crap anyway.”

I agree, but again, no one is forcing you to follow these people. I have no interest in the crap that Oprah, Shaq or one of the other countless mainstream celebs might be spewing out on the internet. I follow people I know IRL or from the interwebs, people who I find interesting (Jeff Attwood – creator of Coding Horror and Stack Overflow, like John Resig – the jQuery creator, Adam Savage – the Mythbuster, Chris Boys – the infamous Papeghost, the pwnr of script kiddies or Shamus Young known for DM of the Rings, Stolen Pixels his blog and bunch of other things) or genuinely funny (hotdogsladies, Matthew Baldwin, Adam Lisagdor, Scott Simpson, Tim Siedell, domnit). How’s that for a starter list? Feel free to add awesome follow-worthy names in the comments.

“Ok, ok. But I already have Facebook. Twitter is like Facebook status without my Facebook contacts, my applications, pictures, notes, feeds and all that good stuff. Why would I ever want to use that?”

Here you are just plain old wrong. Twitter is not a “Facebook status without Facebook”. It is a completely different animal. On twitter most people have public profiles which means anyone can follow them. So you can subscribe to the updates of interesting people without them having to approve you. Can you do that on Facebook? No, on Facebook most people have private profiles (which is actually a good thing, considering the amount of info you can find on a persons Facebook page). This means it is easier to meet people, and have conversations with folks you don’t know and don’t intend to follow.

Twitter also allows you to reach a wider audience. For example, who reads your status updates on Facebook? Friends, family, people you know from school, work and perhaps some internet buddies you trust. You could mess around with the privacy settings and allow more people, but still – your audience will be limited.

Let me give you a real life example. The following exchange could never happen on Facebook, but on Twitter this is normal:

Twitter Conversation (click to enlarge)

It is a different service used for different purpose. If you using it the way you use Facebook then you are probably doing it wrong.

Not only that, but Twitter is unique among social services in that it is not a walled garden. To access anything on Facebook, you need to create an account. Twitter only requires you to create account to post tweets. You can however follow someone without ever logging in using the public RSS feeds. You can also archive and back up your own tweets in the same way.

The public feeds and public API allows people to create interesting applications and mashups on top of Twitter. A good example could be Twistori or Favrd.

What bothers me the most, is the people who are throwing all this Twitter hate around are not old. Old ladies for example seem to be very interested in the service and often ask me how do you set it up, how it works and etc. The haters usually are young, often internet savvy people.

I guess this is a status thing. These people were too cool to use it back when it was geeky and virtually unknown. Now they are too cool to use it, because it is “oh, so mainstream”.

What is your take on Twitter? Have you encountered a lot of this type of Twitter hate lately? Let me know in the comments.

I was googling my self the other day (what, you don’t do it?) and I stumbled onto something interesting. I found few results pointing back to some source code on the github. This intrigued me because I have never really used git, or set up a project on github.com. And yet, there was some source code there with my name attached.

Lo and behold, the code was actually from my JTwitt project. I started it back in April of last year, and mentioned it sporadically on the website. I didn’t really get far with it. I implemented basic, bare-bones functionality that allowed me to send and receive updates to and from Twitter. Then I got busy with some other things, and sort of let it sit there. I haven’t touched the code in ages.

And yet, somehow it found it’s way into a Present.ly application for Android. These guys actually did a great job cleaning up my sloppy code, and making it work with their application. I’m glad I was able to contribute to their project, if ever so slightly. This, ladies and gentlemen is the beauty of the open source movement. If you set your code free, someone will find use for it, and will maintain it for you. I love it!

Now I’m thinking about dusting off this stale code, and starting working on it again. I could probably integrate a lot of the changes and fixes the presently guys did back into my code, to make it more better. :) Maybe…

Well, for one I noticed that other people did the same thing. Both ktwitter and ZenTwitter are similarly simple, and yet they show up as “Linux Apps” on the Twitter Fan Wiki. Both scripts seem no more complex than what I wrote so why the hell not call my script an “twitter client app”?

The thing is that if you have a twitter client project which has it’s own website you can request a custom source parameter for it. So you can sort of brand it. Previously every time I posted using my script, the twitt showed up as “from web”. Now, it says “from twimi” and links to the google code page for the project which is a nice touch. So I pretty much uploaded it up there just to get that custom link underneath my twitts every once in a while. :)

The second reason for posting it there is perhaps some sort of small glimmer of hope that some shell scripting guru will be willing to contribute to it and add new cool features to it. Right now it is very simple an minimalistic. It is a starter script that you can take and modify to make your own curl based client. I’m not going to post the code here because you can just easily browse it on the project page. It’s only about 60 lines of code, not counting the comments, and GPL boilerplate. If you want to contribute let me know and I’ll add you to the project and you can hack away at it and perhaps the thing will grow on it’s own. I don’t really care for the direction we take it in. I don’t care if it grows a front end of some sort, or if you decide to toss curl in favor of something else.

I know this is not a real project. But, everyone knows a little bash, right? And everyone can write a better script than this in 5 minutes or less, right? So just do it!

Suggestions, comments and contributions are as always appreciated.

As you can see, we have a nice 3 way, asynchronous conversation going on here. Now, how hard do you think it was for me to generate the image above and make it thread like that? Can it be done automatically? Nope. The way I did this was to fish out and screen-cap the particular messages one by one. Also, they were not all in one place. I had to go and grab some of them from Miloš’s and Billy’s twitter pages.

Quotably is a valiant attempt to automatically generate stuff like the manually constructed conversation above. Unfortunately due to the fact that Twitter’s threading model is broken by design it doesn’t always work. For example, here is what it caught from this conversation:

As you can see, it’s almost there. It just that a lot of the tweets simply got dropped for various reasons. For example, using the @name notation in your message more than once will confuse the system and it won’t link to anything. The other issue was that at one point Miloš and Billy were talking to each other and while I still saw their conversation happening on my updates page , and it was still regarding the same topic page, Quotably wouldn’t associate it with my name and include them in this thread.

Billy is right, putting the StatusID value in the message itself is probably a bad idea as it would quickly start eating away our precious 140 characters. But Twitter already does some behind the scenes magic, when it links your message to the top tweet of the person you replied to. So in theory you should be able to re-construct the whole thread by simply clicking on the “in reply to” links. This hardly ever works though. Most conversations are asynchronous so by the time you replied to something, the author of that tweet might have posted 3 or 4 new messages and yours gets linked to the most recent one.

There really ought to be a way to link to a message rather than to a person. We are already halfway there – each tweet has a reply button associated with it. Now all we need to do is to make these buttons do something – and preferably without losing any characters in the message area.

Before that happens though, we need a better threading app. Is it possible to do it with the current Twitter setup? Perhaps, but we can’t really really rely on the in-reply-to implementation currently in place. This is what Quotably is doing right now, and it is not really working for conversations such as the one above. So how do we thread messages?

Perhaps we could try to reconstruct a conversation by simply doing a full text search on the @name pattern and then arrange tweets chronologically. So a user could select 2-3 user names, and then have the Quotably like service fetch all their status updates in which either of the chosen users includes an @name pattern of any of the other two. No indentation, or other thread structure would be necessary here, because it is pretty much infer it from the data at hand. But if we list them chronologically we at least capture them the way they originally appeared in your Twitter stream.

It’s not perfect but it could be a nice complement to a service like Quotably. I do not purpose using it as the main threading system, but rather as an alternative which could help users track more complex, many sided conversations between a group of users.

[tags]twitter, quotably, conversations, threading[/tags]

When I came in to work yesterday morning all was well. Few people around the office mentioned something about a slow day, noting their inbox was unusually empty. I didn’t really pay much attention to that chatter, until someone decided to email something to herself and it never came through. Then all hell broke loose.

You see, this problem was essentially hidden from regular users because they could connect to my in-house server on port 587 without any issues. So their emails were leaving their outbox as normal, and then queuing up on the server, never to be seen again. The server itself could not shoot them back failure notifications either, because it could not connect to it’s relay point on port 25. So people were emailing each other all morning without even realizing something was amiss. When they caught on, it was instant panic mode spreading throughout the organization like a fucking wildfire.

For several hours I was methodically checking, re-checking, restarting, and power-cycling every single device and service that had anything to do with email traffic. I was also calling the folks who maintain the off-site server every 5 minutes to see what was their progress. They were convinced the issue was on my side, and I was adamant that it was on their side. After much deliberation, we came to the conclusion that we were both wrong. The off-site server was accessible from everywhere but my location, but there was nothing here in the office which would prevent it from communicating on port 25.

We tested outbound and inbound traffic on their side and it was working just fine so that left only one conclusion – my ISP fucked us over and completely sealed off port 25. Once we realized that, the conclusion was as swift as it was simple. We simply switched the external server to listen on port 587, changed the outbound port in IIS and an avalanche of backed up email started streaming into people’s mailboxes.

Let me run that by you again in case you didn’t notice – once we figured out what the issue was, it took us 5 seconds to reconfigure our shit, and route around it. So if this supposed anti-spam measure is so easy to circumvent, then can someone explain to me how is it supposed to be stopping hard core spammers with their sprawling botnets out there? I’m pretty sure most of semi-modern spam-trojans can be remotely reconfigured to send out emails on alternate ports.

Port blocking has became pretty much an industry standard these days, but I still fail to see how it could ever be effective. What is stopping me from running an email server on port 80 or 443? Will they block these two ports as well? It is just a knee jerk reaction, that might be effective in a short term. It won’t work in the long run though – soon they will run out of ports to block, and regular customers won’t be able to use any kind of non-standard internet services for genuine purposes without bending over backwards.

This is just one of these wholesale, one-click-and-your-done spam solutions. Why do ISP’s do it? Because it’s easy! You block some important ports, and the amount of spam and genuine email routed through your network goes down. You boss is happy, your investors are happy, folks in the security business are clapping their hands marveling at the sudden drop in spam, forgetting it will be back to normal in a month or two as all the spammers will figure out the same thing I did just now.

The only people who are not happy about this are the customers, but Comcast does not really care about them that much anyway as it has blatantly demonstrated in the past with it’s bandwidth throttling, and lackluster tech support.

Also, Twitter > than regular tech support resources it seems:

Despite the fact that we were constrained to 140 characters per pop, talking asynchronously and multitasking, this was still way more pleasant than my experiences with Mr. Rooter and Mr. 125 Times. Not sure if that guy is an actually really affiliated with the company in any way, but he seems to be representing them well in the 140 character conversation universe.

One more reason to love Twitter and hate Comcast! :mrgreen:

[tags]comcast, port 25, email, smtp, spam, comcast sucks[/tags]

If you don’t know what AIR is, chances are you also don’t know about Twitter, which means you have been living under a rock for the last 900 years. I’ll explain it anyway. AIR was designed to be a platform for lightweight desktop applications, and has been essentially hijacked by Twitter enthusiasts. The most notable applications running on this platform right now are Twitter clients such as Snitter or Spaz. I’m sure it has some other uses, but I don’t know them. The only AIR apps I have seen so far are Twitter clients, and clients for other Twitter like services. If you know some interesting non-twitter related AIR apps I would love to see some links.

I complained about this before but the selection of Twitter clients on Linux is piss poor. Go to Twitter Fan and just look at the linux section. We have two emacs based clients, a vim based client, one written in bash, one which is essentially curl+zenity, a gnome desktop widget, and two or 3 more clients which just wouldn’t work properly on my machine due to bugs or dependency issues. If you scroll up and down you will see that Windows and Mac have all these nice, shiny graphical clients rich in features.

I installed Linux version of AIR to see if some of the nice AIR twitter clients will work on Linux the same way they work on Windows. The answer is – sort of. I installed both of the clients mentioned above: Snitter and Spaz. Nice thing about Air is that the installation looks and feels exactly the same as on windows. So you get a very pretty graphical installer. I’m using Kubuntu (Gutsy) so the apps got installed in /opt which frankly is where I’d put this type of stuff given a choice. The installer also put shortcuts into my KDE menu, although they used the full package name which contained a very long checksum of some sort. Still, I’m not complaining – it works as expected.

It took me a little while to figure out how to uninstall AIR apps until I realized that they are actual deb packages. So you can easily remove the apps you don’t need using aptitude which is great news. Too bad they don’t let you know about this when you are installing apps. I had to go to their Linux FAQ to find out about it.

The apps are a slightly different story. You can look for yourself. Here is how Spaz looked for me:

You can see a little blue background on this shot – this is not because I did a sloppy cropping job, but because I wanted to show you that the drop-shadow transparency that some of these apps use is not working here. On windows the edges are nicely rounded and have that faded shadow effect. On linux you get the ugly opaque black border. I guess this is one of these features which was not implemented in in the Alpha release.

Spaz also had major problems remembering my username/password. It would keep it in memory for the duration of my session, but when I closed and reopened the app it prompted me to type them in again. Initially I thought it was because Spaz was trying to write some config file in /opt but that was not the case. I guess this is another Alpha related bug.

Snitter didn’t have this issue and remember my password correctly. It also had the same border issue but I already expected that:

Snitter exposed a different AIR bug to me. Whenever the Snitter window lost focus (for example, when you switched desktops, or brought another window to the foreground) it would simply disappear. The process was still running but the window would simply vanish. Killing the process and running the app again solved the process but it was not efficient enough. I’m suspecting it was trying to “minimize to tray” but didn’t know how to integrate with the KDE tray I was using.

So while both applications are perfectly functional, the seem to be using AIR features which are still missing from the Linux implementation. This makes them to annoying to be of use to me on a daily basis. I guess I’ll have to wait for the fully featured version until I can use one of them. In the meantime I found out an app which actually works semi-reliably on linux. It’s Thwirl:

It doesn’t work perfectly though. You need to disable the “hide on minimize” option or it will vanish when you minimize it. Remember that AIR has no clue how to put apps in the KDE tray. This may actually work with Gnome – I don’t know. I haven’t tried it. The second issue is that the popup notifications are a bit wonky. If you get more than one, they sometimes end up one on top of the other. The fade effect totally doesn’t work for them either – they just start blinking and jumping around. I’ll need to see if it can be disabled. Either way, Twhirl is a winner on Linux. :P

As a side note, I would love to know if you are currently using any other Air apps right now. Especially if it they are not related to twitter in any way. I’m wondering where this platform is going. :)

Btw, iVenus – you wanted some screenshots so here they are. :)

DomParser parser = new DOMParser();
parser.parse(new InputSource(xmlStream));
Document d = parser.getDocument();

Pretty straightforward stuff – in fact, you probably find the same few lines in just about every single DOMParser tutorial out there. The xmlStream is an InputStream instance object with the XML data. Where do I get it from? I pull it off the Twitter as I described here. I tested this code before, and got the XML to print out in the console so my InputStream is not the issue here. Every time I called the parse method I got few dozen errors like this:

s4s-elt-character: Non-whitespace characters are not allowed in schema elements other than ‘xs:appinfo’ and ‘xs:documentation’

It was basically one error per each node which had text data as a child. I was googling this message for hours and it seems that no one has a clue what causes it. I’m definitely not the first person who got it, but I have yet to see a working solution.

In the end I decided to abandon DomParser. There is about a bazillion different ways to parse XML files in Java so I simply switched to the JAXP parser (javax.xml.parsers). Now my code looks like this:

DocumentBuilder builder = DocumentBuilderFactory.newInstance().newDocumentBuilder();
Document d = builder.parse(xmlStream);

Both snippets are essentially equivalent and achieve the same thing. So as far as I’m concerned DocumentBuilder > DOMParser. Still, if anyone has a clue what is that s4s-elt-character error all about, please leave a note in the comments so that future generations do not have to suffer because of it.

[tags]java, java xml parsing, DOMParser, DocumentBuilder, JAXP, Xerces, Apache, programming, XML[/tags]

Either that, or an internal server error. I have been twittering less and less in the last month or two – but at this point I hardly use it anymore. Every time I get around to post something it is down. I have an account on Jaiku that I hardly ever use. Maybe I should switch to that?

Anyone knows what the hell is going on with twitter lately? I know they were getting hammered hard with traffic, but for some time there seemed like they got the hand of it, and worked out how to load balance and cache everything.

I’m really interested in Twitter because it is a great case study in using Ruby to handle a high traffic social network. It seems that they are slipping now – I’m curious to see if this is because of problems scaling up Ruby on Rails, or is it something completely unrelated to the language and platform of their choice.

[tags]twitter, twitter down, twitter downtime, ruby, ruby on rails, rails, twitter ruby[/tags]

Let’s assume that we are trying to post an update to twitter with the following data:

String username = "someone@example.com";
String password = "password";
String status = "Look! I'm twittering using Java";

Twitter uses basic HTTP authentication which requires you to send a username:password pair encoded in base64:

String cridentials = new sun.misc.BASE64Encoder().encode((username + ":" + password).getBytes());

Now let’s create a HTTP connection:

URL url = new URL("http://twitter.com/statuses/update.xml");
URLConnection conn = url.openConnection();
conn.setDoOutput(true);

Authentication is activated like so:

conn.setRequestProperty ("Authorization", "Basic " + cridentials);

Now we can send the update by simply writing to conn’s output stream:

OutputStreamWriter wr = new OutputStreamWriter(conn.getOutputStream());
wr.write(data);
wr.flush();
wr.close();

Finally, if you wand you can grab and print out the XML response sent back to you by Twitter server:

BufferedReader rd = new BufferedReader(new InputStreamReader(conn.getInputStream()));
String line, output = "";
while ((line = rd.readLine()) != null)
{
System.out.println(output);
}

That’s it – that’s all you need to do.

[tags]twitter, java, jtwitt, java twitter client, twitter client, twiter updates[/tags]

It seems that their issue is not the execution speed, but the Rails design itself. In a recent interview a Twitter developer Alex Payne said:

Emphasis was mine. The lack of support for multiple databases seems a huge bottleneck issue for high traffic sites. Most of us don’t even think about stuff like that when considering using a platform to build our applications with.

I still think RoR is a good platform. It just might not be ready for the enterprise scale apps like Twitter. Most of us however will never experience this issue – unless of course we get slashdotted or something. I’m confident that the Rails team will use the Twitter problems to improve scalability features in future releases.

[tags]ruby, ruby on rails, twitter, bottleneck[/tags]