Terminally Incoherent

You might remember that like a 100 years ago I posted a small snippet of code showing you how to post Twitter updates using nothing but curl. Well, I took that snippet of code, polished it up a bit and re-branded it as Twimi and posted it on Google Code under GPL. I know, I know - the script proper is like 3 lines long and it doesn’t really do anything other than calling curl. I’m pretty sure you could whip up a way better script like 5 minutes. Why the hell would I put up something so trivial up there?

Well, for one I noticed that other people did the same thing. Both ktwitter and ZenTwitter are similarly simple, and yet they show up as “Linux Apps” on the Twitter Fan Wiki. Both scripts seem no more complex than what I wrote so why the hell not call my script an “twitter client app”?

The thing is that if you have a twitter client project which has it’s own website you can request a custom source parameter for it. So you can sort of brand it. Previously every time I posted using my script, the twitt showed up as “from web”. Now, it says “from twimi” and links to the google code page for the project which is a nice touch. So I pretty much uploaded it up there just to get that custom link underneath my twitts every once in a while.

The second reason for posting it there is perhaps some sort of small glimmer of hope that some shell scripting guru will be willing to contribute to it and add new cool features to it. Right now it is very simple an minimalistic. It is a starter script that you can take and modify to make your own curl based client. I’m not going to post the code here because you can just easily browse it on the project page. It’s only about 60 lines of code, not counting the comments, and GPL boilerplate. If you want to contribute let me know and I’ll add you to the project and you can hack away at it and perhaps the thing will grow on it’s own. I don’t really care for the direction we take it in. I don’t care if it grows a front end of some sort, or if you decide to toss curl in favor of something else.

I know this is not a real project. But, everyone knows a little bash, right? And everyone can write a better script than this in 5 minutes or less, right? So just do it!

Suggestions, comments and contributions are as always appreciated.

I love Twitter, but ever since I started using it I felt that the way it implemented the reply system was a bit lacking. On one hand they made it very simple to reply to people. Simply start your message with @name and it will automatically link it to the most recent tweet of that person and then put it on their “replies” page. There is a problem here which is probably best illustrated by this conversation:

As you can see, we have a nice 3 way, asynchronous conversation going on here. Now, how hard do you think it was for me to generate the image above and make it thread like that? Can it be done automatically? Nope. The way I did this was to fish out and screen-cap the particular messages one by one. Also, they were not all in one place. I had to go and grab some of them from Miloš’s and Billy’s twitter pages.

Quotably is a valiant attempt to automatically generate stuff like the manually constructed conversation above. Unfortunately due to the fact that Twitter’s threading model is broken by design it doesn’t always work. For example, here is what it caught from this conversation:

As you can see, it’s almost there. It just that a lot of the tweets simply got dropped for various reasons. For example, using the @name notation in your message more than once will confuse the system and it won’t link to anything. The other issue was that at one point Miloš and Billy were talking to each other and while I still saw their conversation happening on my updates page , and it was still regarding the same topic page, Quotably wouldn’t associate it with my name and include them in this thread.

Billy is right, putting the StatusID value in the message itself is probably a bad idea as it would quickly start eating away our precious 140 characters. But Twitter already does some behind the scenes magic, when it links your message to the top tweet of the person you replied to. So in theory you should be able to re-construct the whole thread by simply clicking on the “in reply to” links. This hardly ever works though. Most conversations are asynchronous so by the time you replied to something, the author of that tweet might have posted 3 or 4 new messages and yours gets linked to the most recent one.

There really ought to be a way to link to a message rather than to a person. We are already halfway there - each tweet has a reply button associated with it. Now all we need to do is to make these buttons do something - and preferably without losing any characters in the message area.

Before that happens though, we need a better threading app. Is it possible to do it with the current Twitter setup? Perhaps, but we can’t really really rely on the in-reply-to implementation currently in place. This is what Quotably is doing right now, and it is not really working for conversations such as the one above. So how do we thread messages?

Perhaps we could try to reconstruct a conversation by simply doing a full text search on the @name pattern and then arrange tweets chronologically. So a user could select 2-3 user names, and then have the Quotably like service fetch all their status updates in which either of the chosen users includes an @name pattern of any of the other two. No indentation, or other thread structure would be necessary here, because it is pretty much infer it from the data at hand. But if we list them chronologically we at least capture them the way they originally appeared in your Twitter stream.

It’s not perfect but it could be a nice complement to a service like Quotably. I do not purpose using it as the main threading system, but rather as an alternative which could help users track more complex, many sided conversations between a group of users.

I’m posting this a day late because it took me a whole morning to figure this one out. It appears that Comcast has completely blocked both inbound and outbound traffic on port 25 for my company. For a few years now we have been running a in-house authenticated SMTP server using IIS. It was running on port 587 and basically relayed emails to another server at an off-site location on port 25. Why was it set up this way is a topic for a whole other rant, but it worked well for us until now.

When I came in to work yesterday morning all was well. Few people around the office mentioned something about a slow day, noting their inbox was unusually empty. I didn’t really pay much attention to that chatter, until someone decided to email something to herself and it never came through. Then all hell broke loose.

You see, this problem was essentially hidden from regular users because they could connect to my in-house server on port 587 without any issues. So their emails were leaving their outbox as normal, and then queuing up on the server, never to be seen again. The server itself could not shoot them back failure notifications either, because it could not connect to it’s relay point on port 25. So people were emailing each other all morning without even realizing something was amiss. When they caught on, it was instant panic mode spreading throughout the organization like a fucking wildfire.

For several hours I was methodically checking, re-checking, restarting, and power-cycling every single device and service that had anything to do with email traffic. I was also calling the folks who maintain the off-site server every 5 minutes to see what was their progress. They were convinced the issue was on my side, and I was adamant that it was on their side. After much deliberation, we came to the conclusion that we were both wrong. The off-site server was accessible from everywhere but my location, but there was nothing here in the office which would prevent it from communicating on port 25.

We tested outbound and inbound traffic on their side and it was working just fine so that left only one conclusion - my ISP fucked us over and completely sealed off port 25. Once we realized that, the conclusion was as swift as it was simple. We simply switched the external server to listen on port 587, changed the outbound port in IIS and an avalanche of backed up email started streaming into people’s mailboxes.

Let me run that by you again in case you didn’t notice - once we figured out what the issue was, it took us 5 seconds to reconfigure our shit, and route around it. So if this supposed anti-spam measure is so easy to circumvent, then can someone explain to me how is it supposed to be stopping hard core spammers with their sprawling botnets out there? I’m pretty sure most of semi-modern spam-trojans can be remotely reconfigured to send out emails on alternate ports.

Port blocking has became pretty much an industry standard these days, but I still fail to see how it could ever be effective. What is stopping me from running an email server on port 80 or 443? Will they block these two ports as well? It is just a knee jerk reaction, that might be effective in a short term. It won’t work in the long run though - soon they will run out of ports to block, and regular customers won’t be able to use any kind of non-standard internet services for genuine purposes without bending over backwards.

This is just one of these wholesale, one-click-and-your-done spam solutions. Why do ISP’s do it? Because it’s easy! You block some important ports, and the amount of spam and genuine email routed through your network goes down. You boss is happy, your investors are happy, folks in the security business are clapping their hands marveling at the sudden drop in spam, forgetting it will be back to normal in a month or two as all the spammers will figure out the same thing I did just now.

The only people who are not happy about this are the customers, but Comcast does not really care about them that much anyway as it has blatantly demonstrated in the past with it’s bandwidth throttling, and lackluster tech support.

Also, Twitter > than regular tech support resources it seems:

Despite the fact that we were constrained to 140 characters per pop, talking asynchronously and multitasking, this was still way more pleasant than my experiences with Mr. Rooter and Mr. 125 Times. Not sure if that guy is an actually really affiliated with the company in any way, but he seems to be representing them well in the 140 character conversation universe.

One more reason to love Twitter and hate Comcast!

At the beginning of the month, Adobe finally had a sudden bowel movement and defecated an AIR release for Linux all over the internets. Naturally it is an alpha release, which means some things may still be very, very broken.

If you don’t know what AIR is, chances are you also don’t know about Twitter, which means you have been living under a rock for the last 900 years. I’ll explain it anyway. AIR was designed to be a platform for lightweight desktop applications, and has been essentially hijacked by Twitter enthusiasts. The most notable applications running on this platform right now are Twitter clients such as Snitter or Spaz. I’m sure it has some other uses, but I don’t know them. The only AIR apps I have seen so far are Twitter clients, and clients for other Twitter like services. If you know some interesting non-twitter related AIR apps I would love to see some links.

I complained about this before but the selection of Twitter clients on Linux is piss poor. Go to Twitter Fan and just look at the linux section. We have two emacs based clients, a vim based client, one written in bash, one which is essentially curl+zenity, a gnome desktop widget, and two or 3 more clients which just wouldn’t work properly on my machine due to bugs or dependency issues. If you scroll up and down you will see that Windows and Mac have all these nice, shiny graphical clients rich in features.

I installed Linux version of AIR to see if some of the nice AIR twitter clients will work on Linux the same way they work on Windows. The answer is - sort of. I installed both of the clients mentioned above: Snitter and Spaz. Nice thing about Air is that the installation looks and feels exactly the same as on windows. So you get a very pretty graphical installer. I’m using Kubuntu (Gutsy) so the apps got installed in /opt which frankly is where I’d put this type of stuff given a choice. The installer also put shortcuts into my KDE menu, although they used the full package name which contained a very long checksum of some sort. Still, I’m not complaining - it works as expected.

It took me a little while to figure out how to uninstall AIR apps until I realized that they are actual deb packages. So you can easily remove the apps you don’t need using aptitude which is great news. Too bad they don’t let you know about this when you are installing apps. I had to go to their Linux FAQ to find out about it.

The apps are a slightly different story. You can look for yourself. Here is how Spaz looked for me:

You can see a little blue background on this shot - this is not because I did a sloppy cropping job, but because I wanted to show you that the drop-shadow transparency that some of these apps use is not working here. On windows the edges are nicely rounded and have that faded shadow effect. On linux you get the ugly opaque black border. I guess this is one of these features which was not implemented in in the Alpha release.

Spaz also had major problems remembering my username/password. It would keep it in memory for the duration of my session, but when I closed and reopened the app it prompted me to type them in again. Initially I thought it was because Spaz was trying to write some config file in /opt but that was not the case. I guess this is another Alpha related bug.

Snitter didn’t have this issue and remember my password correctly. It also had the same border issue but I already expected that:

Snitter exposed a different AIR bug to me. Whenever the Snitter window lost focus (for example, when you switched desktops, or brought another window to the foreground) it would simply disappear. The process was still running but the window would simply vanish. Killing the process and running the app again solved the process but it was not efficient enough. I’m suspecting it was trying to “minimize to tray” but didn’t know how to integrate with the KDE tray I was using.

So while both applications are perfectly functional, the seem to be using AIR features which are still missing from the Linux implementation. This makes them to annoying to be of use to me on a daily basis. I guess I’ll have to wait for the fully featured version until I can use one of them. In the meantime I found out an app which actually works semi-reliably on linux. It’s Thwirl:

It doesn’t work perfectly though. You need to disable the “hide on minimize” option or it will vanish when you minimize it. Remember that AIR has no clue how to put apps in the KDE tray. This may actually work with Gnome - I don’t know. I haven’t tried it. The second issue is that the popup notifications are a bit wonky. If you get more than one, they sometimes end up one on top of the other. The fade effect totally doesn’t work for them either - they just start blinking and jumping around. I’ll need to see if it can be disabled. Either way, Twhirl is a winner on Linux.

As a side note, I would love to know if you are currently using any other Air apps right now. Especially if it they are not related to twitter in any way. I’m wondering where this platform is going.

Btw, iVenus - you wanted some screenshots so here they are.

I was messing around with the Apache Xerces based XML DOMParser class (from the com.sun.org.apache.xerces.internal.impl.xs.dom package)for the JTwitt project and I noticed some quirky behavior. I used the following snippet of code:

DomParser parser = new DOMParser();
parser.parse(new InputSource(xmlStream));
Document d = parser.getDocument();

Pretty straightforward stuff - in fact, you probably find the same few lines in just about every single DOMParser tutorial out there. The xmlStream is an InputStream instance object with the XML data. Where do I get it from? I pull it off the Twitter as I described here. I tested this code before, and got the XML to print out in the console so my InputStream is not the issue here. Every time I called the parse method I got few dozen errors like this:

s4s-elt-character: Non-whitespace characters are not allowed in schema elements other than ‘xs:appinfo’ and ‘xs:documentation’

It was basically one error per each node which had text data as a child. I was googling this message for hours and it seems that no one has a clue what causes it. I’m definitely not the first person who got it, but I have yet to see a working solution.

In the end I decided to abandon DomParser. There is about a bazillion different ways to parse XML files in Java so I simply switched to the JAXP parser (javax.xml.parsers). Now my code looks like this:

DocumentBuilder builder = DocumentBuilderFactory.newInstance().newDocumentBuilder();
Document d = builder.parse(xmlStream);

Both snippets are essentially equivalent and achieve the same thing. So as far as I’m concerned DocumentBuilder > DOMParser. Still, if anyone has a clue what is that s4s-elt-character error all about, please leave a note in the comments so that future generations do not have to suffer because of it.