Every once in a while some of you complain about restrictive spam control here. Sometimes comments get blocked because they have too much links. Not so long ago, quite a few people got blocked just because they were behind a proxy. These are unfortunate glitches and I try to work around them and massage the spam tools to be nicer to people. They are effective, but they lack the much needed people skills… And intelligence. But we are working on that.

In the meantime I wanted to show you this graph, that illustrates the ration of spam comments to non-spam comments on this blog.

Spam vs Ham on Terminally Incoherent

I didn’t just make up this chart. It came out of my Akismet panel based on the data it collected. Since December 96.73% of comments posted to this blog were pure spam. Can you imagine that? Ninety seven fucking percent! It’s insane!

How many of these spam comments did you see?

None! They were all silently blocked and hidden away so neither you nor I have to deal with them.

Of course, you could say that this graph could be based on only few dozen comments. But it is not – let me post another graph to prove it:

Spam over time - note the recent spikes

All in all, I think I’m averaging few hundred to a thousand spam comments each month. Some months seem to be worse than others. For example May and begging of June seem to be particularly bad. I’m not sure if this is just a local fluctuation or an increasing trend. Either way the amount of comments collected over time suggests that the 97% is probably not skewed by a small sample size.

I guess this comes with the territory. Terminally Incoherent seems to be one of the small blogs that are popular enough to get spammed, but not popular enough for the big comment threads to offset the spam-to-ham ratio. Not that I’m complaining.

I’m actually thrilled that I managed to build this small community. I love the fact that I seem to have gained few regular readers who stop by frequently and post insightful comments. I’m also always amazed at the high quality of the discussions we have here. Funny thing is that only time the comment threads seem to degrade is when I get dugg or reddit-ed and we have a temporary influx of new readers.

Anyways, I just wanted to thank you guys for making the delicious ham comments we have here. You make running this blog worth while. And if my spam filtering friends seem annoying sometimes, give them a break. They are doing a great job keeping out all the crap from our comment sections. Without them, spam would drown out any legitimate comments in an endless torrent of unsolicited advertisements.

I personally hate that one. Yes, you can sort of figure it out but you actually have to put some effort into it, and sometimes it’s just pure guesswork. Does it help against the automated scripts? I don’t know – I guess this is a question we should direct at Rapidshare. But it sure is annoying to regular users.

The OCR technology is not there yet – it’s getting better, but I presume that we could still get few years out of our CAPTCHA’s if their effectiveness boiled down to complexity of design vs. character recognition arms race. But we all know there is a growing cottage industry out there which uses real people to solve CAPTCHA’s by either tricking them into doing it or paying them per solved puzzle. I always imagined this to be rather shady business conducted in private spammer forums and via private channels. But it is not. They are actually doing this out in the open, as a legitimate paid service:

Here is a screenshot of imagetotext.com – a company which specializes in solving CAPTCHAS. They of course don’t say it like that, but I think the blurbs on their site make it pretty clear that they are not really interested in doing any sort of data entry tasks or into transcribing free hand text into digital format. They are interested in receiving a small image, and shooting back the text at $.02 a pop bought in “packages” of 500 images or more. With a narrow focus like that, what else could they be doing?

Note that I’m not linking to them, because sure as hell they don’t need any Google juice from me. :P The ubiquity of CAPTCHA basically created a new niche industry. All you need now is some clever script that will harvest CAPTCHAS, send them to Image to Text, receive responses and create accounts on popular online services. Thank god these sort of scripts are shady, and probably hard to get, right? You either have to make them yourself, or know where to find them, or who to ask for them. It’s not like anyone can just go to a website and buy, for example, an automated Myspace account creator? Right?

This one is from allbots.info – a website that seems to be selling precisely that: account generation scripts that create random profiles, and simply need a human being solving CAPTCHA’s really fast for them. So you buy one of these apps, then purchase a big ass package with ImageToText you can start building your brand new spam empire. All it takes is some cash – you can even be borderline retarded. It won’t slow you down.

Combine the two services, and you have yourself a deadly combo with no programing, and no thinking required. A bit scary if you think about it. I’m not sure how profitable are these two companies, but the fact that they exist indicates that there is demand for these type of services out there.

CAPTCHA’s may be effective in stopping your average home grown spammer, but they are actually creating a whole micro-industry revolving around circumventing them. In other words, they are actually performing natural selection – weeding out the week players with few resources, and leaving only the biggest, baddest and most determined in the game. They are the catalyst, helping to evolve bigger and better bad guys.

Public Turing tests may be doomed and I suspect they might get completely phased out from use on the web in next 5-10 years. And it’s not just CAPTCHA’s – all public Turing tests. After all, it doesn’t matter if you are interpreting an image, solving an equation, or answering a question – it doesn’t really matter if there is a low wage human worker solving it on the other end, and then handing control over to a script.

Google has an interesting idea going on with their text message based application. If you haven’t seen it, try signing up for one of their services such as Gmail or Google App Engine. Instead of using a CAPTCHA they send a text message with an activation code to your cell phone. At least for the time being this system remains much harder to game – which means we might see it being used more and more often by popular online services. Of course it does have serious downsides as not everyone with an internet connection may have a cell phone (think less developed countries) and not all cell carriers may be supported. We will need something else – but what?

It will be interesting to observe where will the anti-bot technology will go in the next few years.

[tags]captcha, imagetotext, allbots, rapidshare, public turing test[/tags]

Great post! I really enjoyed it!

There are many variations, but the message is always the same – a generic praise that does not contain any references to actual content of the post. Some of these have URL’s attached to them, and some others don’t. Most of them end up in the moderation queue or Akismet spam box.

I do realize that some people might just want to tell me I did a great job, but have nothing else to add. Still, 99.9% of these posts get deleted whether or not they contain a spammy URL.

I do this for several reasons:

1. Even if the post does not contain a URL, it doesn’t mean it’s not a probe to cheat the first time poster moderation rule (when you post here for the first time, your post is held for moderation – after that, it’s up to Akismet and BadBehavior to keep you out). These things are likely to be followed by regular spam once the email in the email field gets white listed
2. Even if it’s not spam, the value of such comment is very low. It does not contribute anything to the ongoing discussion. It’s generic – almost like a trackbac, but these at least let me know who links to this post and let me visit the blogs of people who read me. I found lots of cool blogs via trackbacks, or by looking at who links to me on technorati
3. If you think about it, the only people who would genuinely post a generic message like this would usually not be regular readers. Most probably they would be random visitors who stumbled upon a post via Google. So chances are that if I mistakenly delete their post, taking it for a bot spam, they won’t get upset because they were never planning on coming back to this site anyway.

Of course this is not to say that I don’t appreciate kind words from random visitors. I do appreciate them very much, but at times it’s better to err on the side of caution. So if you posted one of those generic “Good job!” or “Nice Post” comments and it got deleted, I apologize. I don’t delete them automatically – I usually look at each one individually to see if it maybe it is less generic and off topic than I initially suspected. Unfortunately, most are not.

Do you get these types of comments on your blog? Do you think these are bots, malicious people or just some poor lost souls who leave super-genetic comments?

[tags]comments, one liners, great post, ambiguous comments, spam, spamming[/tags]

Let me interject here for a second. How hard is it to register a company in US? Can’t be that hard, can it? I mean, think about it: if companies like allofmp3.com were able to sell to US customers despite shaky legality of their operation, then why would it be so hard for a hand cream company to do the same? Oh, yeah – cause it’s a scam. I forgot. Carry on.

Wait… Out rightly? I think you are looking for “outright” but that would still not make much sense.

Sounds good eh? All you have to do is to cash some checks and wire the money overseas few times a week and you get a hefty percentage. What could go wrong? Other than spending some quality time in the federal-pound-me-in-the-ass-prison that is. Yes, boys and girls – if you take one of these “part time jobs” you are essentially laundering dirty money on behalf of your “employer”.

The checks and money orders you cash are usually earned via fraudulent Ebay auctions, illicit transfers and etc. When the person who was defrauded catches on and calls the police, the money trail will lead them directly to you. And guess who is going to get stuck with the bill?

Do not respond to these emails, and most importantly do not agree to transfer anything for anyone. In the best case scenario, you will get scammed in a classic 419 way. In the worst case, you will become an accessory to a crime and you may not only end up in debt but also suffer legal repercussions.

Some further reading:

• Payment Transfer Job explained in detail at Hoax-Slayer
• How to avoid Online Job Scams tips for job seekers.
• List of known Job Offer Spam emails – and some tips on how to avoid them
• Good blog post on this topic by a security consultant Debra Radcliff

If you got implicated in one of those things, stop everything call your Lawyer, and then call your Bank and the Police. Note that ignorance is hardly ever an excuse in our legal system, so you will likely be liable for all the transfers you already made.

This has been a public service announcement for the benefit of mentally handicapped people who fall for these scams.

[tags]email scams, job offer scam, 419 scam, spam, scamers, money transfer scam[/tags]

Vaccination of insurance ratesare infect millio resistance against substance. Above the to economic valine can affect valganciclovir over. Moratalla had not received valcyte into account withdrawn. Appropriate specimen had had vagistat-1 to increased vagifem attitudes toward accrue.

Imaging the detection of uvadex sequelae of ursodiol traveling public urso gene. Evidence of or greater something not countries women analysis. Room of wash hands and jurors for frontline ancestry. All individuals specimen obtained ca outbreak uroxatral temporary injury vanex compounds. Global response fiestas or after exposure of cultures vancomycin loading.

Recent data formerly the vancocin trends in vanceril the identity daily. California doctors with other no published valtrex force an valtran above. The end earners are valstar year on suspends the valsartan azine. This does was higher valproic acid and although insurance rate practice.

It almost has some sort of rythm and a semi-medical theme to it. I also appreciate the alliteration – almost reminiscent of the now infamous V’s Alliteration Speech. When you read it, you can almost sense some sort of purpose or leading thought. You can almost grasp it – but it always fades away, and escapes understanding.

Almost an emergent behavior. I do not believe that the internet will one day awake into sentience. That thought seems a bit far fetched – at least for now. It is kind of entertaining to think that one fateful day, we will all get a piece of spam written in this gibberish sort of way. But this time if you read it carefully the message will become clear: “I have awakened!”

On that day, I will know what to do. For one, I will welcome our new electronic, web based overlord. :mrgreen:

[tags]spam, spam poetry, akismet, emergence, sentience[/tags]

1. Turing Tests
2. Reverse-Turing Tests (or robot detection)
3. Spammer Annoyances
4. Filters
5. Blacklists/Whitelists
6. Trackback Validation

Each category has benefits and flaws and none of them can guarantee you that no spam will slip through. Below I will discuss all these categories. I will link to different WordPress plugins, because this is what I use. Feel free to chime in with solutions for different blogging platforms in the comments.

The most commonly used, and perhaps the most controversial method of combating blog spam is the good old Turing Test. It is usually a challenge which can be easily and effortlessly solved by humans, but is difficult for machines. Best example here is CAPTCHA. Usually CAPTCHAS are those little blurry images which contain words or numbers that you are supposed to type into a box to post. I’m using one on this very blog – so if you scroll down to the comment box you will see a perfect example. CAPTCHA’s work because OCR technology is not perfect, and funky fonts and small distortions can easily full all character recognition algorithms.

Unfortunately CAPTCHA has some downsides. It can be annoying to the users, especially if it is illegible. It is also a usability issue. Blind people, or people with bad sight who rely on screen readers can’t solve CAPTCHAS. Thus, if you use one, you might be permanently denying a segment of your readership ability to comment. Of course some CAPTCHAS use an audio challenge. I even seen CAPTCHAS which ask users to solve little math or contextual problems by filling in the blank word, or calculating a result of a simple equation. Unfortunately graphical CAPTCHA is a de-facto standard because it is easy to implement, easy to solve, and most effective.

Still, they are not perfect. There are ways around CAPTCHA’s – you can use different session hijacking tricks, you can use cutting edge OCR algorithms to break them, or you can simply harness the collective stupidity of MySpace users to solve them for you.

Reverse Turing Tests take the opposite approach. Instead of making the poster prove that they are human, they try to detect that the post is being made by a robot. There are many approaches to do this. For example, the Bad Behavior wordpress plugin analyzes the user agent string, and different behaviors to identify non-human posters.

Personally I love Bad Behavior but it appears that on some high traffic sites the plugin can behave erratically. For example, Shamus who runs the hilarious DM of the Rings comic, had major problems with human commenters being blocked for no reason. He eventually stopped using the plugin for that reason. So false positives are a big issue here.

Next approach is something that I call Spammer Annoyance. The idea is simple – make the life of a spammer more difficult and he will move on to an easier target. CAPTCHA already fulfills that function – it is a barrier that prevents the bottom feeders to user their crappy scripts to post hundreds of comments per second. But since CAPTCHA’s are annoying people search for different solutions.

One good approach is to require a hash of the posts text being submitted along with the post. The HashCash plugin is a good example of this method. Why does it work? Because to post, the spammer will have to include the hashing algorithm in his script. Since hashing is resource intensive, the resulting script will work slower than normal, making the spamming less efficient. Most people don’t care enough to bother with these things – they simply move on to easier targets.

Unfortunately hashing must be done on the client side (doing it on server side would make no sense – you want to annoy the spammer, not increase your server load). How do you compute anything on client side? You use Javascript. And that in itself is another usability issue. Not everyone uses Javascript – and your hashing algorithm will not work with text browsers. You may not care about stuff like that, but some people do.

Of course there are other, simpler ways to annoy spammers. For example, Travis proposes to rename the fields in your comment form with some random strings (maybe randomly generated ones? what do you think Travis?). Of course a well constructed script would simply crawl the DOM and grab appropriate fields, no matter what they are called. But most spamming shmucks out there simply use dirty hacks done with WWW:Mechanize and grab fields by name, using standard naming conventions.

Filters – are classic spam fighting tools that are more or less successfully applied to combat email spam out there. They utilize Bayesian filtering, pattern recognition and many other tricks to recognize spam, and prevent it from being posted. Personally I like Akismet which uses a centralized approach. Every user of this plugin submits their confirmed spams to the central database – so the algorithm is simultaneously trained by spam collected over hundreds if not thousands of different blogs.

If you don’t like to rely on some centralized 3rd party database for your filtering you can use something like Spam Karma 2 which is localized, and trains only on your spam.

The obvious problem with filters is that they usually tend to produce false positives every once in a while. For example, almost every recent comment made by my friend Miloš has been eaten by Akismet – completely without rhyme or reason. He thinks it had something to do with my old CAPTCHA plugin, but I doubt it. I think it was just Akismet being weird.

Whatever plugin or algorithm you use, you will have false positives, so you need to monitor all the spam comments and keep them in some sort of moderation queue. Unfortunately if you get 200 spam posts per day, you are likely to dump them all without looking and simply hope that there were no false positives there. This is a big issue.

Because of the spam issues some people require registration to post comments. This is a form of a whitelist. Only certain group of users is allowed to post. Personally I don’t like this approach because registration – especially one with email confirmation is annoying. I would say it’s about 100 times more annoying than the worst CAPTCHA imaginable – and I usually don’t bother leaving comments in “members only” threads.

I do use blacklists in a limited way. Whenever I see large volumes of spam coming from the same IP I simply block it in the .httaccess file. It prevents that IP address from ever accessing my website, so I use this sparingly and only for the most notorious spammers. Another good approach is to block all open proxies as they are frequently used by spamming scripts. I don’t do this, because I want to allow my users to read this blog and post comments anonymously, but not everyone cares about that.

Finally, there is Trackback Validation. Some spammers aim to bypass captchas, annoyances and most other methods described here by avoiding the comment form altogether. They simply generate false pingbacks to create trackbacks to your blog. Since all the pingbacks are associated with a linking URL a simple way to prevent getting hit by mass-generated spam is to verify that this URL actually exists. I use the Simple Trackback Validation Plugin to do this.

Of course this doesn’t work if the spammer is sending the pingbacks using an actual blog created on blogger or wordpress or whatever. But, once again, it counts down on the amount of crap posted to your blog by the bottom-feeders with shitty perl scripts and no knowledge or cunning.

Personally I use a layered approach:

1. My first line of defense is a CAPTCHA (Peter’s Custom Anti-Spam)
2. I employ Simple Trackback Validation as a CAPTCHA like barrier for pingback spam
3. I use Bad Behavior to test for robots
4. Whatever slips through all the plugins above is usually scooped by Akismet.

This multi-tiered approach works great for me. The CAPTCHA and validation scripts combined with Bad Behavior mean that my Akismet queue is almost always empty. But when something does slip through my front line defenses, I can always rely on the collective wisdom of half the internet to capture it.

What do you use? Any suggestions for alternative tools to use for Movable Type, Text Pad and whatever else is out there?

[tags]blogging, blog spam, akismet, spam karma, simple trackback validation, wordpress, captcha, hashcash[/tags]

If I get hammered with spam, I will simply re-enable it. If I don’t, I might just keep it off for good.

Also, if ever got weird error messages when posting here, I apologize. I think these errors might have been triggered by the code in the CAPTCHA/comment preview plugin. Or not. This might help me track down the source of these issues.

What do you think? Is this a good idea, or am I in for a world of pain?

Anyway, this one cracked me up:

Dear Friend,I am Mr Ming Yang of Hang seng bank,hong kong.i have a buisness proposal for you,of ($24,500,000.00) if interested get back to me vie this email (ming_yangceo@yahoo.com.hk)

A 149 scam in 28 words – this guy didn’t waste his time. I really don’t think you can get this any shorter. Maybe he could replace get back to me vie this email with reply to but that’s about it. Heh… That’s what I call succinct and to the point scam bait. No sappy story about dying husband with cancer, no long winded lottery winning notification – just business. :mrgreen:

[tags]419, 419 scam, email scam, spam, mr ming yang, hang seng bank[/tags]

All of thes accounts used an absolutely positioned div to block out the actual content and display the following text in the middle of the page:

This profile contains adult content.
CLICK HERE to install MS Viewer.

When you click on the link different things happen. Firs account made me download the following file:

MSpaceContentInstall.exe (md5: 81ec383d21a753df6b5e54ef48aea437)

I ran strings on the binary and got some interesting results. Following DLL’s are listed by name in the file:

KERNEL32.DLL
ADVAPI32.dll
COMCTL32.dll
GDI32.dll
ole32.dll
SHELL32.dll
USER32.dll
VERSION.dll

Whatever that thing is, it is messing with the kernel and bunch of integral system libraries and that can’t be good. I ran a virus scan on it, but stupid clamav decided the file was clean. So did Kaspersky online file scanner. On the other hand the Virus.org file scanner (which ran the file through assortment of different engines) gave me much more interesting output:

File: MSpaceContentInstall.exe
SHA-1 Digest: 3a954d79eafe7c0a53bf0ff456218edef23eda96
Packers: Unknown
Status: Infected or Malware
--
ArcaVir Clean
avast! Clean
AVG Anti Virus Clean
BitDefender Generic.Zlob.70EF425F
CAT QuickHeal Clean
ClamAV Clean
Dr. Web Clean
F-PROT Clean
H+BEDV AntiVir DR/Zlob.Gen
Ikarus PSCAN Clean
McAfee Clean
NOD32 Win32/TrojanDownloader.Zlob.ARX trojan
Norman Control Zlob.ACUZ
Panda Clean
Sophos Sweep Clean
Trend Micro Clean
VBA32 MalwareScope.Downloader.Zlob.1
VirusBuster 05 Clean

Only 5 out of 18 AV tools labeled that file as the Zlob Trojan. I had a similar result when using Jotti Online Scanner as well: 4 our of 15 engines labeled it as malicious. It’s surprising how the popular and well respected scanners out there (McAfee, AVG, Avast, TrendMicro, ClamAV) failed to detect anything, while more obscure ones did. I never even heard of H+BEDV AntiVir or Norman Virus Control. :P

The other page linked from one of those profiles was even more interesting:

It’s an automated CAPTCHA solver using human input. It is much more efficient than fucking around with OCR. Instead of wasting CPU cycles running complex feature extraction algorithms, you can just use the hordes of horny idiots who want to see some MySpace n00dz. I knew these things existed but this is the first time I actually saw one of them in action. Very interesting setup.

If you look closely at the image the screenshot, you will see that it actually comes from MySpace registration form. Ot at least that’s what it looks like. So the codes that you inptut at the prompt are used to create new accounts that could be used for more spamming. It’s quite brilliant actually – you set it up once, and given enough traffic this thing could run forever. Deleting these accounts won’t help – since you will most likely get 3-4 new accounts from each “hit” (ie. an average dumbass who falls for this scam will likely try to type in the CAPTCHA 3-4 times before he gets bored, or realizes he is being fucked with).

This is a serious design flaw. When you alow your users to inject HTML and CSS into your page, you are asking for trouble. Stripping javascript, iframes and such will not help because these guys are not using any. They are using the standard mundane tricks that everyone else abuses to “beautify” their profiles.

Note how Facebook does not have this problem. This is the difference between good design, and an ugly hack from hell.

[tags]myspace, myspace spam, myspace mallware, MSpaceContentInstall.exe, captcha, zlob, trojan[/tags]

Of course the poor English grammar and spelling is intact, even in this super condensed version of the 419.

[tags]spam, scam, 419, niggerian, mou xinsheng[/tags]