The content, mind you was still there. All the posts and images were intact. They simply lost their category and tag associations. I have never actually seen anything like this before so my first thought was “database corruption”. I’m not sure how the DB could get corrupted, but WordPress is famously finicky about these sort of issues. It is not uncommon to see a badly written plugin touch one of the core WordPress tables in a bad way and make it freak out.

I promptly logged into the server and started running exploratory queries, but most of them came back looking very normal. The tag and category tables still had all of the entries in there, and posts were still correlated with them via foreign keys. The schemas of all the tables looked normal and I couldn’t detect any sign of plugin induced damage or even malicious tampering. All the information was in the database, but the UI refused to make the connections.

Luckily this is not the first time (and probably not the last time) I have seen WordPress go completely wonky. I have learned that running an active WordPress site without nightly backups is pretty much actively seeking out headaches. So after scratching my head for two hours, I decided to roll my VM back to the last night’s snapshot and see if that fixes the issue. Before that I decided to check how much disk space I have left.

It turned out that I had literally zero bytes.

Just on a lark I blew away the contents ~/temp and refreshed the site. The categories and tags have magically returned, but only partially. It appears that in order to render the tags and categories and associated pages WordPress needs to write a buch of temp files to disk. I have no clue why this happens, but I’m assuming it is an optimization strategy of some sort that is intended to limit the number of database reads per page view. However, if your disk is filled to the brim, it can’t do that. Therefore it fails silently and does the best to render the page without that additional information.

Apparently WordPress and/or MySQL need to write to disk when they do the queries that produce the category and tag list on your blog.

— Luke Maciak (@LukeMaciak) May 22, 2014

On one hand it is quite amazing that inability to perform some internal core caching does not bring the entire site down. On the other hand it seems wrong to me that such an operation is necessary. But despite using WordPress for many years now I have never actually felt compelled to peel the hood back and look at it’s database queries, so I guess I shouldn’t criticize something I don’t know all that much about.

Over the years I have gotten pretty good at cleaning up linux machines from accumulating temp file cruft. So it only took me a few minutes to identify the source of my disk bloat. It was the temp directory used by my WP-Cache plugin which ballooned up to few Gigabytes somehow. Apparently the old cache files were being discarded but never deleted. Blowing away the entire cache reduced by disk usage from 100% to 35%.

To prevent this sort of thing happening again I wrote a tiny guard-dog script that checks my disk usage on a weekly basis and prints out a nice report that is then emailed to me via a cron job:

#!/bin/bash

# use colors if available
[ -f "$HOME/scripts/colors" ] && source $HOME/scripts/colors

command -v awk >/dev/null 2>&1 || { echo "awk not found. Please install it and try again"; exit 1; }
command -v du >/dev/null 2>&1 || { echo "du not found. Please install it and try again"; exit 1; }
command -v df >/dev/null 2>&1 || { echo "df not found. Please install it and try again"; exit 1; }

# grab the % usage of the primary partition (typically line 2, col 5 on df)
read USAGE <<< $( df -h | awk 'FNR == 2 { print $5 }' )

# Make red if usage is above 60
if [ ${USAGE%?} -lt 60 ]; then
    Color_On=$Color_Green
else
    Color_On=$Color_Red
fi

echo -e "\nDisk Usage Report"
echo -e "-----------------\n"

echo -e "Disk usage: \t $Color_On$USAGE$Color_Off\n"

df -h

echo -e "\nLog file spot check:\n"

# Adding output to temp file so we can sort it later
# -sh provides human readable summary
# -BM sets the block size to Megabytes
du -shBM /tmp 2>/dev/null >> /tmp/du$$
du -shBM /var/log 2>/dev/null >> /tmp/du$$
du -shBM /srv/www/*/logs 2>/dev/null >> /tmp/du$$
du -shBM /srv/www/*/*/*/wp-content/cache 2>/dev/null >> /tmp/du$$
du -shBM /srv/www/*/*/*/wp-content/uploads 2>/dev/null >> /tmp/du$$

sort -nr /tmp/du$$

rm /tmp/du$$

The full version of the script is actually available here. The colors script I’m importing up top is also on Gighub if you want to check it out.

If you ever notice tags or categories vanishing from your blog, don’t panic. It probably just means your disk is full.

What was the reason for the downtime? I’m still not sure. Around 2pm Chris shot me an email notifying me the site was unresponsive. Fortunately, today was my day off so I could devote my full attention to the issue. Unfortunately at that point I was out and about running some errands and away from a computer. Still, being crafty technocrat that I am, I logged into my Linode app and I rebooted the VM. Because, hey – golden rule of troubleshooting: turn it on and off again.

That could have actually been what did the site in. When it came back, I was treated to the dreaded Error establishing a database connection message. This is a typical, WordPress catch-all message that can be caused by a host of underlying issues. So I started going down the list and addressing all of them.

First, I made sure MySQL was running and restarted it. This did absolutely nothing.

Next, I opened my wp-config.php and added the following magic line to the end:

define('WP_ALLOW_REPAIR', true);

This allows you to use the built-in repair routine by navigating to /wp-admin/maint/repair.php. For about a minute there it seemed like everything was going to be all right, until I got a lovely message claiming that wp_options table is marked as crashed and last repair failed. So built in repair was a dud.

I figured maybe the MySQL client will succeed where WordPress failed so I ran:

mysqlcheck --repair --all-databases

The results were about the same as above. The wp_options table was broken and beyond repair.

Unwilling to give up, I rolled out heavy artillery:

sudo myisamchk -r /var/lib/mysql/my_db/wp_options

If you have to run it with sudo you know it’s serious business. Unfortunately, this too has failed. This time I got even more meaningful error message: something like Error: not enough memory for blob at..

I kinda got preoccupied with the memory message and spent some time trying to go around it. I found out that you can increase the size of buffers that myisamchk uses for repair functions using command line switches like –sort_buffer_size so I tried things like:

myisamchk -o -f  /var/lib/mysql/my_db/wp_options –sort_buffer_size=4G

Eventually it occurred to me that wp_options should not have any blobs in it, and whatever data these tools are trying to load is not actual WordPress data but random binary garbage. On other words, chances were that the table was fucked beyond repair.

So after many hours of troubleshooting I gave up and restored my site from backup. As you may know my host is Linode, and they offer a really neat backup program. For like $5 per month they offer to take nightly snapshots of your VM. About a year ago I wisely added that feature because I was to lazy to set up backups on my own. I can now say with full confidence that it was worth every penny. I literally clicked one button, and my site was rolled back to last backup, which coincidentally was mere 7 hours ago. The restore took about 10 minutes, and as you can see the site is back in business.

Some data was lost in the process. While my latest blog post is up, the comments on it were lost, for which I apologize. Still, losing only 7 hours of data is actually nothing to scoff at. Actually, it’s only like 3-4 hours of real data, because there were no comments posted while the site was down, since noon. So all in all I think I made out ok.

Moral of the story: BACKUPS ARE GOOD! Seriously. Without backups I would still be down, and probably curled up in a fetal position under my desk, crying.

Also WordPress is a piece of shit. Oh, don’t get me wrong – it’s amazing when it works. In a way love it, I’m used to it, and I don’t want another platform. But I am always shocked and amazed how easy it is to completely wreck it by doing absolutely nothing. A silly DB error, and the entire site is out of business. If you use WordPress, backups are doubly important because of issues like this one.

If Valve has openings, I think I would totally fit into their corporate culture. I already operate on their time, and I love all their games. I’m a shoe in!

While I was tinkering with the design, I discovered something rather interesting. Well, interesting to me at least. Chances are I might be the last WordPress user in the world to discover this, but I will tell you anyway. Did you know that you can actually preview how your blog will look in a new theme without actually activating it? All you need to do is to add the following to any URL:

/?preview=1&template=themedir&stylesheet=themedir&

Where templatedir is the name of the directory in /wp-content/themes/. You can then click on any link, move around and see how everything looks. For example, here is how my blog used to look under the old theme.

The nice thing about this little feature is the fact that it allows you to mess around with the theme on your live installation without actually affecting how the site looks for visitors. There is just no need to install a test blog, you just use the preview URL and do your thing.

The changes are not just superficial. I finally fixed my tags/categories mess, which was driving me crazy for like 2 years now. You see, when I moved to wordpress, categories was a new thing to me. I started using them the way you use tags these days. I would assign multiple categories to each post, and create new ones for just about anything. So in the past you could have seen a post filed under “programming, software engineering, web design, web 2.0, php, scripting, lamp, mysql”. I ended up with like a quintillion categories on the sidebar, and since I had so many of them, there was really no consistency to how I was using them. When WordPress finally got a tag feature few years ago, I realized I could make things much, much neater and easier if I consolidated a lot of my categories, or converted them to tags.

The way I now understand the category/tag distinction is that you ought to have a few general categories (like programming, entertainment, humor) that indicate general topics you will be talking about. Tags on the other hand, are for specific, narrow topics that you write about frequently. So for example if I was writing a post about doing something in Java I would file it under the programming category and tag it with Java.

This is precisely what I did. I sat there, merged all my categories, converted them to tags (and vice versa) and deleted just about anything that had fewer than 3 posts under it. Right now I have 15 categories (all of which have close to, or over a hundred posts) and few dozen tags and I’m pretty content with this setup. I could probably consolidate few more categories, but this will do for now.

Part of this effort was just me trying to have a better organization, but there was something else that also motivated me. You see, I sort of want to have custom headers for each category. So if you are reading about programming there would be some code there, if you are reading a movie review there would be something movie related, and so on. This could only work if I had a manageable number of categories and would refrain from filing posts under multiple ones. So this is sort of where I’m heading.

Anyways, how do you like the new design? Do you like the category-based-header idea? Does anything seem broken after the theme update? Let me know in the comments.

Why? Because you are likely to be sending private data in plain text over that network. If you ever see me running an open wifi node you can safely assume I am probably sitting thee with a packet sniffer collecting people’s facebook passwords so that I can log in into their accounts and change their profile pictures to Goatse with a caption: “I AM A STEALER OF THE INTERNETS”.

Actually, scratch that – I wouldn’t do that because I’m a nice guy. It’s not in my nature to do things like that and – I can’t really hide it. You can tell whether or not someone is a good guy that by observing how many sexy ladies there are in his orbit at any given time. If you see a guy who has more orbiting bodies than Jupiter, you can tell the dude is an incredible duchebag and a horrible human being. So you should probably hang out with him in hopes of intercepting one of the outlying satellites.

Are these astronomical relationship jokes doing anything for you? No? Well screw you then. The point is that Facebook and similar websites can be prime target for packet sniffing. So are things like POP email, IM clients and etc. You shouldn’t trust any network, including yours but suspiciously open wifi networks are the worst. How do you know the owner of your internet gateway doesn’t have it set up to log all the crap that goes through it. You don’t. Even if you think your neighbor Joe wouldn’t do such a thing, you can’t be sure that his nephew Sid who set up his wifi is not a diabolical jerk who collects peoples passwords for fun and profit.

When you are on a strange network you should be at least using SSL to make sure that things like passwords are not being sent in plain text. Most of the sites are pretty good about it and do serve their content via SSL. Most do not default to it though – facebook is a prime example here. The default login page is not encrypted. Furthermore and because of how the Application API works even if you log in over SSL the content you get served is mixed – parts of it come in un-encrypted so you are still leaking data.

Or even worse – what if, for example you are running a self hosted blog like me. Do you use SSL to log in? You do? Holly fucking shit! I don’t! I never actually thought about it, until few days ago!

Granted I don’t steal internet (like some of you people, you know who you are) but I do sometimes log in from school, work, and etc. This is not acceptable. So as you can imagine, the first thing I did after realizing this was to go and set up an SSL certificate for the blog. The second thing I did was to add this line to my wp-confing.php:

  define('FORCE_SSL_ADMIN', true);

What does this do? It forces wordpress to use SSL for all administrative functions and redirects all links accordingly. So for example if I get an email notification about a post in moderation queue I can hit the “approve” link and have WordPress automatically serve me the SSL encrypted page instead of an un-encrypted one. This makes me feel a little better about logging into my own website from networks I don’t actually own.

I still wouldn’t use someones insecure wifi connection though. Unless for torrents of course. Which is why you should should use WPA/WPA2 or better kids. Otherwise some jerk will use all of your available bandwidth to download tons of movies and video games without ever worrying about being caught.

Every once in a while some of you complain about restrictive spam control here. Sometimes comments get blocked because they have too much links. Not so long ago, quite a few people got blocked just because they were behind a proxy. These are unfortunate glitches and I try to work around them and massage the spam tools to be nicer to people. They are effective, but they lack the much needed people skills… And intelligence. But we are working on that.

In the meantime I wanted to show you this graph, that illustrates the ration of spam comments to non-spam comments on this blog.

Spam vs Ham on Terminally Incoherent

I didn’t just make up this chart. It came out of my Akismet panel based on the data it collected. Since December 96.73% of comments posted to this blog were pure spam. Can you imagine that? Ninety seven fucking percent! It’s insane!

How many of these spam comments did you see?

None! They were all silently blocked and hidden away so neither you nor I have to deal with them.

Of course, you could say that this graph could be based on only few dozen comments. But it is not – let me post another graph to prove it:

Spam over time - note the recent spikes

All in all, I think I’m averaging few hundred to a thousand spam comments each month. Some months seem to be worse than others. For example May and begging of June seem to be particularly bad. I’m not sure if this is just a local fluctuation or an increasing trend. Either way the amount of comments collected over time suggests that the 97% is probably not skewed by a small sample size.

I guess this comes with the territory. Terminally Incoherent seems to be one of the small blogs that are popular enough to get spammed, but not popular enough for the big comment threads to offset the spam-to-ham ratio. Not that I’m complaining.

I’m actually thrilled that I managed to build this small community. I love the fact that I seem to have gained few regular readers who stop by frequently and post insightful comments. I’m also always amazed at the high quality of the discussions we have here. Funny thing is that only time the comment threads seem to degrade is when I get dugg or reddit-ed and we have a temporary influx of new readers.

Anyways, I just wanted to thank you guys for making the delicious ham comments we have here. You make running this blog worth while. And if my spam filtering friends seem annoying sometimes, give them a break. They are doing a great job keeping out all the crap from our comment sections. Without them, spam would drown out any legitimate comments in an endless torrent of unsolicited advertisements.

Last night my blog broke for some unspeakable reason. I was unable to post or edit posts – each time I would hit the “Save” or “Publish” button I would be redirected to my homepage without any changes being saved. Just like that.

I spent whole Sunday ripping my hair out, cursing, banging my head against the wall and tweaking every possible setting on the blog I could think of. I disabled and enabled cache and anti-spam plugins. I upgraded the PHP version on the server. I tried disabling mod_security as per this thread which was describing eerily similar issue. I edited and re-edited my .htaccess files about a million times.

The only thing I didn’t actually do was to update WordPress… Cause you know – that’s just crazy talk. As my last resort I contacted Dreamhost support, but since it was Sunday they only got back to me today. Do you know what they said?

“We’d love to help ya but you must upgrade your wordpress first cause your version is way old.”

I’m paraphrasing here – the support guy was actually much more courteous and used expressions like “I’m very sorry to hear about your problem” and “I will be happy to help you” but that line above was pretty much the gist of the email.

At that point I got pissed off and did something reckless. I logged into my CPanel and clicked on the big button that said “Upgrade Your WordPress, Stupid!”. That thing has been taunting me for the longest time but I never had the guts or inclination to click it. I’m one of those… What do they call them… What is the polar opposite of an early adopter? And no, it’s not late adopter. I’m like the anti-adopter.

I run obsolete software and hardware until it breaks. I’m a firm believer in the old adage “if it ain’t broke, don’t fucking touch it less you want to spent the next 6 hours fixing it”. But since my blog just broke I felt compelled to press that button. That was reckless and stupid, mind you cause I happened to be running WordPress 2.0.5.

Yeah, shut up – I’m that bad. I’m gonna turn in my geek badge at the end of the day. The thing is, when you upgrade from that far back, things are bound to get broken. I know this from experience – I’ve seen many broken upgrades in my day. That’s probably why I’m deathly scared of them. But I was desperate, pissed off and I felt reckless. So I pressed the button.

Lo and behold – the blog fixed itself, and without any major fuckups. At least none that I can see right now. I mean, you can read it and I can post again so that means we are back in business. The custom theme went to hell during the upgrade, but I have it backed up so I can easily restore it later today or tomorrow. The important thing is that the damn thing did not break into million pieces like I expected – which is great.

So that’s why the blog looks like shit today. But look at the bright side – until I fix the theme you get an add free Terminally Incoherent experience for the day. Or at least those of you who are not running adblock.

Anyways, I’m happy and relieved and I can go back to writing ridiculously long, and badly spelled posts for your enjoyment. Oh and WordPress 2.7 is kinda spiffy. My only problem is that I can’t find anything now. But I’ll get used to it. :)

Because of this I have been “just getting around to change my theme” since around the year, fucking 2006. So remember that folks – when I say I’m going to do something soon, the realistic ETA you should put in your books is “approximately 2 years from now”. Now if I give you a concrete date and time, then it’s a different story. But I don’t commit to concrete dates easily.

Anyway, I got sick of having my blog look exactly like every other WordPress blog out there and I decided to do a little cleanup. The goal was to inject some semblance of originality and uniqueness into this page, without me having to do any actual work. Over the years, I have tweaked this theme in various random ways, adding custom hacks here and there, and never bothering to comment them. At this point I’d need to diff it with a clean Kubrick theme to figure out which bits are mine, and which are original. Then I would probably need to carefully transplant all these custom hacks onto a new template. That seemed like way to much work. I decided to take my existing theme and beat it with a stick, until it starts looking less generic.

You can’t really call it a redesign. I just added more orange in the body of the blog, dropped the gradient header, and the gray sidebar, repositioned some elements and etc. I was shooting for something simpler and more minimalistic in spirit than what I had before. As usual, constructive criticism, thoughts, comments, suggestions are always appreciated. :)

Hello Lukasz,

I apologize for the interruption, but I have had to temporarily disable your site. Unfortunately, this was causing a huge load on the machine:

popcorn: 01:21 PM# uptime

13:21:58 up 78 days, 21:26, 8 users, load average: 96.58, 73.29, 53.76

After disabling this, the load dropped significantly:

popcorn: 01:22 PM# uptime

13:25:32 up 78 days, 21:30, 8 users, load average: 29.25, 51.65, 49.33

You probably need to enabling caching for this blog or disable any extraneous plugins before re-enabling it. You can check out more about it here: http://wiki.dreamhost.com/WordPress_Troubleshooting

Thanks!

I know that you get what you pay for, but I was not really happy about them just rendering my website inaccessible without:

1. Letting me know there is a problem, and allowing me to fix it
2. Telling me how to restore my website to usable condition again

Fortunately all they did was to rename index.php to disabled_index.php so I was able to quickly get it back online. I also tried to comply with their whiny demands and made Caching much more aggressive. Let me know if you notice any weirdness on the page – especially stuff like CAPTCHA’s not working properly.

Here is my reply to them – note that I was a tad pissed off when I wrote this. In retrospect perhaps I was a bit harsh on them. What do you think:

First, how to I enable my site back? I’m getting emails, my readers are not happy and I’m losing advertising revenue. It would be nice if I got some instructions on how to do it. I’m guessing I just rename disabled_index.php back to index.php? Thanks for telling me that though.

Second, all I’m running on my account is a basic WordPress account with couple of plugins – chiefly WP-Cache, Akismet, and Bad Behaviour which aim to reduce the system load, and prevent spam. Two out of these were specifically listed on the troubleshooting site you linked to in your email. There is really not much else there.

You want to know what I was doing in the last few hours? I was tweaking my .css stylesheet and I cleared the cache few times to see the results. Could something as minor as that be the reason of such a huge spike? I will refrain from doing this in the future, but geez!

I haven’t added any plugins in the last 6 months. I haven’t changed the theme. I haven’t run any scripts. All I do on that account is blog. There is really nothing there that would be out of ordinary – just WordPress installed via the one-click install from the panel. The only changes I really did to it in the past 6 months were small css tweaks.

Third, over the last year or so, I frequently witnessed the system load on popcorn to spike up way above 200. Some of my readers joke around that the site often takes up to 3 minutes to fully load during peak activity hours. Why is that? Is WordPress really such a resource hog? Do I really get this many hits per second? I really doubt it. There is no way for me to control how much traffic do I get. I’m really not running some crazy scripts in there – it is just a fairly standard WordPress install with a default theme. I can mess around with it a bit and try to switch off plugins but I really don’t think that will reduce the load of this website that much – I mean there is not much to disable there.

Really, I understand that it is hard to run a shared hosting company. I was always with you guys. I have been always very tolerant, and a loyal customer. I even recommended you to some of my friends despite the apparent flaws. I dealt with the regular downtimes, and I didn’t really say anything when you guys accidentally charged me over $200 for the next 3 years of hosting few months back. It happens. But disabling my website for undisclosed time period on a whim without giving me any prior notice and any possible chance to fix the issue before the website goes down is just pushing it. It’s unprofessional, and very disappointing. I’m hurt, I’m angry and I’m annoyed. Perhaps it is time for me to start looking for a new home.

If you see an activity spike on popcorn later today it might be because I’m backing up all my files and preparing to move to a new host.

So please tell me, what am I supposed to do here? What can you offer me? I’m not really interested in running a website composed from static HTML pages – I actually do want to use WordPress and I do want to keep runing Akismet and other anti-spam plugins which keep it clean. I don’t really want to spend hours, hacking WordPress and trying to figure out how to optimize the plugins that I use. I just don’t have time for that. And I definitely want more people visit my website – not less.

So what are my options? Will my blog get disabled every time there is a traffic spike? I mean, what if (God forbid) one of my posts ends up on a front page of Digg tomorrow? I’m not saying it will but it happened before, and I hope it will happen again. What then? Will popcorn spontaneously burst into flames?

Why should I stay with dreamhost?

I’m not a happy customer right now.

Was I out of line? I really think I will need to find a new host soon. I know I say it all the time, but stuff like this might just be the thing that will push me to finally do it. So this is once again time to pimp your host. Who do you use? What are their prices? What plans do they offer? How is their support like?

Update – it seems that I actually got a relatively knowledgeable and thoughtful response:

Hi Lukasz,

I’m the resident WordPress nerd around these parts and I can understand how upset you could be. Still, your site was causing undue strain on the server. I don’t think it has anythig to do with clearing out WP-Cache tho. I do notice that you are running Bad Behavior as an anti-spam plugin. Normally. that’s okay – but for whatever reason Bad Behavior and our system don’t seem to play nice with one another when a site is being hit by a spam attack. That very well could have been what was happening here – but as I wasn’t the one who disabled you, I can’t tell.

I run the WordPress blogs on our here (blog.dreamhost.com and dreamhoststatus.com) and I’d be happy to give you a few pointers to keep things lean and mean. First up, you’re doing the right thing by running WP-Cache. Unfortunately, the programmer isn’t actively updating that plugin anymore. The successor to that plugin is currently WP-Super-Cache. More info about it can be found here:

http://wordpress.org/extend/plugins/wp-super-cache/

As for spam fighting plugins, sticking with Akismet is good – but using WP-SpamFree in place of Bad Behavior is a bit kinder on CPU cycles. We use this as the one-two punch on our blog and the spam that Akismet has
to process has gone down considerably. Check it out when you have a moment:

http://wordpress.org/extend/plugins/wp-spamfree/

Turning that on and getting rid of Bad Behavior will actually take your overall memory usage down quite a bit, so please try that out and let me know how it goes.

Also, I’ve poked around your database and have noticed that the table for Top Post By Category has eaten up 111.6 MB of your database. That’s more than 3/4 of your total MySQL database size right there. If you are no
longer using that plugin, it might be a good idea to clear that category out. In fact, it’d probably be best to stop using it as that will help speed things up as far as page generation goes.

If you’re curious as to why you should stay with us, hopefully the observations I’ve offered you on how to make your site run a bit better in our environment will make a convincing argument. Most companies would leave the ball in your court when it comes to optimization – but I honestly want you to get all the traffic you can. Outside of the WP blogs here, I have my own sites (hosted here) and I know the feeling of folks outside of your immediate social circle checking out what you’re doing. It’s pretty nice.

Anyhow, try my suggestions out if you’d like and please go ahead and let me know if you have any further questions. I’ll be more than happy to help you out in whatever way possible if you do.

The nice thing about Dreamhost – one of their few redeeming factors is that their support staff is usually quick to respond and usually knowledgeable and helpful. Well, with the exception of that tool who renamed my index.php and didn’t tell me about it that is. My past interactions with them were very positive. Can’t hurt to try his suggestions for now. The top posts plugin is gonzo for now! I liked having it in the sidebar, but I did not realize how database intensive it was. Oh well, good riddance. I will try to switch over to wp-super-cache and wp-spamfree over the weekend which may or may not speed up the site a bit. We shall see.

[tags]dreamhost, hosting, blogging[/tags]

1. Image Shack and Photo Bucket have prohibitive storage and bandwidth caps
2. Flickr has an upload cap and a restrictive TOS
3. S3 is cheep but not free

There is an alternative to offsite image hosting though: Coral Cache.

The image above is a good example of Coral Cache-ing your images. It is both hosted on my server, and distributed by the Coral Cache service. How is it done? Click on that image and check out the URL. Every time you want to Coral Cache something simply add .nyud.net to the domain name. That’s all there is to it.

If you do it manually for all your larger images you get a fine grained control on which sections of your website are cached and which are not. On the other hand, this approach can become difficult to maintain. If one day you decided you no longer want to use Coral Cache you will have to search and change the URL for each image.

What you really want is to have normal URL’s for all the locally stored files, and then use mod_rewrite to automagically alter all of them to use Coral Cache. For example, if you are like me and you usually upload image files via the browser using WordPress’ default upload dialog you can use the script below:

RewriteCond %{HTTP_USER_AGENT} !^CoralWebPrx
RewriteCond %{QUERY_STRING} !(^|&)coral-no-serve$
RewriteRule wp-content/uploads/(.*)$ http://www.yoursite.com.nyud.net/blog/wp-content/uploads/$1 [R,NC]

The first two lines are recommended on the Coral Cache wiki. The last line does the actual rewriting.

Put this in a .htaccess file in the root directory of your blog and observe the magic. The first time you load a page it might take a bit longer than usual, but with high traffic, this trick should quickly pay off. I’m not sure how often Coral flushes their cache, but if you have a low traffic site, this might actually slow down the loading a bit if the time between visits is larger than the lifetime of a file in the cache.

Of course the best part about using url rewriting is that you can easily switch it off at any time. When the digg effect subsides, or traffic goes down significantly, and you no longer need the cache you can simply comment out these 3 lines from your .htaccess file. In fact, you can just commend out the last line, as the two conditions can’t really do any damage by themselves.

Note that side effect of this trick is that you no longer need to worry about the hotlinking issue. Since all images will be redirected to the Coral Cache the lazy assholes can hotlink to you all they want. In fact you might as well stick the following two lines right after the Coral Cache conditionals:

RewriteCond %{HTTP_REFERER}!^$ 
RewriteCond %{HTTP_REFERER}!^http://(www\.)?yoursite\.com.*$ [NC]

First line excludes requests without a referrer, while the second one excludes the local requests. You may also want to add few lines here that will exclude Googlebot and other similar services which you may or may not want to index your images. But proper hotlinking prevention strategies are a a topic for a whole new discussion.

Then, when your server gets dugg comment out all the hotlinking related lines, and you go back into ultra-bandwidth saving mode.

Also, please feel free to correct or improve my mod_rewrite code. It is definitely not perfect, but it is functional. I would greatly appreciate any input or constructive criticism here. :mrgreen:

[tags]bandwidth, hotlinking, coral cache, mod_rewrite, nyud.net, cache, cacheing[/tags]

What does this plugin do? Absolutely nothing, except for showing up in your plugin management screen:

click to enlarge

Worpress scans every php file in the wp-content/plugins directory and parses through the comments in those files to create the entries in the plugin manager. You write the comments, and they magically appear in those boxes. I will show you a full comment block in my next example.

Let’s write a very simple plugin – a word filter for your comments. What is a word filter? It’s a script that will replace occurrences of one word with another. You can use word filters to censor obscenities from your comments, or just mess with your readers by swapping words like male and female, straight and gay, or perhaps if your blog is popular enough to attract the First Poster change “first” to “I’m a looser”. In my example I will try to switch foo into bar and test into done.

For that purpose I set up two fake comments on my test blog:

If the plugin works, all the foos and test should change to bars and dones. I will show you the plugin first, and then explain the tricky bit(s):

 "bar ",
				" test" => " done");

	foreach ($words as $key => $value)
	{
		$text = str_replace($key, $value, $text);
	}

						
	return $text;
}

// wordpress hook
add_filter('comment_text', 'test_wordfilter');

?>

I think the function is pretty straightforward – take a string, create array, loop through array, replace every $key with a corresponding $value, return a string. If you don’t get it, tough. Figure it out.

The line you want to pay attention to is this one:

add_filter('comment_text', 'test_wordfilter');

This is a wordpress hook. There are two types of hooks – actions and filters. Actions are triggered by various events such as publishing a post. Filters on the other hand let you manipulate the way data is displayed on the screen.The function add_filter takes two parameters.

The first parameter is the name of the filter. In our case it is comment_text because we only want to use our word filter in the comments. If we wanted to also apply this filter to blog posts we would need to add another filter like so:

add_filter('the_content', 'test_wordfilter');

You can find a full list of available filters and actions in the WordPress Plugin API Refference.

As you probably guessed, the second parameter is the name of the function you defined in the plugin. Note that your function name should be unique, not to collide with other plugins and native wordpress functions. Most filters will return some desired data and send it as a parameter to your function.

Note that this plugin includes a full comment block with all the attributes filled out. If you look at your plugin manager, here is how it will show up:

click to enlarge

Now all we have to do is to activate the plugin. If it works, it should tell you that activation was successful, and the entry should become green. If you have some errors in your script, you will get a brief error message. Usually it’s something among the lines of “unable to activate the plugin due to fatal error”. This makes debugging your plugins bit difficult, so it’s a good idea to use an editor which checks PHP syntax as you type, like for example Komodo Edit. This will cut down on the number of silly typos, and forgotten semicolons in your code.

Once the plugin is activated, pop over to our comments section we can see that it worked:

Note that this plugin is far from perfect. For example it changes the word foobar into barbar which might not exactly be what we wanted. It’s probably a good idea to change the str_replace fucntion to preg_replace and use regular expressions to ensure that only the words we want get replaced. But that’s something you can do on your own.

Now you have a fully functional plugin to play around with. How easy was that?

[tags]wordpress, wordpress plugins, plugin, php, blog[/tags]