The Firewall Saga: Part 1

My office is protected by Checkpoint Firewall-1. It is one of those fancy-shmancy enterprise level firewalls, that thankfully I do not need to maintain. You see, me and my brethren in suffering are the “internal affairs” team. The firewall falls under the our “department of defense” (or “the networking guise” as we call them) jurisdiction and I am absolutely A-ok with that. I used to be in charge of internet facing firewalls, and I always worried that I missed something crucial. Nowadays I can just sit back, point fingers and blame the firewall guys if something bad happens. The only downside of this is that we don’t have access to the firewall box, and the networking team is in a different building (and different town) altogether – so every little change takes about a week. We like to pretend that this is our “change of controls authorization process” (in fact, that’s I how I documented it) but in reality it is usually just them ignoring our requests until one of the managers gets annoyed and intervenes.

The unfortunate side effect of having enterprisey firewall and small-medium business everything else, is that both our ISP and our VoIP provider think I’m full of shit when they ask me what kind of “router” I use. Firewall-1 is not in their handy-dandy manuals and therefore must not exist. Usually our conversations go like this:

“So what is your router. It will be a little box that says something like Netgear or Linksys”

Checkpoint Firewall-1. It’s not a box, it is a Linux based firewall solution.

“Um… Ok… Let me see. Ok, we are looking for a router. Small box… Antennas. Can you try ti find it.

And so on. It usually takes at least 40 minutes and one or two escalations to actually establish what firewall we are using. After doing this several times usually just lie and say we have some random Linksys router, and then have them walk me through basic troubleshooting steps from the manual saying “It did not work” to everything. Believe it or not, it actually takes less time and I don’t have to listen to them making confused sounds with their mouths for 20 minutes.

We, here in the “internal affairs” team treat that piece of hardware as our personal Lord Voldemort. We pretend it does not exist, and refer to it as “that thing that shall not be named” because naming it would give it power. Keep in mind that that it never crashes, and we never, ever reboot it. Mostly, because we can’t.

Our firewall had a strange quirk that became apparent after a big power outage in the area during which the building’s backup generators gave up, and which drained all our battery backups. It was a long time ago – and the first time it went down since it was installed. Upon rebooting, it failed to find appropriate license keys and promptly cut all network traffic. This of course included the remote tunnel via which the network team used to maintain it. After some panicking, and some phone troubleshooting we got the license keys back in place and restored communications.

We should have known better, but at the time pretty much all of us went “Wow, that was a weird glitch. It probably won’t happen again.”

It did. Once, twice, three times. Network guys eventually acknowledged this was a problem, but after every single incident they were 100% sure the problem was completely resolved. I eventually figured out how to prevent it from affecting our productivity using simple visual cues and access controls.

I made a sticker that said “DO NOT REBOOT THIS MACHINE, EVER!” and placed it on the front of the Firewall box. Then I locked the rack it was in with a key, and hid the key in my drawer, after affixing it with a tag that said “DO NOT REBOOT THE TOP BOX IN THE RACK”. If anyone needed to access that rack, they had to answer one simple question: “Which box is not to be rebooted?” If the answer was anything but “the top one” the key would stay in the drawer.

Eventually, I managed to convince the powers that be that the problem persisted, and the networking team tasked with fixing it. And that’s how it all started.

One fine morning, two members of the networking contingent show up in our office. Let’s call them Toby and Barry (not their real names). Toby is the guy lugging around the heavy objects and babysitting installer progress bars, while Barry is the guy babysitting Toby and telling him which configuration options to pick. Also, their car seems to travel at relativistic speeds and experience time dilation everywhere they go. When these two are on the road, “we’ll be there in about 15 minutes” usually means three hours.

Of course no one told us they are coming. I simply get a phone call from Toby around 9am, informing me they are 15 minutes away from our office and that they will need the access to the Firewall rack. I go WTF, my boss goes WTF and we basically pass the WTF around the office until everyone is thoroughly confused. Don’t get me wrong – we are all kinda happy someone finally decided to fix this issue for good, but then again these guys don’t have the best track record. Also entire staff is getting anxious, whining that out of all days in the year, today is the day when we cannot afford to have any downtime. And tomorrow. Tomorrow is also the only day in the year when we can’t afford any downtime. And the day after tomorrow too.

So while the entire accounting staff is hyperventilating and having panic attacks I take down the “Beware of the Leopard” plaque from the front of the firewall rack, pull out the key, unlock it and then wait for our guests. Then I go to lunch, come back and wait some more.

They arrive half past noon, and head straight to the IT department. We meet and greet, and they unveil their plan: take the firewall down, grab a disk image with clonezilla (as a backup), then update it to the latest and greatest version. My boss looks skeptical, and wants to know how long is it going to take. Toby and Barry ponder this for a minute and give their best guess estimate: “twenty minutes max, maybe less”. After you adjust for dilation it does not look like something that can be managed in the middle of a work day. At least not without the accounting staging a mutiny. The verdict goes down to do it at the end of the work day.

So now we have Toby and Barry hanging out with us for the rest of the day. I find out that Toby has never seen a Michael Bay movie he did not love, and I lose a little bit more faith in humanity.

Finally it’s 5pm and we get to work. Firewall is backed up in no time and Toby pops in the update disk into the drive. He tries to mount it but fails and looks preplexed. He takes it out, puts it back in, tries again and starts scratching his head. He calls Barry over and they repeat the whole ordeal again twice. I’m watching this and begin to wonder if the networking sent us their very own Loyd Christmass and Harry Dunne.

Barry brings his laptop, pops the disk in to verify it works and has data on it. It does, and it has. So they go back to the server rack and repeat the entire procedure two more times. No dice. Then they turn to me.

“Any idea why this drive is not reading our disk?”

I walk over, take out the disk and inspect it. The little letters around the central hole spell out a word: “DVD-R”. I look over at the disk drive in the Firewall box and see another word there: “CD-ROM”. I wordlessly point these things out to my new friends. It takes them a few seconds but then it sinks in.

You want to know what is the funniest part about this scenario? The ISO they have burned on the DVD is only a bit over 600MB in size. It would fit on a regular CD-R without any problems. But Toby burned it on a DVD disk because apparently that’s what was on his desk. Also, Barry (the only one of the two who brought a laptop) did not copy the ISO to his machine.

We dispatch Toby to find an electronics store that is still open and obtain a USB DVD-ROM, while Barry and I hold down the fort and try to figure out if we can download the ISO from checkpoint website, or if we can just try to copy the data from the DVD onto a CD. Toby lucks out, and gets a drive in an electronics store just across the street and we are back in business.

The drive goes in, update starts executing and we all are relieved that we managed to snatch victory out of the jaws of defeat. Then something unexpected happens. The installer suddenly craps out and crashes in a weird way. Little googling tells us that there is actually no upgrade path from the version we have installed to the version we were trying to install. They are too far apart, and you either have to do a clean install or incremental update through the five or six versions that separate the two. Apparently my friends did not bother doing any research.

To add insult to the injury, the failed upgrade rendered the machine un-bootable so now we have to restore it from the image we took earlier in the evening. By the time we are done it’s around 9pm and we decide to call it a night, and reconvene the next day.

Fast forward a day, and at least 3 annoyed speeches about lack of competence delivered by different managerial personas. I am happily pointing fingers and deflecting blame with my “I am just babysitting them, they are the checkpoint experts” routine.

Toby and Barry show up around 6:45pm (time dilation – you have to adjust for the time dilation with these guys) and have a concrete plan. Back it up, export all the database rules, do a clean install, import the rules back in. In fact, clean install will most definitely fix the license key issue so this is a good thing.

This time around everything goes well. Toby babysits the installation, while Barry and I browse the web on our phones. Suddenly, Toby starts whimpering. Not a good sign.

We go check up on him, and it turns out that the new version of the Firewall software is not fully backwards compatible and there are some issues importing our rules. After several trials, and some online research we decide that the best thing to do is to re-create the rules from scratch. Barry steps in, and for the next hour and a half he painstakingly re-creates our setup.

Then for shits and giggles we decided to reboot the damn thing to see if it looses the license key again.

Guess what happens? Same thing as always. Firewall comes back up, looks around and goes “Fuck you guys, y’all dirty pirates!”

Toby and Barry are stumped. We all just wasted two evenings and accomplished absolutely nothing (well other than getting the firewall upgraded). They retreat to their home base to report on their critical mission failure, while I lock up the rack, and hang the “Beware of the Leopard” plaque on the front again. For me, it’s business as usual.

Next time on the Firewall Saga, Toby and Barry hatch a new plan and a piece of hardware experiences free fall for a few brief seconds. Stay tuned.