The Firewall Saga: Part 5

Welcome to the penultimate yet another installment of the Firewall Saga (it was supposed to be penultimate but it did not work out that way). If you haven’t been following it, please try to catch up. It will make more sense that way.

When we left off last time, a firewall replacement somehow left me with a non routable IP address – a problem that, beyond any shade of doubt was my ISP’s fault. I have called Verizon, only to realize their outsourced tech support call center was entirely incapable of dealing with problems of this complexity. I needed to talk to a network engineer to resolve a router configuration issue but they misunderstood and sent me a repair monkey to jiggle the cables and power-cycle the local appliances. I called them back, and ranted for about 20 minutes, heavily over-using words such as incompetence, outrage, lack of professionalism, dropping the ball, disrespecting the customer, etc… It made me feel a bit better, and they did promise to definitely escalate my issue to second tier.

I walk into work, the following morning, wake up The Intern who has dozed off at his workstation, acquire coffee and start sifting through all the spam in my inbox. Few years ago, my inbox was pristine clean – mostly untouched by the filth of spam messages. My co-workers used to marvel at this phenomenon, and inquired how do I manage to save off the avalanches of crap that flooded their email daily. Unfortunately I do not have a secret technique. I’m simply careful not to give out my email on the internets, and vigilant about deleting and flagging anything that seemed suspicious. Then I went on vacation, and my boss told me to put up an auto-reply “out of office” message. Nowadays it seems like my email is on the list of every single disposed Nigerian prince, penis enlargement specialist and Viagra salesman. Also, apparently I have money in 50+ different banks, who constantly threaten to close my account if I don’t give them my PIN and passwords. It has gotten so bad, that it is actually easier to white-list internal company correspondence and emails from known clients and partners. I currently have close to a 100 filtering rules that help me to fight with the sea of unwanted spam, and make the important and urgent emails instantly visible by application of labels and priority folders.

Email filters – a forgotten arcane art, mastered only by the chosen few. I know for sure that the only client-side filters used in my company have been set up by me. No one else even knows such things exist.

I’m in the middle of fiddling with my tangled web of email filtering rules when I hear my phone ringing. I’m expecting to hear yet another complaint about the time sheet app. If you recall, the whole firewall bonanza somehow broke the VPN tunnel to that remotely hosted server. So for the time being, I am unable to reboot it or tinker with it. But since the non-routable IP issue is more pressing I have pushed the VPN problems aside. Especially after what happened the last time I attempted to fix it. Surprisingly, it was not an internal call. It was a Verizon representative doing a courtesy call. It went a little bit like this:

“Hi, this is Bob from Verizon and I’m just doing a follow up courtesy call about your recent issue. I see here that yesterday we have sent an on-site technician to your location. He has marked the issue as resolved. I wanted to make sure that everything is working correctly and see if there is anything else we can do for you.”

Here are some of the emotions I’m feeling at that exact moment: anger, annoyance, disbelief, rage, befuddlement and hunger… The last one, because I didn’t have a chance to get anything to eat that morning. To help you visualize my reaction, here is a two panel re-enactment of that event. Just imagine that the iPhone is a big clunky office phone, and the coffee mug is a paper cup, and that my shirt has a collar:

He marked it as what?

So it turns out that the field technician that visited us the other day decided to say he fixed the issue. In retrospect, I guess I can understand how it happened. It is very likely that this guy does not work directly for Verizon. He probably works for some local company that Verizon uses to outsource all the cable wiggling, wire snipping and power cycling it needs to do at customer locations. They are likely set up to receive work orders from up high. When they fail to resolve a customer facing issue (for whatever reason) it probably counts against them. So this guys manager probably just said “fuck it, since it was not a local problem we will just put it in the system as resolved”.

But that only occurred to me much, much later. As I’m sitting there on the phone my driving, logic clouding emotion is anger. The upside is that “Bob from Verizon” seems to be speaking perfect English. This is something new. All the support drones I dealt with up until now had very heavy accents. So chances are I’m actually talking to someone physically located in the states. Probably still not an employee of Verizon, but perhaps his call center/department can get me what I need.

So I recount my long and sad story, spearing him no gruesome details. When I’m done, he apologizes profusely then promises to get my issue resolved. He gets a support drone on the phone and together we rely the issue, and it’s importance to him. The thick-accented drone gets in touch with tier 2 support. Tier 2 support insists on sending a network specialist to our location. I try to protest, and try to make a compelling case against it but it seems like there is no use. Apparently they have to make absolutely sure the issue is not local, before they escalate it to the network people. So we make an appointment. I call Barry and let him know we have this guy coming. Together we set up a spare laptop, plug it into our network, assign it a static IP and set firewall to pretend it is our server. The guy will be able to jump onto it and verify that no packets are coming in. I also print out a network diagram, and Barry sends me a document that contains all the relevant firewall rules. When our Network Specialist comes with a visit, we ought to have enough evidence to show him the problem is definitely not on our end.

The next day, our “Network Specialist” Steve arrives at the office. Only, he doesn’t look like a specialist. Of course, looks can be deceiving – and geeky guys can sometimes look peculiar. But this guy just does not look like a networking dude. He a middle aged man, wearing a trucker hat, shorts and a crumpled up t-shirt. The large coffee stain on the front, seems to be locked in territorial combat with his the armpit sweat stains. His gray handlebar mustache gives me an impression that he would be much more comfortable rebuilding motorcycles than troubleshooting network issues. But I decide to give him a benefit of the doubt.

I take him to the server room, where we set up our perfect trap. Next to the rack, there is a little stool, and on it there is the orgy of evidence. The network diagrams, the firewall rules, the trace route logs and the little test laptop ready to be fired up and tested. His eyes glaze over a bit as I talk so I ask him what tests he needs to run, and explain how we rigged the test laptop. He goes:

“Son, no offense but I have no clue what any of what you just said means. I was under impression yous guys had no internet connection…”

I have a sinking feeling in the pit of my stomach.

“You are not a ‘Network Specialist’, are you?”

“What? Hell no! Kid, I was retired up until last week. This is my first day on the job. I sure ain’t no specialist!”

Well, fuck.

I explain my predicament to him. I was promised a specialist, but I got him. Tier 2 refuses to move forward until they have someone on-site run the checks they require. So I hatch a crazy plan. I have now a physical Verizon representative on the premises. Well, more like a trained monkey really – I don’t think he knows anything about anything, but he should be able to follow simple instructions. If we can get the tier 2 assholes on the phone, they can walk him through the required tests. Then we can move on.

Steve agrees to this crazy plan, but says he will probably need his company laptop which he left in the truck. Fair enough. I escort him out of the office and let the front desk know he will be coming right back and to send him right to me.

Steve is gone for about an hour an a half. When he finally shows up, I notice his shirt has acquired ketchup and mustard stains and is beginning to look like a genuine abstract painting. I ask him what happened and he launches into a long winded explanation how he first decided to have a smoke, then he realized he was hungry, and how he has low blood sugar and etc.. I let it go. The sooner we can do these tests, the faster I can get him out of my hair. I show him where to set up, and watch him pull out his ancient flip phone and dial a number.

Then he gets to a voice menu. Then they put him on hold. I shake my head in disbelief:

“Wow, they put you guys on hold too?”

“Of course.” he gives me a wide, gap-toothed smile “People think we have some special, internal number but we don’t. We call the same tech-support number as you do, when you have a problem”

That sinking feeling I mentioned before – it’s back with a vengeance. Steve patiently waits for “the next available representative” while I contemplate suicide for the twentieth time this week. Eventually I get bored watching Steve, and I excuse myself figuring I might as well get some work done. I interrupt The Intern’s intense game of tower defense and tell him to go keep Steve company, and make sure he does not try to mess with the equipment, or walk out with any of our servers. Oh, and to call me when Steve finally gets a live person on the phone.

After about 20 minutes I get a phone call on my desk. It’s not The Intern, but one of my other coworkers.

“Luke, I think we have a problem…”

Oh, God… As if I didn’t have enough problems.

“Damn it… What did you break this time…”

“No, this is more of a Human Resources problem.

Oh, sweet relief! At least I won’t have to deal with this.

“And you are calling me about it because…”

“Well, it involves your server room. I think we have a hobo infestation.”

I chuckle, and explain that he was actually sent by Verizon.

“Ah, that’s what they all say. Next thing you know they start breeding and you get like a dozen homeless people living in your server room. Mark my words man.”

“What do you suggest, oh wise one?”

“Nuke it from the orbit. That’s the only way to make sure.”

“Well, I have The Intern babysitting him…”

“Yes… And? I don’t follow..”

“Right, good point. I’ll see what I can do about it, but you have to submit a ticket for it first”.

About an hour later, I go check up on my server room buddies. I find The Intern intently watching Steve munch on a sandwich. I give him a disapproving look and tell Steve he can it in the lunch area because we do not want food in the server room. He is apologetic:

“Sorry about that. I just got hungry, and with my low blood sugar… You know how it is. I’m still on hold, and they can pick up any time. I figured there is no harm in a little snack…”

To emphasize his point, Steve emphatically waves the sandwich around as he talks. On one of the swings a tomato slice gets dislodged and soars through the sky, hitting the opposite wall with a loud smack. In astonishment Steve slightly releases his grip, and a slice of ham, and some lettuce slither out from between the bread and land on his laptop keyboard. He grabs them, stuffs them in his mouth and then shakes the laptop off sending crumbles, lettuce shreds and other unidentified bits of food on the floor.

I ask The Intern to clean it up before anyone notices, and sternly march Steve to the lunch room, trying to decide whether I should kill Steve or myself first.

Next time, more fun with Steve, and hopefully the climactic resolution. Well, maybe.