The Firewall Saga: Part 4

The saga continues. If you haven’t been following this series, you can catch up to speed here. What follows might be funnier that way.

It is the day after the Beef Instrumentality Incident #631. We are finally chugging along on all cylinders, the users are mostly placated and I finally have some time to call Verizon about my on non-routing IP address. I’m not empty handed either. Barry was kind enough to arm me with traceroute logs for all our IP’s captured from two different outside locations. Which is not much, but they show that the packets sent to that one problem address route fine until they hit the Verizon network. And then, boom, they get shunted into the depths of cybervoid instead of being safely delivered into our office.

Barry also dug out, and sent me his impressively anal retentive case notes from a few years ago, when the same thing happened. These things are dense with tiresome detail: actual dates of calls he made, names of support drones spoke with, etc. According to the notes it only took 5 days and about a dozen phone calls to get the IP routable again. I figure I can do it in half of that time considering the extensive documentation I am armed with. Hell, maybe I can even do it in a shingle phone call.

Around 10 am I let my coworkers know I will be calling Verizon, and that I probably will be on hold till closing time. I ask them to drag me out of there after 5pm, bid them farewell, and gather some necessities. Two cups of coffee, bottled water and snacks in case I am trapped by the phone for weeks. I do some mental preparation, then dial the number. I go through the byzantine labyrinth of voice menus and finally end up in the wait queue. I put the phone on speaker and proceed to do some busy work while listening to their horrible on-hold music.

20 minutes into the call, a coworker from the next cubicle over starts parroting the looped voice assuring me that “Your call is very important to us. Please hold for the next available representative.” After the third or fourth time, I join in and we both say it together.

45 minutes into the call, every single person in the IT cave in on the joke. Every time that looped sound byte repeats, five voices rise up in unison. We are like a group of monks chanting an ancient prayer.

50 minutes into the call, someone decides to hit the Staples “That was Easy” button every time we chant our little chant.

An hour into the call, The Intern manages to perfectly imitate the elevator music with his mouth. By this time, my phone speaker volume is cranked all the way up and it’s a regular sing-along party.

At 65 minutes, someone wanders into the IT bunker with a question, hears us chanting, says “You guys are a bunch of nerds” and leaves. We decide we must do this more often. Anything that keeps users from dropping by with non-essential, non-work-related questions is worth working into our daily routine. Oh, and in case you were wondering that user just had a question about home theater sound systems. Because, you know – system administrators and programmers know all about home multimedia setups.

Finally, after an hour and 20 minutes on hold someone picks up. There is some booing in the background as I disengage the speaker. Apparently everyone was having fun.

I jump through all the requite identification hoops. Then I launch into a 15 minute detailed explanation about our routing issue. I explain to him the dozen or so local and remote tests we performed to verify this is not a local configuration problem. I offer to send him the traceroute logs so that he can see the problem happens only for a single IP address. I also tell him that this seems to be a recurring problem, and give him a quick rundown of Barry’s case history. The guy on the other end patiently listens to all of this, and once I’m done goes:

“Thank you for that information sir. It appears this is a router configuration issue. What we will need to do is to power-cycle the router. It is the little black box with the antenna that we have provided you when we set up your internet. What I want you to do is to unplug it from power, wait 60 seconds and then plug it back in…”

Granted, I sort of expected this to happen. I patiently explain that we do not use their cheap, off-the-shelf appliance with crippled custom Verizon software. I also reiterate that this is a routing issue, not a local configuration error. I ask him to escalate this call to the team that handles network problems, and point to the case history to support my claims. I even give him the exact date when the previous ticket was escalated to that department (thanks to Barry and his disturbingly obsessive note taking).

“I’m sorry sir, but I cannot help you if you are not using the ActionTech router we have provided you. I will need you to unplug your current router, and replace it with the ActionTech before we can continue this troubleshooting.”

Suddenly I realize I might have dialed residential support line. I ask the guy to verify, but he claims I actually called the right number. He is a proud member of the Small and Medium Business department. Surprisingly to everyone, including myself this sets me off on a weird Socratic Method rant.

I tell him that we have this many machines, this many servers and this many persistent VPN tunnels that connect us to other offices and data centers. Then I ask him whether he would classify this as a small or a medium business. Not knowing where the hell am I going with this he agrees that we are probably in a “medium” class.

Next I ask him whether or not the $20 off-the shelf, router they gave us can handle maintaining several persistent VPN tunnels? Does it have a commercial grade firewall software that would allow us to do real time packet inspection and intrusion detection? I ask him if that router has any of the features that we get audited for, and that we are contractually obliged to have in place to protect our client data?

He hesitantly agrees that it probably does not have all these features. I have a hunch he doesn’t know I’m bluffing and that we never, ever get security audits (except for internal ones). Still he insists that I temporarily connect the ActionTech just for the sake of troubleshooting.

I ask him whether he would classify the ActionTech router as an enterprise level device for medium business users, or a personal use appliance for residential clients?

He agrees it is more on the residential side.

For my cup the grace I go:

“Ok, so let me get this straight. We are paying for business class FiOS connection, and business class support. Why then are you reading troubleshooting steps from the residential support checklist, asking me to dismantle my entire network architecture and connect via an a off-the-shelf, residential device?”

Then I once again plead with him to escalate this to network support team, or to connect me to his manager. He mumbles a bit confused, then asks to put me on hold while he consults with his supervisor. A coworker from the next cubicle chimes in:

“You tell them, dude! Gotta be stern with these assholes.”

I interpret this as a compliment, seeing how I am usually not the most assertive person. In fact I sort of feel an inkling of pride for coming up with that question and answer session. Then the phone suddenly goes to dial-tone.

I have just rolled like a natural 20 on my persuasion check. I have had this guy on the ropes! He was about to do what I asked him to do! And the motherfucker hangs up on me! To make matters worse, now I have to call them again, and repeat the exact same exercise with another asshole who will ask me questions about my ActionTech router.

Boiling… Murderous… Rage…

If I was Bruce Banner, I would probably be rampaging green skinned monster in ripped up purple pants by now. Fortunately I never participated in any gamma radiation experiments so I merely let out an agonizing groan, slam my phone down real hard and decide to take an early lunch. I nearly collide with The Intern who fetched himself a fresh cup of java. Fight or flight reaction kicks in, and sends him spilling half of the mug as he is frantically scampering out of the path of my angry walk.

I return sometime later, with a clear head and full stomach. I’m determined to get this call done today, so I sit down and repeat the entire procedure. This time I don’t put it on speaker because I don’t think anyone else is in the mood to sing along. At least I know I’m not.

I spend close to an hour on hold, but fortunately this time I get someone whose IQ does not seem to be a single digit number. He still tries to make me fetch the ActionTech router, but I eventually drops that troubleshooting path. Instead, he decides to follow a different branch on his troubleshooting decision tree.

“Sir, because of your unique network configuration I’m afraid I will need to send an on-site technician to perform some local tests.”

God, damn it! No! This is a configuration issue on your end. All I need is five minutes of time of one of your network engineers. There is one entry in your routing tables that got fucked. I just need someone to go and un-fuck it. Just let me speak to someone who knows what a “routing issue” is. Please! It happened before. Look in your case notes. It should have all the information. My notes say we spoke to some dude named Richard. He fixed it last time. Can you please get me that guy!

He ignores my pleas, insists on sending out a guy. We go back and forward like this for about 15 minutes, an I eventually manage to twist his arm into escalating the issue somewhere higher. In fact, he agrees to forward my traceroute logs to the Tier 2 team for reference.

“Ok sir, you can send it to my email. It is V as in Victory, Z as in Zebra 123456789995-0 at hotmail.com”

Hotmail?

“Is that your personal email? Don’t you have like a verizon email account?” I inquire out of sheer curiosity.

“I don’t know. This is what they set up for me and told me to use sir…”

Few more innocent probing questions reveal that my friend on the other end of the line doesn’t even work for Verizon. He works for an outsourcing company. They are not directly affiliated with Verizon – they are just hired by it to act as a storm shield against the wrath of dissatisfied customers. So of course they don’t get to have verizon email accounts. Providing legit emails to folks who handle their front-line customer support is apparently not important to Verizon. They are perfectly fine with the legion of support drones sharing a few dozen hotmail accounts, and looking very, very unprofessional.

I get him the logs, I grab a case number and finally hang up. It’s almost 4pm. I have wasted almost an entire work day trying to get a single stupid issue logged in the Verizon system and escalated to proper department. Still, I feel like I have accomplished something. The case is being sent to the second tier, so perhaps someone who actually works for Verizon will get a chance to look at it. I might have wasted way to many hours on this but I feel like I have a realistic chance at beating Barry’s week-long turnaround for this issue.

Next day, I spend entire morning catching up on work I didn’t have a chance to do while fucking around with outsourced Verizon support drones. Around lunch time, I get a phone call from the front desk. Apparently some “Verizon Guy” showed up, searching for Luke.

It turns out that Verizon sent out a field technician to our office anyway. They said they wouldn’t. They said the issue was being escalated to Tier 2, but apparently that is not what happened. Right now there is a guy in our lobby and there is absolutely nothing he can do to fix the issue. But I figure that maybe I can explain the problem to him and get him to forward that information up the stream. Hell, maybe he can plug himself in on our network, run whatever diagnostics he needs to rule out a local configuration issue being the cause of our problem.

I go and fetch hom, bring him back to the server room, show him where the FiOS box is, and how it connects to the firewall. I ask him what tests does he need to do, and make sure he knows we can’t bring anything down during business hours. The guy looks at the server rack, the tangle of network cables going in and out of various switches, all wide eyed and slack jawed. He goes:

“Dude… I just thought you are going to have a bad connection, or maybe a broken router or something… This…” he gestures at the server rack housing the firewall “This is way out of my league, man.”

Apparently no one even told him what the problem was. The dispatch just said the client was experiencing connection issues. There was nothing there about routing problems. And even if there was, this guy was not trained to troubleshoot issues like that. He was armed with a spare modem and a wire crimper, and trained to jiggle cables and power cycle basic network appliances. But he seems like a nice guy, so we chat for a bit, and laugh at Verizon’s lack of competence. He says he will talk to his supervisor and see if he can pass the message along up the chain of command.

He leaves, and I clear the rest of my schedule for another grueling Verizon support call. Also I contemplate committing a ritual suicide.

Next time on Firewall Saga: Verizon sends out an “Network Specialist” to our location. Hilarity ensues.