The Firewall Saga: Part 7

It has been almost a week since Steve’s visit. My daily interactions with Verizon have settled into a very predictable pattern. Every afternoon I get an “unexpected” visit from an on-site tech. I explain the problem is not local, and they leave. I call Verizon and complain. I bitch, moan, threaten to change service and etc. I talk to a floor manager at the call center who does not have any power or resources to help me. All he has is an online chat tool and ticketing system for escalating things to tier 2 – the exact tools the phone drones use. His boss can’t help me because he is in no way, shape or form affiliated with Verizon, but rather manages the call center company. After a while they placate me with solemn promises of swift resolution, reimbursement and etc. The following morning I get a call, notifying me that my issue was market as resolved. I call again, bitch, yell and complain some more. The issue gets escalated. Tier 2 picks it up, and bumps it back down with a note to dispatch a technician to check all the connections, and jiggle all the wires. And the cycle repeats.

Every time I call, I give them the same speech. We had this issue once before. Somehow you have resolved it. All you need to do is to look about a year back in your case history and figure out what was done back then. Unfortunately, I get the impression that the people I’m dealing with either do not have access too, or do not keep case notes that would reach that far back. I’m documenting all the dates, and failed attempts to rectify the issue, because I fully intend to ask them to refund us all this lost time.

In the meantime a brand new version of certain crappy, proprietary web app comes out, and the angry shoes brigade gets annoyed. I can’t upgrade that server, because if you recall from previous installments of this series, it is located in another data center, and the VPN tunnel is broken.

I figure that the Verizon issue does not look like it is going to get resolved anytime soon. The lack of connectivity with the data center is becoming a nuisance, so I decide to call up Barry from the network team. I figure that I will get him, and Charlie from the data center and we’ll just keep rebooting the damn machines and tweaking Firewall rules until they sync up and establish a viable connection.

I couldn’t have picked better timing to call about this. It turns out that Barry and Toby are going to be in our data center the next day installing and configuring some new hardware. Toby is apparently in charge of hauling all the equipment onto the site, while Barry will be bringing his networking skills. Even better, Barry has to drive past my office on his way to the data center so he agrees to just stop by in the morning. This way we will have one of these guys on each end of this conundrum, and we won’t have to rely on people like Agent Beef to be our hands, eyes and ears in the rack-space. Our plan is to get things done in the early morning, before the managers and directors start slowly trickling in after 9am.

Next morning I arrive at the office extra early. When I leave the house it is still dark outside. When I arrive at the office, my car is the third vehicle on the completely empty lot. As I grab my laptop bag from the trunk, the day star crests over the horizon and vomits painful bright orange light onto the deserted sea of concrete. People talk about dawn as if it was something beautiful and romantic – but every single time I see one, it is like getting stabbed in the face with a condensed beam of fatigue.

The malnourished, dirty hobo birds that stupidly picked the parking lot as their feeding ground are woken by suns slow upwards creep and decide it is time to scream their fucking beaks off like it’s some big event. I flip them off, and curmudgeonly drag my sleep deprived carcass into the building.

I bump into one of my early rising coworkers – he is on the super early shift, to field those 6am phone calls from our workaholic clients whose morning routine includes 3 important business calls, shower and coffee. Some people jog in the morning – these folks make work related calls for sport I guess. Never understood this attitude, but then again who am I to judge.

My coworker inquires about my unusually and uncharacteristically early arrival. I attempt to tell him that I have an appointment with Barry to fix some outstanding issues with our network but what comes out of my mouth is:

“Hhhnngrrr brrrry ntfffff!”

Somehow I manage to convince my stiff and unresponsive body to make a zombie shuffle up to the coffee machine. I suck on it for about 20 minutes, and then collapse at my desk. I try calling Barry, but he is incommunicado.

So I wait… And wait… And wait some more.

9am rolls around and Barry is still MIA. Finally he calls me around 9:30 to let me know he is about 15 minutes from the office. He arrives at 10:45. Barry lives out of sync with the normal time-space continuum and his personal field of influence time works differently. Of course Toby is not at the data center yet, so all we can do is to run some local checks, and make sure the firewall rules are correct as we wait. I use this time to fill Barry in on my dealings with Verizon. He is amused, appalled but not very surprised. He offers to do a tag-team call with me so we can take turns yelling at Verizon. I doubt that we will accomplish anything new, but I am willing to try anything at this point. Hell, I don’t even want to strangle him for making me wake up so early and then failing to show up till almost 11am. That would require too much energy, and in my sleep deprived state I am all about energy conservation.

Eventually Toby gets to the data center around noon and we get to do some troubleshooting. It appears that firewalls on both ends see each other, but for some reason can’t establish a tunnel. Unfortunately Toby’s end uses a very dumb dedicated network appliance which is not giving us any good diagnostic data or meaningful error messages. After few reboots of the appliance Barry gets an idea.

“Toby, what is the time on that appliance?”

Toby scrambles to find the information in the web interface. You can hear him click on a dozen of tabs and/or links before he finds a status page. Finally he goes:

“It’s is showing time as 12:25pm EST”

I watch Barry run the date command on the firewall’s console. It spits out 12:13pm EST. Way off! He quickly resets the time on our end, tries to re-establish connection and we see the VPN tunnel snap to life. Apparently the authentication algorithms were thrown off by the time discrepancy on the two systems. When Toby and Barry set up this replacement firewall in Part 2 they probably did not bother syncing it with an NTP server. Most likely Toby just glanced at the wall clock when setting up the date – one of those cheap, unreliable battery powered things that tend to drift a lot. That was the reason why we got cut off from the data center.

Now if we could only fix the non-routable IP issue that quickly. Since Barry is already logged into the firewall he decides to poke around a bit. We more or less exhausted all the possibilities last time, but he figures we can perhaps take screenshots, and logs and use them to support our claims. We get Toby to plug his laptop into an external line (not the VPN one) and send packets to the non-routing IP, while we watch the activity on the screen. Something weird happens – we see text scrolling down the screen. Packets are coming in.

I jump of my stool and scramble to boot up that laptop we set up for Steve. It is running IIS, and a simple test webpage and the firewall is set to route all the inbound traffic to it’s internal IP. When it’s up, we ask Toby to try hitting that IP with his web browser.

There is a short pause and he goes:

“Oh shit! I see an animal!”

I whip out my phone, and sure enough – there is my test page:

The title attribute for this page was Mushroom, Mushroom.

“Barry… What the fuck just happened?”

Barry is just as stumped as me. The only thing we changed on the firewall today was the time. It is impossible that a 10 minute system clock drift could possibly have any effect of routability of one of our 5 IP addresses. Nothing we did today could have possibly resolved or issue. And yet, our insurmountable problem, somehow fixed itself, literally overnight (when I checked it last night it was still broken). How did this happen? I have no clue. Barry had a hypothesis or two:

• It is possible (but unlikely) that my complaints somehow got forwarded to the right department
• Perhaps some network engineer noticed this issue during regular maintenance and fixed it
• It is also possible that Verizon routers have some self healing protocols that cause them refresh their routing tables every once in a while

I feel relieved, but also a bit cheated. I sort of wanted Verizon to acknowledge this problem and resolve it. If it happens again (and it may) we will be back to square one, dispatching useless technicians to fix a routing issue. On the other hand, the thought of no longer having to deal with Verizon made me extremely happy. I was just sick and tired of the entire ordeal – especially the brain dead Verizon tech support drones and on-site technicians.

You would think that this is the end of the story, but it is not. There is still one event of note that I still haven’t mentioned. But to get to it, we have to advance the clock by about a month or two. I am finally free of grief and residual pain from this ordeal, and I’m turning it into a long running multi-part series on Terminally Incoherent. Around the time the part of the story where I introduce Steve hits the web, I suddenly get a text message from on old friend:

Do you want me to file an internal escalation for you?

Note that this is completely out of the blue, and out of context for me. By now I’m done with this whole issue. It is ancient history that made for a funny series of articles. So my response is along the lines of “Huh? An escalation for what? Why?”. Then he explains:

For your nonroutable ip from your firewall saga

This, ladies and gentlemen is the exact moment when I punched myself in the face. Somehow I managed to completely forget that I know someone on the inside. I have insider contacts within the bowels of Verizon. And apparently while I was sitting here contemplating firebombing their headquarters, this person could have filed an internal ticket for me. A ticket that that could have potentially helped to fast track this entire ordeal.

Thus concludes The Firewall Saga. I have some more stuff like this in the pipeline – though probably not as long. Now that the series is over, I went back and added a little navigation table at the end of each post. This way, if you decide to share this story with a friend, they can just click through to the end and read the entire thing without too much hunting around.