It’s Monday morning, and I’m sick. I thought that I was smart when I got my flu shot a few months ago. I figured that I can cock-block the influenza virus, and skate through the winter unscathed. What I did not anticipate was that being free of flu I was a prime target for things that are much worse. One of these super-bugs that makes you congested, and makes swallowing feel like gargling with sandpaper got the best of me. I would have stayed home, but I don’t have sick days.
In their infinite wisdom, the powers that be decided that staying home sick counts as a “vacation” so I wisely opted to drag my sneezing, germ producing carcass to work, making it my mission to spread the malignant disease as far and wide as possible. My hope was that I was going to be able to burrow myself in my cubicle, put some code on the monitor so it looks like I’m programming and then just survive till 5pm battling fever and sneezing fits.
Alas, this was not to be. As soon as I sit down and open my email, support ticket notifications start streaming in:
- Ticket #6451 [High]: CAN”T SAVE AANYTHING!!1
- Ticket #6452 [Medium]: G Drive not working
- Ticket #6453 [Critical]: facbook.com saiz page not fund – pls unblock!
- Ticket #6454 [High]: Can’t save to LAN.
- Ticket #6455 [High]: When trying to save to G: drive it says “Drive full or right protected”
- Ticket #6456 [Medium]: Is the public network share full? Can’t save anything.
- Ticket #6457 [Low]: Can’t save to network share. Probably over quota. Please extend.
It goes on like this for at least 20 more messages. The emerging pattern is somewhat clear – something is terribly wrong with the network shares. I’m fraught with a sense of déjà vu. We already had this problem last week, and we narrowly avoided the catastrophe, by installing some extra hard drives and moving bunch of archival data off the main network shares. The whole operation took several hours, but I managed to reclaim close to a 200 GB of space by deleting logs, dumping out garbage files and moving really, really old files that no one has touched in ages to a separate drive. Then I sent out an email to some of the administrative staff telling them what has been done, and advising them to do some more archiving. Last Thursday things seemed to be mostly under control.
I quickly log into one of the servers, to check how much of that rescued space was devoured over the weekend. Apparently all of it. The drive used for one of the network shares has exactly 57Kb of free space.
At that very moment, Jeremy bursts into the IT cave system. I have no clue what Jeremy does at our company, because I personally do not care. I am sure he had told me what his important job functions are at more than one occasion, but I have never considered that information to be relevant enough to actually commit neurons to it.
“Thank God you’re in! I know you just got here, but we are having some serious problems with the LAN so if you could look into that…”
I lift a finger to stop him from rambling.
“I know. I was just reading the tickets… Listen Jodie…”
One of the other things I have never commit to memory is Jeremy’s name. There is a limited number of neurons in my head, and now that I have shaved my beard I can no longer offload Unix skills to facial hair follicles so I tend to locally optimize storage this way. I just call him girl names for consistency.
“Oh, come on – that’s not even close!”
“Whatever. Listen, Sally Ann Margaret…” I pause for a second to see if he approves this new handle I just made up. He gives up, and allows me to continue.
“Remember last week when we moved the 1997 through 2009 folders off the main network share to the ‘Archive’ one?”
“That freed up around 200 GB of space. How the hell did you guys fill out 200GB over the weekend?”
He shrugs. “I don’t know. We’ve been scanning a lot of invoices, and other stuff…”
Last summer the corporate deities that bestow the blessed health insurance benefits upon us, have decided that it would be a good idea to scan everything in sight. This development coincided with a batch of new “all-in-one” printers we have installed near High Hrothgar – also known as the area where all the directorial critters have their offices. Someone has discovered that you can scan a document and have it emailed to you as a PDF. So naturally, their first reaction was to scan everything in sight. The initial feature shock, soon became a company policy but no one told anything to the IT department. So we have been oblivious that a few floors above there was a major digitization project going on, armed only with few aging Windows servers and replicated files shares to support it.
We only discovered the scanning project by accident, when we noticed the servers are low on drive space. After shaking up few users we managed to get them to spill the beans. Apparently they were not supposed to tell us, because their supervisors were afraid we will start making ruckus about buying new servers and shit.
So we went and put in a proposal to build a dedicated architecture for this project. Preferably some sort of a NAS to which we could slowly add new drives as the project grew in size. That got rejected immediately because it would actually cost money, versus the current solution of not doing shit that was completely free. Eventually we did get an approval to buy some new drives, and beef up the windows servers, but that was a temporary fix.
A fellow NOC Denizen, Larry actually made a chart. He measured the amount of storage we consume every day, adjusted it for possible growth and cross referenced that with the amount of internal storage we could possibly add to these servers. According to his math, we would get starved for space, and the system would become unsustainable within 3-4 years if we were lucky.
“Pshh, 5-7 years?” said the omipotent directorial oracles, who always think we are low-balling sustainability figures “In 7-8 years we might have flying cars, and colonies on Mars. Don’t you worry yourself about things that will happen 8-10 years from now.”
And so we were left with a teetering wreck just waiting to happen. In other words, business as usual.
I wander of to Larry’s desk. He is redditing like a pro, and ignoring the morning commotion surrounding the storage shortage. I don’t blame him. I would too, but they have found me first.
“Hey asshole, your math skills are shit.” I give him a friendly greeting.
Larry, a rotund gentleman, swivels around in his chair and gives me a shit eating grin:
“I will have you know, that my math skills are impeccable, you abominable cunt.”
Pleasantries out of the way, I give him the basic rundown of our current crisis. Something just ate 200GB over the weekend. Whatever it was, it was not included in our “Impending Storage Disaster of 2014″ projections (also known as “Peak Storage 2014″ for short). Now we need to identify it, and figure out how to deal with it. I had a very specific mission for Larry…
But more on that next time.