We are out of space: Part 1

It’s Monday morning, and I’m sick. I thought that I was smart when I got my flu shot a few months ago. I figured that I can cock-block the influenza virus, and skate through the winter unscathed. What I did not anticipate was that being free of flu I was a prime target for things that are much worse. One of these super-bugs that makes you congested, and makes swallowing feel like gargling with sandpaper got the best of me. I would have stayed home, but I don’t have sick days.

In their infinite wisdom, the powers that be decided that staying home sick counts as a “vacation” so I wisely opted to drag my sneezing, germ producing carcass to work, making it my mission to spread the malignant disease as far and wide as possible. My hope was that I was going to be able to burrow myself in my cubicle, put some code on the monitor so it looks like I’m programming and then just survive till 5pm battling fever and sneezing fits.

Alas, this was not to be. As soon as I sit down and open my email, support ticket notifications start streaming in:

  • Ticket #6451 [High]: CAN”T SAVE AANYTHING!!1
  • Ticket #6452 [Medium]: G Drive not working
  • Ticket #6453 [Critical]: facbook.com saiz page not fund – pls unblock!
  • Ticket #6454 [High]: Can’t save to LAN.
  • Ticket #6455 [High]: When trying to save to G: drive it says “Drive full or right protected”
  • Ticket #6456 [Medium]: Is the public network share full? Can’t save anything.
  • Ticket #6457 [Low]: Can’t save to network share. Probably over quota. Please extend.

It goes on like this for at least 20 more messages. The emerging pattern is somewhat clear – something is terribly wrong with the network shares. I’m fraught with a sense of déjà vu. We already had this problem last week, and we narrowly avoided the catastrophe, by installing some extra hard drives and moving bunch of archival data off the main network shares. The whole operation took several hours, but I managed to reclaim close to a 200 GB of space by deleting logs, dumping out garbage files and moving really, really old files that no one has touched in ages to a separate drive. Then I sent out an email to some of the administrative staff telling them what has been done, and advising them to do some more archiving. Last Thursday things seemed to be mostly under control.

I quickly log into one of the servers, to check how much of that rescued space was devoured over the weekend. Apparently all of it. The drive used for one of the network shares has exactly 57Kb of free space.

At that very moment, Jeremy bursts into the IT cave system. I have no clue what Jeremy does at our company, because I personally do not care. I am sure he had told me what his important job functions are at more than one occasion, but I have never considered that information to be relevant enough to actually commit neurons to it.

“Thank God you’re in! I know you just got here, but we are having some serious problems with the LAN so if you could look into that…”

I lift a finger to stop him from rambling.

“I know. I was just reading the tickets… Listen Jodie…”

One of the other things I have never commit to memory is Jeremy’s name. There is a limited number of neurons in my head, and now that I have shaved my beard I can no longer offload Unix skills to facial hair follicles so I tend to locally optimize storage this way. I just call him girl names for consistency.

“Jeremy.”

“Julie…”

“Jeremy.”

“Genvieve…”

“Oh, come on – that’s not even close!”

“Whatever. Listen, Sally Ann Margaret…” I pause for a second to see if he approves this new handle I just made up. He gives up, and allows me to continue.

“Remember last week when we moved the 1997 through 2009 folders off the main network share to the ‘Archive’ one?”

He nods.

“That freed up around 200 GB of space. How the hell did you guys fill out 200GB over the weekend?”

He shrugs. “I don’t know. We’ve been scanning a lot of invoices, and other stuff…”

Last summer the corporate deities that bestow the blessed health insurance benefits upon us, have decided that it would be a good idea to scan everything in sight. This development coincided with a batch of new “all-in-one” printers we have installed near High Hrothgar – also known as the area where all the directorial critters have their offices. Someone has discovered that you can scan a document and have it emailed to you as a PDF. So naturally, their first reaction was to scan everything in sight. The initial feature shock, soon became a company policy but no one told anything to the IT department. So we have been oblivious that a few floors above there was a major digitization project going on, armed only with few aging Windows servers and replicated files shares to support it.

We only discovered the scanning project by accident, when we noticed the servers are low on drive space. After shaking up few users we managed to get them to spill the beans. Apparently they were not supposed to tell us, because their supervisors were afraid we will start making ruckus about buying new servers and shit.

So we went and put in a proposal to build a dedicated architecture for this project. Preferably some sort of a NAS to which we could slowly add new drives as the project grew in size. That got rejected immediately because it would actually cost money, versus the current solution of not doing shit that was completely free. Eventually we did get an approval to buy some new drives, and beef up the windows servers, but that was a temporary fix.

A fellow NOC Denizen, Larry actually made a chart. He measured the amount of storage we consume every day, adjusted it for possible growth and cross referenced that with the amount of internal storage we could possibly add to these servers. According to his math, we would get starved for space, and the system would become unsustainable within 3-4 years if we were lucky.

“Pshh, 5-7 years?” said the omipotent directorial oracles, who always think we are low-balling sustainability figures “In 7-8 years we might have flying cars, and colonies on Mars. Don’t you worry yourself about things that will happen 8-10 years from now.”

And so we were left with a teetering wreck just waiting to happen. In other words, business as usual.

I wander of to Larry’s desk. He is redditing like a pro, and ignoring the morning commotion surrounding the storage shortage. I don’t blame him. I would too, but they have found me first.

“Hey asshole, your math skills are shit.” I give him a friendly greeting.

Larry, a rotund gentleman, swivels around in his chair and gives me a shit eating grin:

“I will have you know, that my math skills are impeccable, you abominable cunt.”

Pleasantries out of the way, I give him the basic rundown of our current crisis. Something just ate 200GB over the weekend. Whatever it was, it was not included in our “Impending Storage Disaster of 2014″ projections (also known as “Peak Storage 2014″ for short). Now we need to identify it, and figure out how to deal with it. I had a very specific mission for Larry…

But more on that next time.

This entry was posted in sysadmin notes and tagged , . Bookmark the permalink.



16 Responses to We are out of space: Part 1

  1. Liudvikas LITHUANIA Mozilla Firefox Windows Terminalist says:

    Not surprising, 200gb is what I call – “Hey look at that giant ebook pack I saw on thepiratebay, I’m going to download it for shits and giggles even if I don’t care about 99.9% of those books.”
    I no longer comprehend how I lived without 100mbit internet and with 20gb hard drive just under a decade back.

    Reply  |  Quote
  2. Luke Maciak UNITED STATES Google Chrome Linux Terminalist says:

    200 GB of ebooks is a bit on the large side though – even if we are talking scanned PDF’s of full color books (like RPG manuals for example) they are usually under 50MB in size, and some of these that are OCR’d text compress very well.

    It’s more common to see “adult entertainment” torrents of that size in the form of hi-def video “site-rips” or “mega-packs”. :P

    Reply  |  Quote
  3. Liudvikas LITHUANIA Mozilla Firefox Windows Terminalist says:

    I actually saw fiction pack weighing 133gb on piratebay, obviously it’s not a tidy epub only pack, but still.

    Reply  |  Quote
  4. Liudvikas LITHUANIA Mozilla Firefox Windows Terminalist says:

    Btw, what’s your preferred ebook format?

    Reply  |  Quote
  5. Victoria Netscape Navigator Mac OS says:

    No sick days? OMG, it’s terrible! I’m so happy we are allowed to work from home. I just Dropbox all the files and that’s that.

    I juggle 2 drives at home: 500 GB on my machine and 2TB that works like 1TB mirror drive and I run out of space ALL the time.

    Reply  |  Quote
  6. Liudvikas LITHUANIA Mozilla Firefox Windows Terminalist says:

    @ Victoria:
    Welcome to United States of getting raped by corporations. No holidays or sick days allowed! I have no idea what lawmakers thought, but there it is.

    Reply  |  Quote
  7. Luke Maciak UNITED STATES Google Chrome Linux Terminalist says:

    @ Liudvikas:

    I don’t really have a preferred format. Whatever works on teh Kindle is fine by me.

    @ Victoria:

    Yeah, no paid sick days is fairly typical for US. In fact the whole concept of “vacation days” is considered an optional perk for your benefits package. That said, I could theoretically work from home if I had to (unless something physically breaks that is). Sometimes that’s what they do when we get snowed in – they just forward your office phone to your cell, and you work from home. :)

    @ Liudvikas:

    Well, we do get national holidays off. Only the big ones though. Today is President’s day for example, and a lot of people have off. Most people I know are working though. So for me all it meant was less traffic on the road during the morning commute.

    Wait… Is it still rape if we get paid for it? :P

    Reply  |  Quote
  8. Liudvikas LITHUANIA Mozilla Firefox Windows Terminalist says:

    @ Luke Maciak:
    Yes, it is.
    It’s not like you can say – screw you guys I’m going home. You need money, so the choice is either get raped or move to europe.

    Reply  |  Quote
  9. Luke Maciak UNITED STATES Google Chrome Linux Terminalist says:

    Liudvikas wrote:

    Either get raped or move to europe.

    That should be the advertising slogan of our new “European Travels” agency. :P

    Reply  |  Quote
  10. Liudvikas LITHUANIA Mozilla Firefox Windows Terminalist says:

    @ Luke Maciak:
    I’m in! But you know there’s just so much raping we could accomplish by ourselves. We would need a lot of helpers if we intend to rape everyone not moving to Europe. I’m a little bit discouraged by the expenses that such expansive work force would entail.

    Seriously though, why leave Poland? It’s a bit too religious for my taste, but it’s still Europe.

    Reply  |  Quote
  11. Luke Maciak UNITED STATES Google Chrome Linux Terminalist says:

    @ Liudvikas:

    Let’s outsource raping to India. That should cut the costs. :P

    Why leave Poland? It was mostly random. It was late 90′s, and our name was pulled out of the hat in the Green Card Lottery. It meant I had a shot at schooling and career in the big US of A which back then was the mighty super-power that had a booming economy, and was at the forefront of all this crazy internet stuff. It was an exciting opportunity that I did not want to miss. And truly it was exciting. I remember a senior in HS here in US and watching the economy explode upwards as the .com bubble inflated it bigger and bigger. It was awesome being young, naive and watching this amazing technological revolution sweeping the nation, changing the way people did business. Then of course the bubble burst around the time when I was about the graduate college with my CS degree. Fun times!

    The truth is that I really like it here – I don’t see myself going back. I think I made out pretty well – I’m not sure if I would have gotten the education I did back there. The whole thing about moving, starting over in a new country motivated me to work hard. I was a C student in Poland, but when I came here I became a straight A student. If you told the 16 year old me that he would be teaching at a university as a part time job, he would probably not believe you.

    As for Poland being religious – it’s fairly relative. It’s mostly a cultural thing – majority of people are Catholic in a cultural sense. We share customs, traditions and religious holidays – but most people (especially in big cities like Warsaw) are quite liberal and not fervently religious or conservative.

    Reply  |  Quote
  12. Liudvikas LITHUANIA Mozilla Firefox Windows Terminalist says:

    @ Luke Maciak:
    That might work if you figure out how to rape people over the phone.

    I wonder how much school programs differ in USA, maybe your straight A’s were just the result of easier schooling :)

    As for religiousness – the previous pope was greatly appreciated by the Polish. Perhaps it’s just nationalistic pride (which is even worse), it smells bad in either case. Of course USA is worse in both counts, so it’s no big deal.

    Reply  |  Quote
  13. icebrain PORTUGAL Mozilla Firefox Linux Terminalist says:

    Great post; I’ve gone back and forth between smiling and rolling my eyes ;) And the cliffhanger is killing me!

    Liudvikas wrote:

    @ Luke Maciak:
    Yes, it is.
    It’s not like you can say – screw you guys I’m going home. You need money, so the choice is either get raped or move to europe.

    Well, for some values of europe. Right now I wouldn’t move to, say, Greece or here (Portugal).

    We are supposed to get sick days, but considering that around 20% IIRC of our workforce doesn’t have any contract at all – like freelancers, except they have all the disadvantages of a contracted worker – that doesn’t mean much. We, in a democratic time, have managed to remove all the working rights our grandparents fought for during the fascist regime. Quite an accomplishment.

    Reply  |  Quote
  14. icebrain PORTUGAL Mozilla Firefox Linux Terminalist says:

    That sounded bad. I meant rolling my eyes to your managers’ assertions, not to the post!

    Reply  |  Quote
  15. Mitlik UNITED STATES Internet Explorer Windows says:

    Who stops there?

    Also, please don’t let this be another dead ended stub like Fan Day

    Reply  |  Quote
  16. Luke Maciak UNITED STATES Google Chrome Linux Terminalist says:

    @ icebrain:

    Heh! I figured that. :)

    @ Mitlik:

    I do intend to finish that one sometime soon. I just ran out of inspiration and did not feel like writing more about that clusterfuck day. But yeah, I do have half-finished draft for part 2 and an outline for part 3. They just need some work, and I have been procrastinating. :)

    Reply  |  Quote

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>