Oxford: Critical Support Failure

Back in November I wrote a piece on mobile tools for college students. In it I named Google Drive to be one of the most useful resources available. Because of it’s ubiquity (work on every computer and mobile device) and ease of access (all you need is a browser) and robustness (Google never really goes down of looses user data) and the fact that it is free, it is almost a perfect cloud service for storing homeworks and term papers.

I actually recommend it to my own students as one of the ways they can avoid data loss, and never again be in the typical “dog ate my homework” situation. The use case I usually put forward to illustrate it is this: you can start writing your paper in a campus computer lab, then hit save, go back to the dorm and continue right where you left off. Then when you go home for the weekend, and forget to take your laptop your paper is still saved on the cloud, so you can safely finish it for Monday even without your computer. It is saved and backed up automatically, it has built in version control and you never have to bother with thumb drives or emailing yourself the right file ever again.

Not to mention that the build in collaboration tools make Google’s platform the perfect tool for work on group projects. And if there is anything college educators are fond of, it is forcing students to work together against their will.

Granted, the service is not without its problems – but the more jarring privacy concerns do not really apply to the purely academic use I recommend it for. For that, it is invaluable.

This is why I was shocked and surprised to find out Oxford University decided to permanently block all access to Google Docs and Google Drive services. Yep, that’s right – they blocked it at the firewall so you can’t use it unless you go through a proxy.

Why would they do such a thing? Because apparently pishing via Google Surveys service is a thing to do at Oxford these days:

Almost all the recent attacks have used Google Docs URLs, and in some cases the phishing emails have been sent from an already-compromised University account to large numbers of other Oxford users. Seeing multiple such incidents the other afternoon tipped things over the edge. We considered these to be exceptional circumstances and felt that the impact on legitimate University business by temporarily suspending access to Google Docs was outweighed by the risks to University business by not taking such action. While this wouldn’t be effective for users on other networks, in the middle of the working day a substantial proportion of users would be on our network and actively reading email. A temporary block would get users’ attention and, we hoped, serve to moderate the “chain reaction”.

If this seems ridiculous knee jerk reaction to you, that’s because it is. I would say this was a clear case of cutting of the nose to spite the face, but I think the truth here is much simpler. I think that this is merely an example of “I don’t use this service and therefore is useless” attitude that is unfortunately quite prevalent in the IT industry. Some BOFH decided to disable extremely popular service that has immense utility to both students and professors alike, on a whim and as a half-assed baind-aid solution to a prevalent problem.

Once could argue that because of the magnitude of the problem they had to do something, but a quick trip to their archives shows that they had this Google Survey problem for quite some time now. So its not like this was something new that just popped up and forced them to act. It was an ongoing issue that has been escalating over many months.

I do acknowledge the fact that there are no easy ways to deal with spam. Blocking Google Docs is something you could easily get away with on a corporate network. That’s because corporate IT typically provides users with the tools they need to do their job, and gets to define the allowed communication channels that meet company security policies, to the exclusion of all the others. The IT role in academia is a little bit different, and such a block seems like a gross overreach. Hell, my finely tuned bullshit detectors can actually detect hints of institutionalized apathy on the part of the IT crew, and executive meddling from the University officials that put pressure on the IT staff to do something about a problem they don’t understand.

The problem with this solution is that it merely scratched the surface of the problem. While Google Survey service makes Pishing easy and convenient it is nowhere near as prevalent as the more traditional approach of hosting the pages just about anywhere else on the internet. It also creates a worrying precedence that could turn into a policy. Why stop at blocking Google Survey? What if the spammers shift to Dropbox? Do you block that too? What if they use cheep Dreamhost accounts? Do you put the entire IP block owned by that company on the blacklist? The thing about spammers is that they are nothing if adaptable. They are keenly aware that their pish pages have a lifetime that can be measured in mere hours. Their schemes are designed to work with a revolving pool of attack pages. Blocking a single service is merely a temporary setback to them – one they can probably circumvent in about an hour (and that’s if they are lazy).

Oxford could have handled this issue differently. The less unorthodox and approach would have been aggressive server side filtering. I’m quite certain that automatically holding emails containing Google Survey links in quarantine would have been much less disruptive choice.

User education could have gone a long way too, but the blog post reveals exactly how they feel about that aspect of the job:

Now, we may be home to some of the brightest minds in the nation. Unfortunately, their expertise in their chosen academic field does not necessarily make them an expert in dealing with such mundane matters as emails purporting to be from their IT department. Some users simply see that there’s some problem, some action is required, carry it out, and go back to considering important matters such as the mass of the Higgs Boson, or the importance of the March Hare to the Aztecs.

Translation: our users are idiots, we tried educating them but it didn’t take, so fuck’em. If I was a professor at Oxford I’d be a little bit offended by these assertions. Granted, users do tend to be idiots. I often re-iterate this point on this very blog. But I would never dream of putting down my own user base like this in an official company blog post. It is just a tad bit unprofessional. You’d expect more from someone employed by a prestigious university such as Oxford.

If I’m reading the comment thread at Hacker News correctly, the block was lifted approximately 3 hours after the linked blog post went online. The reason? Massive outcry both from the students and instructors. As expected, Oxford community has embraced Google’s cloud service and incorporated it into their workflows. Many users suddenly found themselves cut of from their research data and unable to retrieve their class assignments.

What can IT professionals outside of Oxford learn from this?

Firstly, regardless of what you think of an application or service it might be invaluable to your users. It is never a good idea to disable or phase out software and services without first determining how it will impact your users. I learned that the hard way few years ago when my department tried to phase out old versions of office and get everyone on Office 2007. It was nearly a catastrophic failure, and not just because of the OOXML quirks. It turned out that a significant portion of our clients still relied on a the nearly forgotten Microsoft Binder application, which was shipped with Office 2000 and XP, and was available as an Add-In in Office 2003. Suddenly dozens of employees found themselves unable to open important, mission critical documents and had to be immediately downgraded in order to be able to do their work.

Secondly, regardless of how hard a project may seem, the best solution to a problem is the one that is least disruptive to the users. A perfect IT department is one that is virtually invisible to the end user. Banning and blocking of services should only be considered in extreme circumstances, and after making sure they won’t cause unnecessary disruptions. For example, blocking Facebook and Twitter to prevent people from using them on company time, or tweeting corporate secrets might seem like a great idea. But if your marketing department just launched a news super-important, viral social media initiative you are very likely to get in massive trouble unless you can blame the decision on someone with pointy hair.

Finally, while users are silly and at times act dumb it is never a good idea to make fun of their incompetence in official press releases and public blog posts. I’m not saying we should coddle them – they are responsible adults, and it is their own choice to remain willfully ignorant and to push themselves into irrelevance. But, who are we to judge them for that? Patronizing, passive aggressive comments in official documents can and likely will backfire and ruin your day.