What happened to the review scores?

You might have noticed that I stopped using the hreview format, and the 5 star scoring system for my reviews. I don’t know if you miss that part, but I decided to drop them. Why? Well, part of the reason is something that Shamus said in his blog really struck a chord with me. I can’t find the exact link at the moment, but he basically said that putting scores on reviews is completely meaningless and futile exercise which doesn’t really add any value to the review itself. I agree with that. To me, a review should be more of a more or less an objective in-depth discussion of the product in question, rather than arbitrary “awesome” or “lame” score.

In fact, the score is usually the most subjective part of the review – that’s the part where you make a judgment without any quantitative support. Or rather, the review should be a support for the score, but it is not always the case. I noticed that my personal scoring system had really no structure or guidelines. I would award stars based on how I felt that day, not based on how good was the object I reviewed. I went back to the archives and I noticed many discrepancies. For example I gave “End of Evangelion” a score of 4 because at the time I thought it was awesome and deserved a high score but I didn’t think it was absolutely perfect. So a 4 seemed right. Then at some other time I was reviewing some silly comedy and gave it a 3.5 because I thought it was a bit above the average. According to my scoring system the difference between an excellent, thought provoking anime that made my head spin for days and a forgettable average comedy was exactly half a star. WTF?

Naturally I went back and adjusted the scores so the End of Evangelion reviews are now at 5 stars, but this whole exercise made me realize how completely arbitrary this system was. And it’s not just me – I was flipping through the PC Gamer the other day and I noticed exactly the same issue in their reviewing system. What exactly is the difference between the score of 89% and 91%? You’d think that on a 100% spread this would be a completely insignificant minor difference, making the games almost equivalent in quality. But if you read the reviews the 89% is essentially a “great idea, very poor execution” review while the 91% is a “best game evar, but has few tiny bugs you can fix with a patch” one. You hardly ever see reviews that hand out scores below 50% anyway – the game must be incredibly bad for that. And yet you would think that statistically most of the games should score in the average 20-80 bracket with only few exceptional pieces lading in the top and bottom 20%. But if you look back through the issues, you will notice a completely different distribution – most games score above 80. Horrible games score just below 60. No one ever gets the score of 0% despite the fact that someone out there surely deserves it.

So it is a conceptual problem – review scores do not work. Shamus was right, and I was using a faulty model all this time. Hell, I can’t believe it took me so long to figure this out. Needless to say, I’m not going to use these arbitrary scoring methods anymore. If you liked them, let me know in the comments, but I doubt that I will bring them back.

Speaking of PC Gamer and video game reviews (and this is something that Shamus touched upon several times in his blog) I noticed that most of magazines on the market hardly do them anymore. If you flip open a random video game magazine these days, most of the volume is taken up by 4 page “exclusive previews” of new games which are usually nothing more than just thinly veiled press releases handed down from the distributors. Actual reviews are crammed in the back of the magazine, usually accompanied by as single thumbnail screenshot and score box, and spliced in a “4 reviews per page” grid.

I really feel it should be the opposite way – a review of a game should be 4 page long, with in-depth discussion of the gameplay, storyline and screenshots taken by the reviewers themselves depicting actual gameplay, menus and amusing graphical bugs and glitches they were able to find. Sadly, the only people who do this sort of thing these days are independent bloggers who get no money out of the whole thing. It seems that anyone making money out of game reviews (and this includes popular blogs) becomes a “previewer” concerned more with generating hype for a given game, than actually writing critical reviews. Sigh…

[tags]reviews, video game reviews, pc gamer, hreview, review scoring system[/tags]