What happened to the review scores?

You might have noticed that I stopped using the hreview format, and the 5 star scoring system for my reviews. I don’t know if you miss that part, but I decided to drop them. Why? Well, part of the reason is something that Shamus said in his blog really struck a chord with me. I can’t find the exact link at the moment, but he basically said that putting scores on reviews is completely meaningless and futile exercise which doesn’t really add any value to the review itself. I agree with that. To me, a review should be more of a more or less an objective in-depth discussion of the product in question, rather than arbitrary “awesome” or “lame” score.

In fact, the score is usually the most subjective part of the review – that’s the part where you make a judgment without any quantitative support. Or rather, the review should be a support for the score, but it is not always the case. I noticed that my personal scoring system had really no structure or guidelines. I would award stars based on how I felt that day, not based on how good was the object I reviewed. I went back to the archives and I noticed many discrepancies. For example I gave “End of Evangelion” a score of 4 because at the time I thought it was awesome and deserved a high score but I didn’t think it was absolutely perfect. So a 4 seemed right. Then at some other time I was reviewing some silly comedy and gave it a 3.5 because I thought it was a bit above the average. According to my scoring system the difference between an excellent, thought provoking anime that made my head spin for days and a forgettable average comedy was exactly half a star. WTF?

Naturally I went back and adjusted the scores so the End of Evangelion reviews are now at 5 stars, but this whole exercise made me realize how completely arbitrary this system was. And it’s not just me – I was flipping through the PC Gamer the other day and I noticed exactly the same issue in their reviewing system. What exactly is the difference between the score of 89% and 91%? You’d think that on a 100% spread this would be a completely insignificant minor difference, making the games almost equivalent in quality. But if you read the reviews the 89% is essentially a “great idea, very poor execution” review while the 91% is a “best game evar, but has few tiny bugs you can fix with a patch” one. You hardly ever see reviews that hand out scores below 50% anyway – the game must be incredibly bad for that. And yet you would think that statistically most of the games should score in the average 20-80 bracket with only few exceptional pieces lading in the top and bottom 20%. But if you look back through the issues, you will notice a completely different distribution – most games score above 80. Horrible games score just below 60. No one ever gets the score of 0% despite the fact that someone out there surely deserves it.

So it is a conceptual problem – review scores do not work. Shamus was right, and I was using a faulty model all this time. Hell, I can’t believe it took me so long to figure this out. Needless to say, I’m not going to use these arbitrary scoring methods anymore. If you liked them, let me know in the comments, but I doubt that I will bring them back.

Speaking of PC Gamer and video game reviews (and this is something that Shamus touched upon several times in his blog) I noticed that most of magazines on the market hardly do them anymore. If you flip open a random video game magazine these days, most of the volume is taken up by 4 page “exclusive previews” of new games which are usually nothing more than just thinly veiled press releases handed down from the distributors. Actual reviews are crammed in the back of the magazine, usually accompanied by as single thumbnail screenshot and score box, and spliced in a “4 reviews per page” grid.

I really feel it should be the opposite way – a review of a game should be 4 page long, with in-depth discussion of the gameplay, storyline and screenshots taken by the reviewers themselves depicting actual gameplay, menus and amusing graphical bugs and glitches they were able to find. Sadly, the only people who do this sort of thing these days are independent bloggers who get no money out of the whole thing. It seems that anyone making money out of game reviews (and this includes popular blogs) becomes a “previewer” concerned more with generating hype for a given game, than actually writing critical reviews. Sigh…

[tags]reviews, video game reviews, pc gamer, hreview, review scoring system[/tags]

This entry was posted in meta and tagged . Bookmark the permalink.



4 Responses to What happened to the review scores?

  1. Shamus UNITED STATES Internet Explorer Windows says:

    Great post.

    I give it a 4.5 out of 5 stars.

    :)

    Reply  |  Quote
  2. Luke Maciak UNITED STATES Mozilla Firefox Windows Terminalist says:

    LOL!

    Added humor: I almost mechanically deleted the post from the moderation queue due to the high volume of “hey, great post – I think you should expand it and buY V14gR4!” spam. :mrgreen:

    Reply  |  Quote
  3. Matt` UNITED KINGDOM Mozilla Firefox Windows Terminalist says:

    hmm.. if the magazines making money are all stuffed full of previews and have crappy reviewing, and the blogs with proper reviewing instead of previews aren’t making money, then I guess there must be something preventing the good thing from profiting over the bad thing.

    It could be that a very large market segment is stupid, and wants to go “OMG lookit this thing coming out in 6 months, those pre-rendered not-in-game graphics look hawt, will buy”. Whilst the rest of us are rolling our eyes and checking out the blog reviews. But that seems uncharitable – you have to hope that gamers as a whole have a high enough average intelligence that this wouldn’t be the case.

    I’m going to guess it’s advertising related… or possibly that the magazines are actually paid to generate hype about the upcoming game in question. Either way it seems kinda corrupt…

    Reply  |  Quote
  4. Luke Maciak UNITED STATES Mozilla Firefox Windows Terminalist says:

    Precisely. Here is the kicker: who advertises in video game magazines more than anyone else? Video game distributors – the same people whose products get reviewed in these things. Can you say “conflict of interest?”

    If the site or magazine pumps out a really bad review, a game maker may actually consider pulling their ads, and seriously reconsider ever letting that magazine do an “exclusive preview”. It has already happened btw, and a GameSpot reviewer got canned to appease Edios who didn’t like his bad Kane and Lynch review.

    It’s sad but true – if a magazine antagonizes a game maker, they might be jeopardizing their advertising revenue stream.

    Reply  |  Quote

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>