Game Ratings and MMO's: A Response To Dan Stahl

Reviewing MMOs isn’t an easy thing. If you play for weeks and weeks, go through everything the game has to offer and end up with something like “the game is terrible until you reach hour 30,” you’re completely missing the demographic a review is made for – newbie players. If you spend too little time, you run the risk of missing some major feature that may or may not change your mind about the whole thing entirely. It’s a fine balance, and one that – when you have deadlines, multiple jobs, family and general life stuff – is incredibly easy to screw up. A suggestion from Star Trek Online producer Dan Stahl that review scores should be ever changing to represent the constant updates, fixes and expansions released for new games just makes the little reviewer inside me shrivel up and cry.

A typical MMO reviewer will tell you: "I’ve hardly left this room in three months, my girlfriend doesn’t recognise me thanks to my two foot beard and I’ve been reported as missing by friends and relatives at least twice."

And he’s saying it’s all for nothing?!

Rating by Release

For those of you that didn’t see Stahl’s quote from the interview with thealist, here it is:

"In my opinion, the whole game rating business doesn't necessarily do a great justice to MMOs. MMOs are designed to grow over time and get better with every major release. It might be better if sites like Metacritic could find a way to rate MMOs by releases instead of just the initial day one. There are plenty of MMOs that have made huge strides since days one and some that have even gotten worse.”

And I see his point, I really do. Games – not just MMOs – are designed today so that they can be changed after release. This is sometimes a good thing, sometimes a bad thing. On the bright side, there aren’t instances where an entire run of a certain game is released with a glitch that stops the player actually finishing the game – it happened more than once during the SNES/PS1 era and sometimes the game wasn’t actually fixed in latter prints either. On the other hand, Resistance 3 was released alongside a patch that was nearly a GB in size, and then a second patch of equal size was released the day after. Not the reaction you want after you’ve rushed home with a new game.

Changing a Score

An MMO’s success relies on constant upgrades and fixes being implemented so that casual players aren’t put off by glitches and hardcore players have plenty to do. The problem is, many casual players probably don’t see the effects of an update very often. Their character’s final big move being nerfed isn’t a big problem for them, because they’ve never seen the final big move. A bunch of end game content being released is equally as useless to them, because they’re never going to get to the end game. A game reviewer seriously can’t become a hardcore player of any game, usually, because, firstly, they’ve seen it all before and, secondly, there just isn’t the time in the day.

Over the last 12 months I’ve played nearly 50 MMOs. Implementing what Stahl is suggesting would mean a more accurate review structure, but it’d also mean keeping up with all 50 of those MMOs. Even the top 10 would require more time and energy than all but the most hardened geek is willing to put in.  And, ultimately, to change the game’s score from a 7 to a 7.1 because a quest has been fixed is ridiculous. Nobody in their right mind is going to say “well, I didn’t like the looks of it as a game that scores 8/10, but because they’ve added a new country and fixed the controls I’ll definitely play it as an 8.5.”

You're gonna change that rating! Do you hear me!

Who Would Benefit Then?

The only people for whom an ever-changing review score is a definite positive are developers who have released a buggy game. Or who got bad reviews at launch but feel enough changes have been made to warrant a fresh review. Every 18 months or so, there should be new reviews of old MMOs, so long as there have been changes to warrant it. Over the past few months, we’ve re-reviewed a couple of things, both to fit with our new review style/video reviews, but also because games evolve. A bad developer shouldn’t be rewarded for fixing major glitches they launched with though, and that negative review should stay prominent on every site until the game has enough new content to warrant a re-review.

Is There a Better Solution?

On each of our reviews, there’s a chance for our readers to give a game a rating based on what they feel the game deserves. 764 people have told us what they thought of World of Tanks, and a review sample of that size is going to be fairly accurate. Similarly, Metacritic’s user section shares thoughts and comments from a whole cross-section of people, some of them even literate. It may come to a point where the numbers hardly change, but the evolution of people’s thoughts certainly do and it’s always interesting to read the user section after a new major update is released for an old game.

It's all fun! Who cares about Ratings!?!

First Impressions

For humans in general, first impressions matter. The game can be said for video games. There are people who refuse to play Resident Evil 6 because they’re under the impression there’s no horror elements. It doesn’t matter what they read, what videos they see – those people aren’t going to buy that game (no, they’ll just bitch about it on forums). If your MMO isn’t fun at launch, if there are people complaining about major problems, review scores should be the least of your worries. Fixing the game is where you should be looking.  I suppose with a game on the scale of Star Trek Online (which I believe we gave a 9 in our Star Trek Online review, by the way), looking back at bad review scores can be frustrating, because the game isn’t like that anymore. But how much leeway should we give to developers? ( Read more about MMO developers re-inventing the wheel-> ) If you released a game that received multiple fives and sixes, why did you do that? The problem isn’t long lasting reviews, but long lasting memories.

Conclusion

I see Stahl’s point, I really do. Metacritic offers little more than a score and an often glib one-liner meant to tease a user into clicking. It’s unchanging, and checking the Star Trek Online page in a decade’s time will offer many of the same reviews that are there now. That’s not fair on one level, because new players could potentially be put off by a review that’s three years out of date. On another level though, it happened – the game released and it was less than perfect. We can’t tear articles out of magazines, we can’t delete angry forum posts and we can’t change old scores.
I don’t think the score system is ideal for MMOs, but asking us to change it is asking us to forget what a game was. And if we start doing it on a regular basis, it won’t be long before developers are asking us to review a game based on what it COULD be (I’ve been asked to ignore glitches in the past, on the promise of a patch that never came). And that’s when the score system will really start to crack.