At the mercy of reviewers

Published on Wednesday, March 15, 2006 By Brad Wardell In GalCiv Journals

In the 15 years or so I've been making games I've found that there are three types of reviewers.   That's a gross over simplification of course but I'm a grossly oversimplified kind of guy.


The first type of reviewer is the drive-by reviewer.  If you're an indie developer, you are more apt to run into this. That's the review that makes you feel like, as you read it, that they were miffed having to take a time out from their first person shooter to play some little indie title's turn based strategy game.  We don't get these as much as we used to but we still get them. 

These guys scare me because I honestly don't feel like we get a really fair shot.  And I'm not saying they give only negative reviews. And that's the worst part. Because if you complain about the review when it was a positive rating you will get "What? We gave you an positive rating?" You can almost hear the "You make a great little shareware type game, I'm sure really hard core nerds get a thrill out of your little game."

But the drive by review can be negative too. If they run into any problem, don't expect a follow-up. These guys also live to put in snarky comments in their reviews -- nothing makes ya look cooler than to practice your written venom at the expense of some hapless developer.

But don't take my word for this. Go over to some time, think of the best indie games of all time, look at the reviews. Then go and think of some big budget drek that's come out and look at who gave them 90s or higher equivalents.

Casual Reviewers

The second type is the "Eh It looks like it has good production values, seems solid, haven't heard of it, looks good but won't give it better than 4.5/5 because if it was truly outstanding, they wouldn't be a small company." I've said this long before our latest game has come out as I followed the reviews of some truly outstanding and innovative games from small developers get an 80% on the same day they'll put up a 96 for some of the worst shovelware from some mega publisher.

The second type though isn't a drive-by review in that it's obvious they played the game decently. Moreover, these reviews are probably closest to what the overall average will be from both reader reviews and all sites combined. Rather, it's more like there's just a cap.  I don't really have an issue with these types of reviews because let's face it, it takes a long time to really review a game and we are a smaller developer. As long as we're not patronized I'm a happy camper.

The text to these reads like this "Great great great great great great." with a 4/5 rating.

I don't get worried as much about the second-type. I know we won't get super high reviews but I know we won't get really low reviews either as long as we've put together a solid title.

The Ultras

But it's the third type that scares me the most. The third type is the hard core reviewer. The person who is going to take your game apart. The ULTRA reviewer.

Ultra reviewers dissect a game to make sure it does what it's supposed to do. They know the genre, they know who plays them.  It's not enough to make a polished, solid game. Because they know that many gamers could care less about that. To them, the question is, is the game FUN?

These ultra  reviewers scare everyone equally though. Whether you're Microsoft or Stardock, they'll crush you like a bug equally. They're freelancers. Publishers have no power over them. Go ahead, threaten to take your ad dollars away. They'll laugh at you.

They may write for big sites or they may write for small sites. It doesn't matter. Their words carry weight because of years of built up integrity. If they give a big name game a 2 star rating, people take notice. And when they give out 5 star ratings it's equally a big deal.  It's not that they give a lot of negative scores, it's that they're immune to hype.

Ultras in action

QuarterToThree had a good article that dealt with this.  It links to one of Bruce Shelley's (designer of Age of Empires III) blogs.

Bruce wrote:

I am struck by the fact that the two biggest PC gaming magazines in the US gave Age 3 such disparate scores. Computer Gaming World gave it 60% (three stars out of five), while PC Gamer gave it 91%. That seems like it would be very confusing to readers who are considering whether to try the game. Players will establish eventually if the relative score of the game was at one of these extremes or the other, or somewhere in between. Data we’re seeing says that Age 3 has been perhaps the best selling PC game in the world since release in mid-October and that certainly suggests that the 60% score miss-represents the true achievement of the game.

No, what happened is that the CGW review was done by an Ultra and the PC Gamer one was not.  I'm not suggesting Age 3 "deserved" any particular score. That's not for me to say, I'm not a reviewer. Just that I, and others who follow this stuff know who the reviewers are and their reputations.

I'm also sure Bruce knows that game sales are not the sole indicator of a game's quality. As a game designer one is almost tempted to put together a crude formula for what determines sales: DISTRIBUTION X MARKETING X GAME QUALITY X DEMOGRAPHIC SIZE.

If my game were massively distributed, I wouldn't use sales as the thrust of my argument. Deer Hunter made the case that we shouldn't rely too much on sales as the sole determiner.

What he's pointing out is that the CGW review of Age of Empires III was not to his liking. The review (and CGM one) points out why they thought what they thought.

If he thinks that's rough, he should walk in our shoes where we could get the same rating for the sin of not being well known. After all, if we had a great game, wouldn't everyone already be hyped about it? That's the real reason games come with fancy opening cut scenes in case you've wondered. They help provide a clue as to a game's production values which is jokingly called "Driveby-begone" in some developer circles.

Ultras and Indies

Ironically, if you're an indie developer, these mega reviewers are your only chance of getting really high scores. And they don't give them out often as Ensemble observed. So talk about being in the shark tank.  In fact, they're far far more likely to give out negative reviews than any other type.

Let me be clear lest I give the impression that the mark of an Ultra review is someone who craps on a game. No. What I'm saying is that an Ultra reviewer is simply someone who knows the target audience -- that they are the target audience. And they don't care who you are. They care only about the game.

And anyone reading their review -- including the developers -- will observe and despair at the powerful evidence that justifies their opinion.  If your game is good, you'll get a great review and score. But if it's bad, it doesn't matter whether you're EA or Sony or whoever you're dead meat.

When Ultras follow up with developers, they can make the developer realize that they're the worst game designer in the world and that they should have stuck with Electrical Engineering and stayed the hell out of the whole "game thing". And that's even if it ends up a positive review.

Because no game is perfect. The question is purely how flawed the game is and how flawed is it compared to other games in its genre. But as a developer, the case Ultras can make is devastating. Design decisions you thought were sound can suddenly be called into question as they put forth rock solid arguments. Bugs you never heard of are suddenly obvious and easily reproducible.

Not all Ultras follow up. But Ultras tend to be the most thorough and careful. If they run into an issue or have a question, they're the most likely to contact the developers and get clarification.

GalCiv II reviews

Ultras put the fear into me I can tell you that. I'll admit, on GalCiv II, more so than our other games or products in the past, I've followed the posted reviews as well as I can. That's because this game is close to the best I can do.

I can see tons of tweaks here and there but realistically, unlike our other games, I don't look back and wish we could have done a lot of things differently. When we finished GalCiv 1, I wasn't overjoyed. I felt confined by the budget and resources and the MOO3 fans had convinced us we were about to get crushed. And I agreed with them. But not so with GalCiv II. Everything came together. 

So I'm particularly sensitive to the reviews.  Not so much because I expect a certain score as much as I'm worried deep down that I'll finally have proof that I am not the game designer I'd wanted to be. Reviews so far have been positive. There's been a sampling of all three kinds of reviews. And of course, few reviewers neatly fit into a particular category. Like I said, we're talking a gross oversimplification, sort of like ratings themselves.

I've dealt with the ultra reviewers and they're like the Babylon5 Inquisitor episode. They'll send in questions, point things out and by the time they're done, you not only suspect but know that your game is the worst game ever and that you wasted a lot of people's time having the audacity of releasing such junk. I tend to be fully prepared to agree with criticism. It's in my nature.

One of the ultra reviewers, Jeff Lackey (Gamespot), really liked the game. It was a huge relief. Admittedly, I was terrified before it came out. Like I said, these guys find flaws and design decisions that you might have previously thought were good ideas. But when they're done with you, you start tidying up your resume -- just in case of course...

Ultra reviewers finding flaws doesn't mean your game is doomed to a negative review. But when confronted with a design decision or a bug, it's hard not to start having second thoughts on what you've created. They follow-up. They ask questions. They point things out. And when you're done, you're jelly.  

Not all follow-up of course. Sometimes you'll just end up reading a review in which you can tell they understood what we were going for -- for good or bad.  The Ultras are the reasons some indie games get a peppering of very high scores (see Darwinia) and why some big named games get a peppering of not so high scores.

I'm aware of a couple of reviews that involve Ultras that are in progress. They aren't likely to see this post and even if they did, it would matter nothing. I could say "Reviewer X, your mother was a hamster and your father smelt of elderberries!" and it would have no effect.  The reviews may be very positive or very negative. They may cover minor game flaws or, depending on how many words they're allowed, may choose to focus on the key elements good and bad. It's hard to say. 

I plan to link to all reviews that at least seem like they gave us a fair shot. A fair review is all anyone can really ask for.

updated: Edited, cleaned up grammar, terminology.