Metacritic – What’s it good for?

They’ve been thrown about quite a bit, usually to identify Game A as being superior to Game B, but is there any merit to these claims?

First off, there have been numerous criticisms of how metacritic collects and converts scores. I’m not going to get into that right now, but you can have a look at their system here - http://www.metacritic.com/a... just wanted to point it out to anyone interested.

What I want to point out today is this:

There is no scientific basis upon which you can use metacritic in its present state to scientifically and objectively compare ANY 2 games.

Metacritic scores for different games are usually collected from different websites. Even when they are collected from the same website, they are usually written by different staff members.

So that’s the problem. Can you use scores written by different people to compare games? The answer is yes, but it involves a bit of science and statistics.

Stop. Stop right there. Do not close tab due to anticipated boredom. I’ll keep it simple. I promise.
Here’s one example of how you could do it. There are probably better ways.

1. You identify the critics you want to use. Hopefully, Jim Sterling is not one of them.

2. You give each of these critics the same group of games to score.

3. Then, you rank the scores. Eg if they were given 10 games, rank 1 describes the best in the lot, rank 10 describes the worse.

4. Once you’ve done that, you test for correlation between ranks of reviewers. For example, you can use the Spearman’s Rank Order Correlation. This will tell you if there is agreement between reviewers and thus, if their ranks are comparable now and, arguably, in the future.

I say arguably because a reviewer can review in a predictable way for the test and then proceed to launch utter flamebait in the future, but you get the idea.

Whether or not this is the best way or even a valid way is not really the point I’m trying to make. My point is this:

Comparison - true, accurate, unbiased comparison, takes work and effort. Work which is not done by Metacritic.
At the end of the day, the Metascore is a good indicator of game quality - which is what it was designed for.