Steam Gauge: Do strong reviews lead to stronger sales on Steam?

As someone who reviews games for a living, I’d like to think that the critical consensus on a game has some correlation with that title’s success in the market. That would imply that we critics are pushing consumers to buy games that we consider worthy and to stay away from those we don’t like. Alternatively, it could mean that we’re simply good at identifying with our audience’s tastes, predicting through our reviews which games will appeal to the gaming audience and which ones won’t.

Proving this kind of correlation is generally pretty tough, though, thanks to limited public sales data in the gaming space. But that hasn’t stopped some from trying. At the 2008 DICE summit, Activision Vice President of Marketing Robin Kaminsky went so far as to suggest that “For every additional five points over an 80 percent average review score, sales may as much as double.” (Though some have questioned that statement’s accuracy.) Video game market research firm EEDAR tried tying Metacritic averages for the top 10 games by publisher to financial performance back in 2009 and found mixed results. Individual developers have gone so far as to blame Metacritic for their financial problems following troubled game launches.

Now, thanks to our recently unveiled Steam Gauge project, we have another way to try to tie down the relationship between a game’s critical reception and its sales success. In comparing estimates of sales on Steam to aggregate review score averages, we found that better reviews do generally translate to more sales for games, especially at the top end of the critical spectrum. That said, there is a lot of variability in the performance of individual games, and a prevalence of good or bad reviews is far from a guarantee of sales success or failure, respectively.

The usual caveats apply before we get started with the analysis. All the data in this piece represents sales estimates based on a sampling of public data, as outlined in our initial Steam Gauge piece. It’s worth mentioning again that some games listed here are sold through platforms and methods other than Steam, which may skew individual results a bit.

We’re also using a game’s Metacritic score as a general measure of the critical consensus, though we’re well aware that this metric has its fair share of critics. Plenty of reviews are not included in Metacritic’s ratings (including Ars Technica’s unscored reviews), and the site uses secret outlet weightings to skew its reported scoring averages. Not every game receives the four necessary reviews to be listed on Metacritic, either—this analysis is looking only at the 1,300 or so games listed on Steam that do have a Metacritic score.

All that said, Steam itself lists the Metacritic rating on its Store pages (a move recently aped by Amazon), and publishers routinely use Metacritic numbers to measure success internally, for good or for ill. For those reasons, we feel Metacritic ratings serve as a good enough quantitative measure of the very qualitative process of evaluating games.

So how does a game’s Metacritic score relate to estimated Steam sales? Pretty erratically, as it turns out. As you can see from the above chart, the individual, game-level relationship between estimated Steam sales and Metacritic rating is incredibly noisy. Games with all kinds of Metacritic scores can fall all over the spectrum of sales results on Steam—a game with an 80 on Metacritic, for instance, is just as likely to bring in about 56,000 estimated Steam sales (at the 25th percentile) as it is to earn 375,000 estimated Steam sales (at the 75th percentile).

To make sense of all this noise, it helps to take the median sales performance at each Metacritic score level, as shown above (we cut the graph off at a Metacritic rating of 50 here because the data is just too sparse below that). Here, you can see an unmistakable upward trajectory in median sales as review scores increase, especially as those scores run through the 80s and 90s. The median game that receives a score of 70 on Metacritic sells about 89,000 estimated copies on Steam. At a Metacritic score of 80, the median sales go up to about 182,000. At a Metacritic score of 90, the median game skyrockets to just over 803,000 copies on Steam. The same upward trends can be seen at other sales percentiles over the Metacritic range.

Yet even this median data is pretty noisy—somehow, games that earn a 64 on Metacritic (median: about 144,000 estimated Steam sales) tend to sell much better than games that earn an 82 (median: about 76,000 estimated Steam sales). For some reason, games with a Metacritic score of 89 tend to sell worse than those with scores between 83 and 88. In general, an increase of a single Metacritic point can have a negligible or even negative effect on the average game’s performance, especially in the vast swath of games scoring in the 70s, where a plurality of rated Steam titles live.

When you aggregate games together into 10-point Metacritic score ranges, however, the larger trend becomes unmistakable. The median game with a score of 90 or more on Metacritic will sell 50 times as well as the median game that scores less than 30, according to our estimates. The general increase in sales performance isn’t uniform across the Metacritic range, though—a game with a Metacritic score in the 40s can expect the same 40,000 to 50,000 median sales as a game that scores in the 60s.