The spotty death and eternal life of gaming review scores

Since before GamePro started using cartoon faces to measure “fun factor” back in the ’80s, review scores have been an integral part of the game industry. But there are a few recent signs that the practice of sticking a number or a letter grade at the end of review text may be losing some of its appeal. Among the largest of those signs: popular gaming hub Eurogamer announced earlier this week that it’s dropping its long-standing ten-point review score scale in favor of a vaguer “recommendation” system.

While Eurogamer’s editors say review scores have always been somewhat reductive and antithetical to nuance, they now argue that the concept of scoring is particularly ill-suited to the way games are made and released today. “How should we score an excellent game with severe networking issues?” Eurogamer asked rhetorically. “A flawlessly polished game with a hackneyed design? A brilliantly tuned multiplayer experience with dreadful storytelling? If you expect the score to encompass every aspect of a game, the task becomes an exercise in futility. Add an inflated understanding of the scoring scale in many quarters—whereby 7/10 and even sometimes 8/10 are construed as disappointing scores—and you have a recipe for mixed messages.”

Eurogamer isn’t alone in this decision. Before Joystiq regrettably folded last week, the site made a similar decision to throw out review scores last month. “Between pre-release reviews, post-release patching, online connectivity, server stability and myriad other unforeseeable possibilities, attaching a concrete score to a new game just isn’t practical,” Joystiq’s Richard Mitchell wrote of that decision. “More importantly, it’s not helpful to our readers.” Last year also saw a few smaller outlets turning against the gaming review score: TechnoBuffalo in September and GameXplain in April.

The idea of adding a score or grade to the end of a review isn’t unique to games, of course. Reviews of music, movies, TV shows, and even tech gadgets routinely sum up the text with a number, a set of stars, a letter grade, or some other shorthand for classifying the work in relation to its peers. But video games are somewhat unique in just how much focus is put on these scores. Gamers pore over relative scores to the point of obsession, arguing over whether one game really deserved a better score than another or what various scores really mean ad infinitum.

“When I read through the comments on an IGN review, for example, all I see is people talking about the score,” outspoken scoring critic and Kotaku News Editor Jason Schreier told Ars. “Compare that to, say, comments on an [unscored] review from Kotaku or Rock Paper Shotgun, and it’s night and day. I think a lot of people really do want to read and talk about good criticism; we’ve just created an atmosphere where numbers drive the conversation instead. We can fix that!”

For Schreier, sites deciding to finally drop review scores is a burgeoning trend that’s been a long time coming. “[Scores] strip the nuance from video game criticism, forcing reviewers to stuff games into neat little boxes labeled ‘good’ and ‘bad,’” Schreier said. “Sometimes that’s OK—Aliens: Colonial Marines, for example, is pretty clearly a Bad Video Game—but many games are way too interesting to sum up so simply.”

“Take Destiny, for example,” he continued. “Destiny is both brilliant and infuriating, both simple and complex, both polished and unfinished… It’s one of 2014’s most interesting games. It’s something we’ll be discussing for years to come. And it’s a major disservice for gaming outlets to assign it 6s and 7s as if its merits and flaws can be summed up with a single ‘average’ number.”

Modifying the scoring system

After starting at a 9.5, Polygon’s SimCity review score currently sits at 6.5 after three updates.

Credit: Polygon

Against all of those arguments, pro-score editors can point to one simple fact: readers overall seem to like reviews that have scores. When discussing whether to use review scores at Polygon, Reviews Editor Arthur Gies said the site’s founding staff drew on “a lot of institutional history” around the practice. “We had access to peers at other outlets who had gotten rid of review scores after having them who saw the interest in their review content suffer considerably, and sites who added review scores and saw a commensurate increase in interest in their reviews. … The anecdotal accounts and experiences we had suggested that readers want them, whether they admit to it or not.”

To address arguments that static scores are no longer equipped to handle games that can often change significantly after release, Polygon introduced a review update system that allows for scores to be modified over time. So while Polygon scored a game like the 2013 SimCity as a 9.5/10 at release, that score dipped down to a 4/10 when the game faced horrendous server issues after launch. Polygon’s score went back up to a 6.5 a few weeks later, as servers became more reliable but the “Cheetah speed” option remained disabled.

Gies had mixed feelings about how effective the update system has been at responding to the many troubled game launches of late. “As for score updates, I think we had stars in our eyes, or maybe just couldn’t anticipate the developments of the last two years regarding the state of games at launch,” he said. “We were primarily projecting that updates would account for evolution, not broad systemic failure. And clearly, we’re seeing more high-profile instances of the latter than the former.”

The response to this state of affairs at Polygon, for now, has been the introduction of “provisional reviews” for launch day evaluations of games that may chance considerably from their pre-release forms. Plenty of other outlets seem to have the same idea: PC Gamer, VentureBeat and others have decided to offer launch day “impressions” or a “review in progress” for this week’s launch of Evolve, allowing time to evaluate the post-launch experience before providing a fully scored review (Ars Technica also offered launch impressions of Evolve rather than a full review at launch).

The Metacritic problem

Metacritic’s “score of scores” is an essential summary for many gamers, and has effects far outside the review aggregation site.

Even if an outlet might be considering dropping review scores, there’s one other major issue that might prevent them from taking the plunge: Metacritic. Outlets that don’t score their reviews don’t get listed on the Gaming section of the site, and that means giving up the significant traffic and attention Metacritic can bring. For some sites, it can also mean being treated as a second-class citizen by the industry.

“From a purely cynical perspective, numerically speaking (and no pun intended), there are always going to be new sites using review scores, because publishers prioritize outlets that appear on Metacritic for access to their products,” Gies said. “So for every site that abandons scores, there will be several more that hug them more closely.”

“I suspect that for fan sites, or smaller sites, being listed on Metacritic is a sign of ‘mattering’ in the video game journalism industry,” Daily Dot Staff Writer Dennis Scimeca told Ars. “For those sites, being on Metacritic also might be a way to justify receipt of review copies.”

Generally, publishers pay more attention to outlets listed on Metacritic because they believe (with some justification) that a strong Metascore means strong sales for a game. That in turn leads some publishers to link employee bonuses and raises to a game’s Metascore.

That’s particularly troublesome for Scimeca, who uses scores in his reviews on The Daily Dot but doesn’t have those reviews listed on Metacritic. “I worry about the large-scale effects of using review scores, for instance the way publishers misuse Metacritic,” Scimeca said. “If I use review scores, and if my outlet is listed on Metacritic, I am in essence feeding that machine. I may, to some degree, be contributing to a decision as to whether or not a studio gets a bonus. I write reviews for the reader. I don’t write them to inform publishers’ economic decisions as to how to reward studios.”

For all the talk of outlets reconsidering how to handle review scores, though, Metacritic co-founder and Games Editor Marc Doyle doesn’t seem too concerned about a mass exodus. Doyle says less than ten Metacritic-listed outlets have dropped review scores in his 14 years at the site, while others have decided to add or reintroduce scoring down the line. “I’m a fan and an avid reader of game reviews, and I simply want critics to use the system that helps them to best express their impressions of a game,” Doyle said. “Even if that choice is to not use a scoring system. By doing so, they will help regular gamers make the best use of their scarce time and money.”

Even if gaming sites did start dropping reviews en masse, Doyle says a site like Metacritic would be OK. For one, he says Metacritic has discussed featuring unscored game reviews on the site in some fashion. For another, Metacritic can figure out its own scores, even if the outlets themselves don’t provide them.

“Many publications in [music, movies and TV] don’t score their reviews, and our editors have become very familiar with those critics and their reviews over the years, which has enabled us to estimate review scores for them,” Doyle said. “Many of those critics will write to us suggesting a score, and we accept those scores without question.”