What developers think of Steam reviews
Since 2013, Steam has allowed its users to leave reviews on the Steam page of games they have purchased. Using the system, players can leave a written description of the game in question, and then award it a binary “Recommended” or “Not Recommended” rating. These reviews are then aggregated into “recent” and “overall” ratings for the game which are displayed on the game’s Steam page. These ratings range from “Overwhelmingly Positive” to “Mostly Negative.”
Because of Steam’s ubiquity on the PC, Steam reviews have become one of the main ways that developers receive feedback on their games. But how do developers feel about the system itself? Do Steam reviews provide a beneficial service that can help improve games? Or is it a perpetual nuisance warped by review bombing and ‘joke’ reviews that cause stress and confusion to the people who make the games we play?
To find out, I spoke to developers about their experiences with the system, asking them about their views on its advantages and disadvantages, and discussing whether there were any obvious improvements to the system that could be made. I aimed to speak either to solo developers or developers in charge of their own studios, who had direct experience reading and responding to Steam reviews with little or no moderation buffer. These ranged from the vast community and perpetual updates of Rust, to smaller, niche games.
Their responses covered topics such as common player complaints, the contentious ethics of review-bombing, and the strange case of the thousand-hour negative review. For the most part, however, the developers I spoke to believed that Steam reviews can be a useful source of feedback. “Reviews on products are great,” says Garry Newman, head of Facepunch Studios and creator of Garry’s Mod and Rust. “The advantages are obvious. You can warn people if the game is shoddy, or broke, or their sales garb is bullshit. Leaving a positive review allows consumers to reward the developer in an extra way.” They’re also aware of the systems usefulness from the user perspective. “I think it has empowered gamers,” says Bill Gardner. Formerly a level designer at Irrational Games, Gardner is currently director of TheDeepEndGames, whose debut release was last year’s horror game Perception. “It’s a great way for gamers to have their voices heard.”
What comes across in conversation with all of the developers I spoke to is the seriousness with which they take feedback from their games’ communities. “At its heart the Steam review is the most raw unfiltered feedback I’m going to get,” says Phillip Bak, creator of twin-stick shooter Bezier. “It’s simultaneously both thrilling and terrifying. Paying players are not known for their courtesy and to be honest nor should they be.” Of course, this doesn’t mean that developers are boundlessly grateful and pay mind to every review that they receive. Newman in particular has come into conflict with his community over criticism in the past, such as
in this post on Reddit.
Although most developers pay some mind to Steam reviews, the impact of a positive or negative review varies enormously. For a game like Rust which has a huge following, a bad review is a drop in the ocean. “We’re in a privileged position where I think we’ve got enough goodwill in the community that reviews don’t really have any effect on us,” Newman says. “We can get review bombed to death and people will still buy our games.”
The case is very different for a game like Duskers, which has a small but active fan-base that communicates frequently through Steam reviews. “Before [Steam reviews] I was beholden to Metacritic, and as you may guess, for indie games there just weren’t enough data points to get a solid review,” says Tim Keenan, creator of the inventive, claustrophobic sci-fi horror game that was
one of RPS’s favourites in 2016. “Now I can create a niche game and have it reviewed by people who care about those games. When I’m developing, I can scan those reviews to get an aggregate idea of things that players are struggling with and address them.”
Gardner, who has experienced user feedback both as a small developer and as part of a large mainstream studio, explains that with mainstream development the sheer size of the audience makes it difficult to respond to individual concerns, which is less the case in indie development. “In AAA, you’re all but guaranteed to get your game in front of millions of eyes. As such, individual reviews are likely to get a bit more watered down simply in volume alone,” he says. “You’re only ever going to be able to understand a minuscule fraction of the people who played your game. With Perception, I feel I’ve been able to understand the people who are playing a lot better.”
While Steam reviews can be useful in this way, the developers stress that they aren’t the only important source of feedback. “I think there’s a tendency to put more weight on user reviews out of fear,” says Gardner. “While obviously there is a concrete piece of data that impacts the appeal of your product, the impulse to react to reviews rather than comments can be perilous,” Keenan agrees with this. “The only worry is that this tends to make you crazy, and want to reply to any bad review, which would be exhausting.”
Indeed, feeling the sheer weight of the internet pressing down on your game can be a stressful experience for a developer. “When anonymous players are rating your game, it’s very hard to remember that they are individual people. You can easily see them as a unified voice and paint them all with the same brush,” says Katie Goode, creative director of Triangular Pixels, who develop VR games such as Unseen Diplomacy and Smash Hit Plunder. Even if a developer is able to receive and digest the majority of reviews dispassionately, any particular review has the potential to cause considerable distress. “Any review can affect you, your team morale, your studio’s reputation and future, and your mental health deeply,” she says.
Meanwhile, not everyone finds Steam reviews so useful. “I don’t find reviews as helpful as forum discussions,” says Cliff Harris, founder of Positech games. “They are often too short, they are rewarded for being ‘funny’, and developers are told not to reply to them in most cases.” Harris is referring to users’ abilities to tag reviews left by other users as “helpful”, “not helpful” or “funny”. These tags are designed to let users filter out specific kinds of reviews, but they can also incentivise users to write reviews that prioritise being “funny” rather than being “helpful”.
Harris puts the unhelpful nature of Steam reviews down to the small percentage of players who actually leave reviews. “Hardly anybody leaves reviews, about 1%, so you get the ultra angry, or the ultra time-rich, who do not represent the majority of gamers. Unless 5-10% of players are ranking something, you aren’t going to get a realistic appraisal of a game.”
Indeed, one of the quirks of Steam reviews that nearly all developers find baffling, is reviewers who play the game in question for hundreds, sometimes thousands of hours, and then leave a negative review. “NOBODY is so cash-poor and time-rich that 100 hours of entertainment for $20 is a bad deal. It just looks totally silly,” says Harris. Newman agrees. “You gave us $20 and we entertained you for over 1000 hours. What else do we have to do?” Bak, however, thinks there may be more value to these reviews than is initially apparent. “Those reviews are often the most interesting, and I think a player who has sunk that much time might have a lot of valuable stuff to say. Often the thumbs-down rants give me more reasons to purchase a game than the positive reviews.”
Negative reviews with a ridiculously long playtime aren’t the only type of review that cause developers to raise their eyebrows. Gardner, for example, is troubled by reviews that have very short playtimes. “One thing that really doesn’t sit well with me is that reviews with so little playtime on record are given the same weight as someone who has finished the game,” he says. “Perhaps if there were separate pools of reviews where there is a minimum required time played to enter a pool A vs pool B, where people have only sampled.”
Another contentious issue is review-bombing, where users will collectively review a game negatively on account of some perceived slight. This could be a change to the game that a section of the community dislikes, but it can also have nothing to do with the game whatsoever.
A controversial business decision by a publisher,
or a statement made by the developer which the community disagrees with, can both lead to games taking a critical beating on Steam.
“It’s terrible how users can use reviews to bomb negative feedback on what could be an unrelated issue,” says Goode, “Buying the game, writing a ‘review’ laying into the developer, then refunding it… it’s a problem for everyone – obviously for developers, for Valve who seem to enable rage-filled, immature hate, and for the players who are just trying to see if they may like a game for what it is at that current time.”
Gardner agrees with this. “When gamers use reviews as a tool to make a statement it can be pretty frustrating and kind of defeats the entire purpose of reviews altogether. To be clear, I am all for gamers speaking up and making their voices heard. I just think there are ways to do that aside from poisoning reviews.”
But shouldn’t it be up to the users to decide what is important when it comes to making or informing others about a purchase? If, for example, a developer revealed themselves to be racist, that’s an issue that many players would want to know about before they supported the developer with their money. “This is absolutely a fair point,” Gardner says when I raise the issue. “I’d never want to tell people how to use their voice. I personally feel that this is not the right avenue for this type of thing simply because it’s unclear where this sort of protest review stops… We have so many places to set the record straight with social media. However, reviews have their own purpose. They’re there to be an assessment of the work.”
General player tastes and prejudices can also be a problem for developers on Steam. One issue that players frequently bring up is game length, particularly with games that are shorter than average. “We did a late, short early access period and suffered largely from Steam players’ attitude towards short games,” says Jordan Thomas, co-founder of Question! and developer of The Magic Circle. “I can’t really fault the review system itself, the players would mention TMC’s short length even in positive reviews. They thought it was worth ‘warning’ people about.” Gardener has experienced similar problems with his game, Perception. “We see very conflicting feedback. The game is about 4 hours for most people. I think the majority have found it to be a good value, especially considering the games that share some overlap. But there are always going to be a lot of people who expect a minimum of 12-15 hours for a single player game.”
For some developers, it can be tough putting their game on Steam and watching the resulting tsunami of criticism pour in. But it’s important to note that, unlike with critical reviews where analysis and opinion is generally set in stone, with Steam reviews developers can act to alter the tide. In fact, Valve recently split Steam reviews into two categories, prioritising the most recently written reviews to highlight any changes made to the game.
“The dynamic score does work!” says Goode. “In this age of rapid patching and updates, the weighting to newer reviews keeps users and developers informed of the current state of the game, rather than the state it launched with.”
Cliff Harris, however, is less enthusiastic about the flexible nature of Steam ratings. “Pretty much any change to an established system is going to knock your review score down,” he says. “If you make a change that 90% of people prefer, they won’t react in any way to it, but the 10% who preferred the old system will leave negative reviews in a disproportionate amount. Basically changing anything to an existing game can be risky.”
Positive apathy seems to be a general concern amongst developers, this notion that happy players remain quiet while dissatisfied players shout the loudest. But Newman believes there is a way to combat this phenomenon – talk about it. “A while back we were getting reviewed negatively for some changes we made to the game (whatever we do, we get a subsection of players that aren’t happy). We mentioned this in our blog and our reviews went positive for a couple of days,” he says. “This worked because the huge majority of people that are enjoying the game didn’t review it, so they decided to help fight against the negativity by giving us a positive review.”
Is there anything that can be done to further improve Steam reviews? Aside from combating the issues already mentioned, the main thing developers want to see is more people using the system. “Incentivise them!” says Harris. “The text for ‘leave review’ is like an afterthought. Anybody who has played > 2 hours of a game should get a little pop-up, like the screenshot uploader suggesting they give it a thumbs up or down, at the very least,” Bak concurs with the notion of incentivising players to leave reviews. “Awards for the best reviews (Developer spotlight on their favourites). 1% off your next game when you review the last one you bought.”
Crucially though, developers want to see players using the system properly, leaving reviews that explain why they liked and didn’t like the game. They don’t particularly want “joke” reviews that offer no information, and they certainly don’t want their Steam reviews used as a platform for a more general grievance. “Ultimately there needs to be another way of allowing users to voice their discontent, rather than review-bombing,” says Goode. “Users should also have their reviews checked by a bot – have to be a minimum length, consisting of words that actually make sense. If a person is going to publish a review that can affect someone on a personal level, it’s only fair that they have to put in a minimum amount of effort.”
Lastly, Harris also urges users to remember that, when they leave a review, at the other end is a person who will probably read it. “People reviewing games need to remember that virtually no developer is actually greedy or evil, or incompetent,” he says. “There are real humans, probably stressed, tired, worried, and hugely emotionally invested, who will read what you write about what is likely the last two years of their life.”
)) They actually had a system in place to build on, to prevent some of free keys review or what not. But why bother?! Just banned it!
