News Valve changes User Review system to combat review bombs

Chairmanchuck

MetaMember
Oct 9, 2018
38
123
33
Steam Blog :: User Reviews Revisited

Mostly important:

We're going to identify off-topic review bombs, and remove them from the Review Score.
Thanks to Nzyme

We define an off-topic review bomb as one where the focus of those reviews is on a topic that we consider unrelated to the likelihood that future purchasers will be happy if they buy the game, and hence not something that should be added to the Review Score.
Obviously, there's a grey area here, because there's a wide range of things that players care about. So how will we identify these off-topic review bombs? The first step is a tool we've built that identifies any anomalous review activity on all games on Steam in as close to real-time as possible. It doesn't know why a given game is receiving anomalous review activity, and it doesn't even try to figure that out.
It notifies a team of people at Valve, who'll then go and investigate. We've already run our tool across the entire history of reviews on Steam, identifying many reasons why games have seen periods of anomalous review activity, and off-topic review bombs appear to only be a small number of them.
Once our team has identified that the anomalous activity is an off-topic review bomb, we'll mark the time period it encompasses and notify the developer. The reviews within that time period will then be removed from the Review Score calculation.
As before, the reviews themselves are left untouched
- if you want to dig into them to see if they're relevant to you, you'll still be able to do so. To help you do that, we've made it clear when you're looking at a store page where we've removed some reviews by default, and we've further improved the UI around anomalous review periods.


Finally, we've also enabled you to opt out of this entirely, if that's your preference - there's now a checkbox in your Steam Store options where you can choose to have off-topic review bombs still included in all the Review Scores you see.

While we're working on some other features around User Reviews, we thought this one was worth shipping by itself. As always, if you have thoughts or concerns, feel free to voice them in the comments below.
 
Last edited:
OP
Chairmanchuck

Chairmanchuck

MetaMember
Oct 9, 2018
38
123
33
I am not quite sure what would count as a Review Bomb. Devotion, surely. What about Rome 2 Generals?
 
  • Like
Reactions: lashman

lashman

Dead & Forgotten
Sep 5, 2018
30,367
85,134
113
I am not quite sure what would count as a Review Bomb. Devotion, surely. What about Rome 2 Generals?
yeah, exactly ... like Devotion would 100% be off-topic ... i don't think there's any doubt about that

but yeah - what about Rome ... apparently it was even based on fake info someone posted somewhere on reddit ... but how would volvo know that just by reading those reviews? and would they even consider that off-topic?
 

Álvaro de Campos

O nada que é tudo.
Mar 12, 2019
341
858
93
Leaving the reviews untouched and only removing them from the calculation is great, the fact that it's a human deciding what is and isn't off-topic not so much, but I'll take what I can get I suppose.
Also good that the former is optional, although whether it should be opt-in or opt-out is a whole discussion.

I assume there will still be ways around it like drip feeding (reviewing?) over a lengthy amount of time rather than "bombing" as is done now.
 
  • Like
Reactions: lashman

「Echo」

鈴鹿御前。
Nov 1, 2018
2,661
7,356
113
Mt. Whatever
Mixed feelings. I've never participated in one, but I like the fact that consumers can weaponize the review system via "review bombs." This allowed Rainbow Six Siege to avoid Chinese censorship for example. (and then the score recovered) Definitely there are some cases where review bombing has been for nefarious reason though, can't really deny that.

Without effecting the score... I wonder if devs/pubs will care anymore?
 

Dragnix

Does Way Too Long Reviews on Youtube
Feb 6, 2019
9
21
3
www.youtube.com
Again, the biggest problem at the core of the reviews system to begin with is how unsearchable reviews are in general. No real search functionality, no tag ability, nothing. Changes like this would be less of an impact if that ability was in place, but its so simple that you end up with so much overlap situations that some of them get "not counted" because of arbitrary rules. I know for a long time, one of the biggest ways that I grew as a Youtube review channel was to do strong steam reviews with detail (and I still do this), and that a lot of devs, even mid level ones, were willing to give review keys out to channels like mine if they were confident in the game. That became a lot less common when "review key" reviews got taken out of the calculation, and it just showed that those keys that got given to those "bad" reviews took out the goods ones. There's similarities with the review bombing here and how some review bombs are legitimate. Like reasonably legitimate. Some are not. Some are gray area.

Again though, what I will question about Valve is if they will do a good job of communicating why it's not considered a review bomb or not in a timely fashion. Will it play on the side of caution? What if a major release patch breaks a bunch of things, like XInput for something like a fighting game? That may only be a subset of players, will THAT get delisted? Will they explain in a reasonable amount of time?

Again, so many questions.
 

Ex-User (307)

MetaMember
Dec 11, 2018
1,105
2,597
113
I am not quite sure what would count as a Review Bomb. Devotion, surely. What about Rome 2 Generals?
I think the Devotion review bomb was terrible, but even with that game you still run into problems of parsing out reviews.

Some Chinese mainlander writing a review to the tune of "China Number #1, Fuck Taiwan" is obviously off-topic.

But is a review that says "It's a bad decision to put malicious political metaphors into games that should have a deeper meaning" as off-topic? And yes, that's roughly similar to an actual review you can find on the OST store page still. How about someone that says something like "The game is very good, but the metaphors are really excessive"?

Those are a dumb opinions, and one clearly tied to the Winnie the Pooh stuff, but it is still fundamentally an opinion that someone might have. I have no idea how you actually separate this stuff out in meaningful ways. You can certainly remove the clearly off-topic ramblings of someone going on about American hegemony or SJWs or the uppity Taiwanese, but beyond that, what do you do?

EDIT:

Wait a second, excuse me?

Q: I care about some things that I worry other players don't, like DRM or EULA changes. Review bombs have been about them in the past. Do you consider them unrelated or off-topic?

A: We had long debates about these two, and others like them. They're technically not a part of the game, but they are an issue for some players. In the end, we've decided to define them as off-topic review bombs. Our reasoning is that the "general" Steam player doesn't care as much about them, so the Review Score is more accurate if it doesn't contain them. In addition, we believe that players who do care about topics like DRM are often willing to dig a little deeper into games before purchasing - which is why we still keep all the reviews within the review bombs. It only takes a minute to dig into those reviews to see if the issue is something you care about.
Eat dirt Valve.
 
Last edited:

lashman

Dead & Forgotten
Sep 5, 2018
30,367
85,134
113
I think the changes are mostly good, except for DRM and EULA being off topic.
yeah, i don't agree with that one either

although, on the other hand, i don't think we've had even one review bomb about DRM or EULA specifically? or at least i can't remember any

i mean - i remember people complaining in steam forums etc., naturally ... but no review bombs
 

「Echo」

鈴鹿御前。
Nov 1, 2018
2,661
7,356
113
Mt. Whatever
yeah, i don't agree with that one either

although, on the other hand, i don't think we've had even one review bomb about DRM or EULA specifically? or at least i can't remember any

i mean - i remember people complaining in steam forums etc., naturally ... but no review bombs
like massive review bombs no, but on denuvo games you can always find more than a handful of negative reviews which amount to little more than "fuck denuvo" lol.

Not that it's a bad thing imo. I think everyone can agree with telling Denuvo to go fly a kite?
 
  • Like
Reactions: lashman

lashman

Dead & Forgotten
Sep 5, 2018
30,367
85,134
113
like massive review bombs no, but on denuvo games you can always find more than a handful of negative reviews which amount to little more than "fuck denuvo" lol.

Not that it's a bad thing imo. I think everyone can agree with telling Denuvo to go fly a kite?
of course, of course :p haha
 

fsdood

Junior Member
Jan 9, 2019
465
441
63
I only remember one massive review bomb that isn't dumb was the GTA4 mod fiasco. Rest had shitty and reactionary reasoning
 
  • Like
Reactions: lashman

lashman

Dead & Forgotten
Sep 5, 2018
30,367
85,134
113
I only remember one massive review bomb that isn't dumb was the GTA4 mod fiasco. Rest had shitty and reactionary reasoning
eh, there were a few more

  • PAYDAY2 after they added lootboxes (after they promised not to add them)
  • Chrono Trigger ... although that wasn't really a review bomb ... just lots of negative reviews
  • Killing Floor 2 after they added lootboxes (not that it changed anything, the devs just ignored it)
  • there was one where one publisher (can't remember which one or which game it was about) promised a Chinese(?) localization if enough people buy (or pre-order) the game ... but then went like: "nah fam, we won't do it"
  • and of course the R6 Siege censorship one
i'm sure i'm missing a few ... but those listed come to mind
 
  • Like
Reactions: fsdood

fsdood

Junior Member
Jan 9, 2019
465
441
63
eh, there were a few more

  • PAYDAY2 after they added lootboxes (after they promised not to add them)
  • Chrono Trigger ... although that wasn't really a review bomb ... just lots of negative reviews
  • Killing Floor 2 after they added lootboxes (not that it changed anything, the devs just ignored it)
  • there was one where one publisher (can't remember which one or which game it was about) promised a Chinese(?) localization if enough people buy (or pre-order) the game ... but then went like: "nah fam, we won't do it"
  • and of course the R6 Siege censorship one
i'm sure i'm missing a few ... but those listed come to mind
Yeah you're right, I guess I remember the people who constantly complain about "SJW propaganda" or whatever. Hopefully the things you listed are not considered off-topic
 
  • Like
Reactions: lashman

lashman

Dead & Forgotten
Sep 5, 2018
30,367
85,134
113
Hopefully the things you listed are not considered off-topic
yeah, that's what i'm hoping as well .... technically none of them are about DRM or EULA, i guess :p

i suppose we'll find out when the next one of this kind happens
 

Ex-User (307)

MetaMember
Dec 11, 2018
1,105
2,597
113
eh, there were a few more

  • PAYDAY2 after they added lootboxes (after they promised not to add them)
  • Chrono Trigger ... although that wasn't really a review bomb ... just lots of negative reviews
  • Killing Floor 2 after they added lootboxes (not that it changed anything, the devs just ignored it)
  • there was one where one publisher (can't remember which one or which game it was about) promised a Chinese(?) localization if enough people buy (or pre-order) the game ... but then went like: "nah fam, we won't do it"
  • and of course the R6 Siege censorship one
i'm sure i'm missing a few ... but those listed come to mind
  • Arkham Knight was bombed for the PC port to the point of the game being removed from Steam
  • Skyrim was bombed because of the whole paid mods idea which forced removal of the paid mods.
  • GTAV was bombed after OpenIV got a cease and desist letter from T2, and OpenIV was allowed to start distribution again afterwards
  • Crusader Kings II was bombed after DLC prices got raised, and other Paradox games got bombed over complaints about DLC structure in general.
  • Sonic Mania was review bombed because Denuvo was not disclosed on the store page

The "Total War II" and "Firewatch" style bombings are generally outnumbered by the number of legitimate times customers have had complaints.

The handwringing over "useful reviews" is total bullcrap anyways. How is a review that says "Eff this game for having Denuvo" any less useful than some prat writing "10/10 love the game" as their review? People only suddenly begin to give a shit about the perceived value of a review when it's negative about something they care about.
 
  • Like
Reactions: Nirvash and lashman

Alextended

Segata's Disciple
Jan 28, 2019
5,463
8,527
113
There's no harm in having to write properly about the game after actually playing it a while then including a "but it also has denuvo so I can't recommend it" somewhere. If that kind of proper review gets removed that'd be bad. I won't miss the review type where they just got the game didn't even launch it just to tell me it has denuvo and give it a bad rating. There are other ways to find that out for those who do care about it.

Have to agree that we also get useless positive reviews but nobody ever talks about that. Some are even sarcastically positive. Game sucks, 10/10.

Same for a game that gets reviewed badly for its horrible performance like Arkham Knight, I don't see why that would count as off topic. But again it's okay if they do make it so you have to write a coherent sentence about it first. Anyway if we do get a case where there's mass removal of reviews about legitimate game issues I'm sure there will be community pushback and they'll yet again attempt to change it.

And again I don't see why anyone ever put stock in review bombing if the game's overall score wasn't actually turned from positive to negative for some dumb reason. It only brought some attention to an issue but if the game was otherwise hugely liked the score remained and that atttention would soon disappear too once those reviews no longer showed separately as recent reviews (which was also added later to combat review bombs but probably just made it noticable).
 
Last edited:
  • Like
Reactions: lashman

Álvaro de Campos

O nada que é tudo.
Mar 12, 2019
341
858
93
If that kind of proper review gets removed that'd be bad.
Q: So if I post a review inside in the period of an off-topic review bomb, my review won't be included in the Review Score?

A: Unfortunately, this is correct. We've tested our process of identifying off-topic review bombs on the entire history of reviews on Steam, and in doing so, we've found that while we can look through reviews and community discussions to determine what's behind the review bomb, it isn't feasible for us to read every single review. But as we mentioned back in our first User Review post, our data shows us that review bombs tend to be temporary distortions, so we believe the Review Score will still be accurate, and other players will still be able to find and read your review within the period.
So it might be 'removed' depending on when it was posted.
 
  • Like
Reactions: Ex-User (307)

Alextended

Segata's Disciple
Jan 28, 2019
5,463
8,527
113
Well that makes sense. As long as they notify you a review bomb is in progress so you can refrain from doing it at the time.

Of course people will game this system also. Delay the negative reviews. Copy paste other text so it seems like a legit review.

Whatever else. And the press will have more to complain about as if having no user reviews at all and only press reviews is the ideal solution to all.
 
  • Like
Reactions: lashman

Gevin

Watch Madoka
Nov 16, 2018
1,046
3,194
113
I feel like a lot of people are acting like this is not optional, if you care about all of those things you'll still be able to see it
 
  • Like
Reactions: lashman

Dragnix

Does Way Too Long Reviews on Youtube
Feb 6, 2019
9
21
3
www.youtube.com
I feel like a lot of people are acting like this is not optional, if you care about all of those things you'll still be able to see it
but on the flip side, that's IF you saw the optional element of it. The problem with those choice from the UI perspective if those who didn't see this news post may not know this option is available.

So now, I want to think about the scenarios in which would be grey:

  1. What about the Assassin's Creed Odyssey controversy with Cassandra's forced decision? Specifically, with the DlC. Technically, it could be argued that the decision DOES make a difference for specific people.....but does that apply to the main game, or just the DLC? I mean, it could actually effect your opinion of the main game in the end. That's where things get really really murky.
  2. What about an update that effects something outside the game? LIke a Windows 10 update breaking something with your specific game? It is assumed that it seperate enough from the game that it doesn't count?
  3. What about Nvidia vs ATI? What if an update screws all ATI users? Because they are a smaller %, is the logic regarding the "majority"?
My biggest question right now: will you be informed if YOUR review falls into the bombing category? Remember: the way they word it, a legitimate posted at the time will get counted in the review bombing. So a person who did an actual reasonable review gets hit: will they be able to TELL that?
  • Arkham Knight was bombed for the PC port to the point of the game being removed from Steam
  • Skyrim was bombed because of the whole paid mods idea which forced removal of the paid mods.
  • GTAV was bombed after OpenIV got a cease and desist letter from T2, and OpenIV was allowed to start distribution again afterwards
  • Crusader Kings II was bombed after DLC prices got raised, and other Paradox games got bombed over complaints about DLC structure in general.
  • Sonic Mania was review bombed because Denuvo was not disclosed on the store page
The "Total War II" and "Firewatch" style bombings are generally outnumbered by the number of legitimate times customers have had complaints.

The handwringing over "useful reviews" is total bullcrap anyways. How is a review that says "Eff this game for having Denuvo" any less useful than some prat writing "10/10 love the game" as their review? People only suddenly begin to give a shit about the perceived value of a review when it's negative about something they care about.
I think it depends.

Take the Denuvo situation. The thing is.....Steam SAYS that the game has Denuvo or not. They have to disclose that. So it becomes a question: is it reasonable for a review to hold it against a game when it was made clear from the get go that it had it? You were informed, you didn't listen to it: you didn't have to buy it. Sure, if it was added AFTER you bought it, of course, like the sonic situation. But in the other situations where it was there from day one: is that legitimate? What if someone finds that the Denuvo implementation caused enough of a slow down that they believe it's THAT much of a problem beyond the DRM issue? There's a lot of questions here: and it all comes back to the question: will Valve inform the public of the reasons a bomb was declared. Now don't get me wrong, I'm not a fan of the Steam user reviews implementation. It could be much more effective with design changes. But this could work IF it's clear what's considered a bomb or not, so that users can jump up and down if something considered a bomb is in that grey area.

It also should be noted that Valve obviously waited to release this news later on a Friday for a reason. They knew there would be blow back. That's where news/information is dump into the press knowing that it'll get overshadowed.
 
Last edited:

Alexandros

Every game should be turn based
Nov 4, 2018
2,669
11,485
113
I don't like this change, it casts way too wide a net. By all means, filter out racist, misogynistic etc reviews, but why the hell is a negative review about DRM considered off-topic?
 

lashman

Dead & Forgotten
Sep 5, 2018
30,367
85,134
113
but why the hell is a negative review about DRM considered off-topic?
hopefully they'll change that the first time it happens

i don't think it has yet - specifically a review bomb because of DRM ... i mean, sure - there are lots of reviews complaining about DRM, but those won't be removed or "not counted" ... only when it reaches the "review bomb" status

but yeah - i hope they'll change this one bit when it finally happens
 
  • Like
Reactions: Alexandros