Review scores have come to Shacknews
It's been a long time coming. Shacknews has been reviewing games for years, judging games not as products, but as experiences. Instead of breaking down games as the sum of easy-to-digest parts--graphics, gameplay, story--we wanted to provide a critical look at the game as a whole. Starting next week, we're going to start implementing numerical scores to our reviews.
It's been a long time coming. Shacknews has been reviewing games for years, judging games not as products, but as experiences. Instead of breaking down games as the sum of easy-to-digest parts--graphics, gameplay, story--we wanted to provide a critical look at the game as a whole.
Starting next week, we're going to start implementing numerical scores to our reviews. Behind the scenes, this has been a process that has been debated quite passionately by all of our editors. We never questioned "should we do this?" Rather, we wanted to be certain in how we would implement scores. We wanted a scale that reflected our holistic approach to reviews, one that embodies our games-as-experiences philosophy.
Here is the scale that we're going to use:
- 10 - A milestone in gaming
- 9 - An exemplary representation of its genre, with resounding impact
- 8 - A leader of its genre, offering fresh interpretation of the form
- 7 - A solid representation of its genre
- 6 - A flawed representation of its genre, failing to execute on new ideas
- 5 - Insipid, but mechanically sound
- 4 - Insipid, and mechanically faulty
- 3 - Artistically void, but mechanically sound
- 2 - Artistically void, and mechanically faulty
- 1 – Unplayable
We decided on a ten-point scale, as we believed anything more than that would be too granular. (That means no half points, no decimals, etc.) We also believe this is a scale that will be fully utilized. What's the point of a reviews scale if only the top half gets used?
I want to assure you that review scores will not change the approach we take to reviews. Instead of writing to a score, we want to make sure that the score is derived from the text our reviewer writes. Not only will you start seeing scores attached to every forthcoming review on Shacknews--we're going through the process of retroactively applying these scores to our legacy reviews. Eventually, we plan on offering a catalog of reviews that can be easily referenced, which should offer greater context on what a "10" means, for example.
Review scores have been an exciting debate for us here at Shacknews, and we're eager to see how you respond to them. Please offer me any questions and comments you may have in the comments below, and I'd love to hear your suggestions.
-
Shacknews
replyAndrew Yoon posted a new article, Review scores are coming to Shacknews.
It's been a long time coming. Shacknews has been reviewing games for years, judging games not as products, but as experiences. Instead of breaking down games as the sum of easy-to-digest parts--graphics, gameplay, story--we wanted to provide a critical look at the game as a whole. Starting next week, we're going to start implementing numerical scores to our reviews.-
-
I don't think we'll have to go out of our way. While we've yet to back-score our reviews, there are already a number of games that will get scores in that range.
-
-
I know that we are a bit concerned that our scores might result in a lot of phone calls and potentially burned bridges. But if we're going to do scores, we're not going to do them simply to conform to what's "normal."
-
-
Fuck, this is totally a trap. And I'm totally going to fall for it.
In my opinion: 9, 9, 9, 10-
Burn the heretic!
-
-
-
-
-
if the shack actually sticks to this, i'll be very impressed. +1
-
-
-
Who gives a shit about metacritic?
-
Every company
-
Lots of people, actually.
-
[deleted]
-
[deleted]
-
-
-
It makes no sense that everybody doesn't just use a 1-5 scale. If you need 10 increments, do half points. 1-5 is a nice scale everybody understands and that allows you to use the entire range. 1-10 just carries extra baggage where people expect "average" to be 3/4 up the scale instead of in the middle of it.
-
-
Ugh, no. Just no.
-
[deleted]
-
[deleted]
-
I'm usually on this boat, but we're not writing to aim for a particular score. We're keeping written reviews as they are. The only major change is that the staff will get together and collectively assess the review to determine where it falls on our scale. No one's changing their review approach.
-
[deleted]
-
I totally understand your concern. One of the reasons I decided to introduce the reviews scale now (instead of coinciding with a review) was so that we can be transparent about what we think these numbers mean. And, if there's any suspicion of foul play, you guys can call us on that bullshit. Point to this scale and say "hey, your review reads more like X." I want this scale to be transparent to everyone. Ultimately, reviews serve readers NOT publishers.
-
[deleted]
-
-
[deleted]
-
-
-
-
[deleted]
-
While not neccessary for me, I support this move. Especially with that kind of thought you seem to have put into it. I really like the individual meanings for the scores and I believe it's good to give a very clear "score" for a game, in particular for somewhat indifferent reviews.
You're gonna make it easier for your readers and I understand you probably have to go with the flow as well, since almost everyone is using a system like this.-
Thanks! I appreciate seeing something positive. I'm still waiting to take a whipping from the community...
-
-
But you're totally going to love our "top 10 ankles of video games" piece coming up. Who knew ankles could be SO SEXY???
In seriousness, I know the proof is going to be in the pudding, so I hope to have our retroactive scores appear ASAP to give a clear understanding of where we want to take this system.
-
-
I assume this is a pretty good business move. Right now I have to guess that Shacknews isn't being included in any aggregate scoring site. Thats free publicity being missed.
-
Yep. Shack's reviews are higher quality than most on metacritic, so it's good they will be included now. And they'll get some of that internet money.
I actually use metacritic, not for the average, but because it has all the reviews I like to read linked on one page. I always click Giant Bomb, Kotaku, Destructoid, EGM, Eurogamer, AV Club, Polygon and, for games released before spring 2013, 1-UP. Now Shacknews will be conveniently located on the page and I won't have to search for it separately.
-
-
FREE KEEFER!!!
But seriously, this can only mean good things for this site. Review score = metacritic visibility = FRESH MEAT. -
Finally Remo's Reign of Terror has come to an end! http://www.youtube.com/watch?v=GD5ZXuL0HyE
-
This isnt what shacknews deserves, but its what it needs right now.
-
Ug. And we lost Keef as well. Worse and worse.
-
-
7s are good games. I would even argue that 5s are "good" on our scale, but likely have a smaller appeal.
-
-
-
Let's hope so.
-----------------------------------
CPU: Pentium 4 2.5 GHz @ 3.2 GHz
HSF: Scythe Ninja Copper
MB: DFI LANPARTY UT nF4 Ultra-D
RAM: OCZ DDR2 PC2-6400 GOLD GX XTC
HDD: 2x IBM Deskstar 75GXP in Raid 0
GPU: ATI Radeon 9800 Pro
Case: Antec P180
Fan Grills: Punisher
OS: Windows XP SP2-
killer rig
-
Lol @ fan grills
-
-
-
I guess this was inevitable, and I know it drives traffic and shit, but it's just bad for good quality reviews. Even Roger Ebert was notoriously critical of the Star system used for movies and his own Thumbs system, because ultimately the numbers or good/bad evaluation become completely arbitrary and tell you nothing about the subtlety of differences between two games/movies/whatever.
-
I hope that our years of reviewing games without scores has established our credibility. Ultimately, our goal is to continue writing reviews the way we always have. There will just be a number at the bottom of the text.
-
-
He did star reviews because the industry he operated in expected/demanded it. He's publicly written many times that the star reviews are essentially worthless unless you also read the accompanying review, and even then he would have preferred to do away with the stars/thumbs/whatever altogether.
-
Oh I get that, it's just that for many people, the number supersedes the text. They'll see that Call of Duty 25: Still Callin' gets a 9/10, and that Stunning Ambitious Indie Game gets 8/10, and immediately assume that it means that Call of Duty is better than Indie Game, when in reality it probably means that Call of Duty is 90% successful at being a fun and interesting action game, and that Indie Game is 80% successful at being a different type of game - each on their own respective terms. Then you'll get the ad nauseum deluge of "WTF Shacknewbs!? Call of Crap is nowhere near the brilliance of Stunning Ambitious Indie Game!" and so forth. But, good luck in implementing it. I understand why.
-
I'm really glad you brought up this example. One of the reasons why "graphics" and other metrics aren't included in our rubric is precisely to avoid this. In fact, as you can see, innovation and artistic intent is built into our scale. A blockbuster game that plays it safe is less likely to score high than a genuinely interesting indie game.
-
I think your scale is pretty well-designed. How much people remember/care about that in a year's time when looking at your scores on Metacritic is going to be debatable, even if you publish the rubric along with each review. I'm sorry to sound so cynical about this and I laud your approach, I just disagree with scored reviews on a fundamental level.
-
-
Totally agree with you. But truth be told, we're not beholden to Metacritic for our scores. Ultimately, we designed this scale for OUR readers, not THEM.
-
-
-
-
-
-
[deleted]
-
Oh I know, I see those posts all the time. It's something that people react to. Numbers give people a tangible measurement to use as a basis for staking out a position or making arguments. I get it. They're just not for me.
-
-
-
[deleted]
-
-
[deleted]
-
-
For better or worse, you won't get linked from gamerankings or metacritic without a score, so I understand this move. I just hope that means the reviews won't diminish in quality since I think you have to give more detail in a review without a score since it can't be easily summed up as "8/10".
-
No worries. Our approach to reviews will not change.
-
-
[deleted]
-
I love Kotaku's review approach. Ultimately, we decided to be slightly more granular than a "yes" or "no." I do think that our current scale sort of reflects that mentality, though. I would say 5-10 is a "yes" while 1-4 is a "no." Hopefully, our text can explain in further detail.
-
The problem is, no one will pay attention to your text now that you have scores. :(
-
-
-
This is the complete opposite of what should be happening.
-
OMG I BET THE SHACK IS DOOMED AGAIN LIKE ALL THOSE OTHER TIMES THE SHACK WAS DOOMED!!!!
More traffic, means new people, which means lols and nukes. My finger is itchy to pull that trigger!-
I can't wait for the inevitable power trip!
-
-
Exactly! There comes a point where review scores become a bit TOO granular. That .1 difference sure did piss a lot of people off, though.
-
-
-
-
Which is perfect since the Duke loves doling out the pain! Don't you, Duke?
-
-
-
[deleted]
-
This is a great question.
I chose "insipid" because it connotes blandness, a kind of "sameness" that plagues many modern games. "Artistically void" I think takes that to yet another step--avoiding not only making something that "feels the same," but actually is regressive in its approach.
The reason why there's some clear overlap and repetition in this scale is that it originates from a five-star scale we were working with. However, once you add half points to a 5 point scale, it becomes a 10 point scale like ours. Instead of simply saying "halves" are just in-between two numbers, we wanted to have a clear definition of what each number meant. Ultimately, we decided that our primary focus is the artistic intent, followed up by the execution. Essentially, a 7 really is a 3.5/5--but we wanted to define that score as more than "between 3 and 4."-
[deleted]
-
I agree. I think rating systems that colligate a WHOLE bunch of information into a single track are really horrible. :(
Rating systems like this just can't capture the nuances and proclivities that make aspects of games special, even if they aren't well executed from the reviewer's point of view.-
"Rating systems like this just can't capture the nuances and proclivities that make aspects of games special".
Um wouldn't that be the wordy stuff that's about the number? The actual content of the review. Review scores are only stupid if the reader gets to hung up on the actual score.-
Absolutely, but who's going to look at it now that there's a number?
-
-
The problem with review scores is that people don't read the actual reviews. They look at the score and they freak out because the game they have yet to play received a bad score. That's just the internet.
-
-
-
-
-
I did suggest a 1000 point scale, because I agree with you. It would've been fun to go full-stupid. But ultimately, it would only be a joke--and at the end of the day, we still want reviews to offer a service.
-
-
-
One of the biggest problems with scores is the importance the readers/gamers put on the scores. Trying to offer a better score system would just put more importance on it. Scores are fine in my opinion but the main review of the game should always be the written content of the review, not the score.
-
If you want the written word to keep its importance, leave the scores off. Simple. Any system contrasted to the way it was is going to put more importance on the numbers by the gaming community at large. Doing something different could make you stand out at least.
The scoreless review was once the best thing this stupid site had going on, and the reason I actually cared to read the reviews here. The staff changes ensured it wasn't going to last but whatever, a 5 star system like Giant Bomb doesn't put any more importance on scores over their reviews. Nor does the letter rating on 1up. If anything it makes me more interested in reading the full reviews compared to every other stupid fucking 10 point-that's-not-really-10-point site.
-
-
-
-
[deleted]
-
-
-
The usual is one through six are shit, seven is passable, eight is good with a few flaws, nine is great with minor flaws, and 10 is either a masterpiece or they got paid in suitcases full of money.
-
[deleted]
-
-
BlackCat9 is spot on. Either no reviews or a 5 point scale. Anything else is dumb because you aren't going to go out of your way to review horrible games that deserve a 1-2, so your scale is automatically redundant.
Going back and putting scores on old reviews sounds like the worst idea ever :( There is no way to do it correctly with the hindsight that you have.
-
-
-
-
[deleted]
-
-
Pssh, Capcom would sue us into oblivion.
-
[deleted]
-
-
[deleted]
-
They only need one rating. F! </shackers>
-
-
This sounds well intentioned, but it's ultimately a decision to conform to the worst elements of the video game press.
If Shacknews wants to attract more readers, it should try, you know, improving it's editorial content. It's been a slow decline in quality since I started reading in 2007. Reviews are shorter, and less interesting. There is more coverage of banal news like DLC. There are almost no long opinion pieces about the game industry. Shacknews certainly isn't as bad as some place like IGN.com, but that's not saying a whole lot.-
I'm just responding to this to let you know that I see this feedback and, in many ways, agree. I'm definitely pushing for more interesting news, and certainly more editorials to appear on the site. Thanks for being such a long-time reader, and I hope one day, you will PM me to say "hey, this site is now as good as I remember it." Really appreciate the honest feedback.
-
^^^^^^ this right here.
The amount of no-news that has been posted recently is ridiculous.-
I don't think it's "ridiculous." I'm looking at the front page right now, and I don't see any "no-news." But I welcome your feedback, so if you can shoot me some examples via PM, I'd really appreciate it.
-
I actually appreciate some of the minor news stuff. Like Rayman Legends coming to PC. Not earth shattering news, but it definitely helps people like me who only like to read Shacknews and one or two other sites.
I agree that the editorials have declined a little bit. I think 1UP in the mid-late part of the last decade is the best gaming site I've seen. It was unsustainable as a business, but it was awesome site.
-
-
-
-
I'm on board. It will help drive traffic and, ideally, increase page views. For example, personally speaking, odds are if I see Shacknews rate something 10/10 I'm going to read its review even if it previously didn't interest me. That also holds true if I see something rated 1, 2, or 3, because I'll be curious about what made it so bad. Also, if I was seeking out reviews of something, I'm not going to get all e-snob and turn my nose up at the review because there's a score at the bottom of the page.
Get over it. Change helps avoid stagnation. -
I give this review score system....... 3 pregant Sessler pauses....... out of....... 5.
-
Which is a 6 on our scale: "A flawed representation of its genre, failing to execute on new ideas"
Sounds about right. ;P
-
-
I am disappoint.
Honestly though I'm glad we made this far without review scores and while I don't necessarily see them as "evil" I do see them as a step backwards from well thought out written reviews without a score; as soon as you add that number it becomes all most people see.
Here's to hoping you guys can execute it in such a way that it isn't as soul sucking and meaningless as most sites that metacritic pulls from.-
We'll make the font for the score really, really small. Make people work to find it.
-
[deleted]
-
I'm not promising anything, but perhaps we'll be able to allow users to enable/disable scores via the Settings.
-
-
-
[deleted]
-
-
[deleted]
-
[deleted]
-
Ain't nuttin' tah stream with.
-
-
Shack Leers. We Make Games Uncomfortable! tm
-
Is that the creepy feeling I've been having?
-
-
-
[deleted]
-
-
[deleted]
-
[deleted]
-
-
[deleted]
-
I would argue that people follow it because it is a leader in its genre. Hell it defines its genre.
Also players have different priorities.To me CoD immediately gets a few legs up on most of the competition because it runs at 60 fps and has amazing feeling controls. So many games fail miserably in both of those departments that I would have trouble ranking them above a 7. Other people couldn't care less about either of those things.
-
-
-
-
[deleted]
-
[deleted]
-
-
-
[deleted]
-
-
While we have yet to retroactively score our reviews, you've got the score spot-on. That's precisely what that game would get according to our scale.
-
[deleted]
-
[deleted]
-
-
-
-
-
-
because it's not about the score but OMG metacritic that would normalize those stars to shit.
-
Make it 6 stars.
-
[deleted]
-
-
-
-
-
[deleted]
-
-
-
"This game was terrible. 6/10"
"This game was a lot of fun. 7/10" -
[deleted]
-
-
[deleted]
-
[deleted]
-
-
-
Well as long as the way you do your reviews don't change I'm fine with this but at this thread illustrates there will be some backlash, because people will be offended when you don't give The Last of Us the proper 10 it deserves and instead give it an 8...... Still can't believe the rage that incited over at GS which goes to show what happens when you do attach a number to a great review the drones stop thinking.
But once again as long as the actual reviews are concise and well done as they already are then the number means little to me overall, its just for the simpletons. -
So let's say you go back and give a retroactive grade to games like STALKER and Sim City 5. How does a buggy POS on release get rated retroactively and going forward?
-
Unfortunately, we can only assign scores based on whatever copy exists on the site. For example, I quite disliked Uncharted 3 and disagree strongly with our review. And as the years have gone by, I think consensus has shifted a bit. But, we're going to assign the score based on what's originally on the site.
-
-
-
-
[deleted]
-
[deleted]
-
-
Your current review scale is fine, but is way to complicated for the raging fanboy. Here's a review scale that will coincide with the internet's reaction to scored reviews.
10-8 A great game that you need to play.
7-1 A piece of garbage that needs to be destroyed. -
[deleted]
-
The problem is you have people who aren't going to look at what each score means, so anything 7 and below will be shrugged off and they won't even read the review itself, they'll just jump right to the score.
And I guess we can expect those other people who like to complain about scores being too low or too high to start showing up... -
-
Wiggling my toes is ok, scratching my head means it sucks, and sniffing my arm pits means it's average. So I'm giving it a picking my belly button. You figure that out!
-
-
hey everyone, make sure to wave to the shark as we jump over it.
-
Just another step in the inevitable downfall of shacknews.
-
Also first review you should go back and give a number to is DNF.
-
-
I think this is a good idea. Like it or not, people love review score hunting when a new game is out and the shack will now show up in those, which will bring more people here hopefully. They have my bow!
-
-
That's almost certainly the plan.
-
[deleted]
-
I like to think that they do happen but have a way of "unhappening" after a while
-
-
-
-
I echo the "Ughhh" sentiment. I just got done turning off ALL the ratings on Rotten Tomatoes with using CSS and Stylish in Chrome. Showing simply "rotten or fresh" icons and hiding the score for movies suddenly made that site useful for me.
I think the same is true about video games, you really only need two, maybe three categories for them: Good, Bad, and Must Play -
I could very well be wrong but didn't Garnett, in a early podcast, state that for good or ill scores would never be a thing while he was in charge? If I'm remembering right then one of two things has changed either he is longer in charge of reviews or he changed his mind in favor of potentially more traffic.
My hope is that you hold true to the score reflects the review. Have sometimes seen reviews that lambasted a game only for it to get a high score.
Really though I don't put much thought about whether I should purchase a game based on a review or score. I might take certain critiques into consideration but by far I base it on whether the concept/story sound interesting and the mob logic that you see on forums or WC (Though I do realize that a lot of talking points might just be carried over from a review rather than having a conversation). -
-
Reasons?
-
[deleted]
-
-
[deleted]
-
-
-
-
-
Really disappointed to see this. I read ShackNews reviews in part because of the editorial decision not to associate them with scores and let the text speak for itself. It also doesn't speak well for the health of the site, so regardless, I hope the move does drive visitor numbers for you.
-
-
It's worth linking "I Would Give It An 80": http://www.shacknews.com/article/50124/i-would-give-it-an
Numerous developers have told me that their mixed-review games actually deserved an 8/10. That seems to be the baseline. Sometimes I think they do believe it deserves higher, but that's the score they think they can say without being overly presumptuous.
By the standards laid in the scores above, I fully expect Shacknews to give the "I would give it an 80" games a 6.
-
-
[deleted]
-
Why should a game that doesn't have new ideas be offhandedly classified as flawed?
6 - A flawed representation of its genre, failing to execute on new ideas -
[deleted]
-
-
I agree.
-
-
Why introduce scores now?
-
-
[deleted]
-
-
-
[deleted]
-
This rating system is flawed. The description for #9 really should be #10. "Milestone" means a significant event or turning point. That does not automatically assume that it is a turn for the better. It could also be a turn for the worse. Therefore, "milestone" should not be used to refer to the highest rating. "Exemplary" and "resounding impact" are more suitable for #10. Please correct it. Thank you.
-
Hope the numbers don't become completely meaningless. Some stuff I read and watch give out 7/10s for things they don't even like lol.
-
[deleted]
-
One of the benefits is the amount of proper discussion of the actual game that takes place in the comments instead of 50 comments bitching about a number.
-
Good point, "I would have given it a ..." will dominate rather than "I hated/loved" this part because
-
-
I'm not a big fan of 10 value scoring. Ultimately half the scale never gets used and it generally falls into the same pattern of:
6 = bad
7 = decent
8 = good
9 = great
10 = exceptional
In my opinion, a five-star scale or a letter grade scale works much better.
6 = 1 star = F = bad
7 = 2 stars = C = decent
8 = 3 stars = B = good
9 = 4 stars = A = great
10 = 5 stars = AA or S = exceptional-
Also, a 5-tier rating system makes greater allowances for differences in personal taste. Giving a game a specific rating down to the 10th degree loses some of it's value in my opinion.
-
-
According to Shacknews' weird scale, it should be a 6 at least. It's basically its own genre with Heavy Rain.
-
