(Topic ID: 285776)

Abuse of rating system.

By rai

3 years ago


Topic Heartbeat

Topic Stats

  • 229 posts
  • 89 Pinsiders participating
  • Latest reply 2 years ago by Doctor6
  • Topic is favorited by 4 Pinsiders

You

Linked Games

No games have been linked to this topic.

    Topic Gallery

    View topic image gallery

    934B57EF-552F-4903-86D9-BE3E432A4159 (resized).png
    8CBF4AC1-0775-40EB-BBB4-D379C84E6ACE.gif
    Keyboard commandos.gif
    04DA4240-C1A2-4B4B-BBD9-1B1FC11F9E4E (resized).jpeg
    IMG_20201231_184528229 (resized).jpg
    05969F5A-388B-4C11-88E0-A6B7419F0ED9 (resized).jpeg
    Let the Hate Flow.png
    BF3689B2-BF92-46C1-9523-47A4BD96D639 (resized).jpeg
    toysrating (resized).png
    5168BF12-AC2F-4403-A3C1-C064798D3E78 (resized).jpeg
    E94AB8F8-EE85-49C1-A898-933DCD69AB8B (resized).png
    31BADAB9-9E73-489A-ACAD-7196BB449B23 (resized).png
    FC93CA11-0D34-4A6B-8D8A-B024454FD023 (resized).png

    There are 229 posts in this topic. You are on page 1 of 5.
    19
    #1 3 years ago

    I realize this has been covered before but just had to say the rating should be honest and for fun. I know we all like different. But it’s also clear when a game is being rated just to lower the overall rating.

    Here’s an example of a Pinsider 2 days on the site so he probably has other accounts here as well. He rated several pins at 10 and then several pins at 1.0

    1.0 is the lowest possible in every category. So if it has good music or lights or theme integration he still just rated as the worst possible pinball machine.

    Also likely every pin is rated somewhere between 7 and 10 and so if someone tried to reverse sand bag it would only be a 2-3 point grade upgrade but if someone rated at 1 that’s a 7-8 point downgrade.

    Might be there should be a way to toss out any votes of 1.0 or 2.0 or so on as bogus.

    FC93CA11-0D34-4A6B-8D8A-B024454FD023 (resized).pngFC93CA11-0D34-4A6B-8D8A-B024454FD023 (resized).png31BADAB9-9E73-489A-ACAD-7196BB449B23 (resized).png31BADAB9-9E73-489A-ACAD-7196BB449B23 (resized).png

    #2 3 years ago

    This Lowezz 2 days on Pinside actually has unlocked an achievement as ‘ratings expert’ on Pinside. He’s not picky what he rates down either like all the top pins.

    E94AB8F8-EE85-49C1-A898-933DCD69AB8B (resized).pngE94AB8F8-EE85-49C1-A898-933DCD69AB8B (resized).png

    #3 3 years ago

    Pathetic. If you're gonna do this, just go away. But we've known this for years. The top 100 is a useful guide at best. That's it. It's a popularity contest.

    #4 3 years ago

    "I realize this has been covered before but just had to say the rating should be honest and for fun. I know we all like different. But it’s also clear when a game is being rated just to lower the overall rating."

    I agree with you. It is obvious that some users try to inflate the ratings of new pins that they are trying to flip. 1.0 ratings should never be allowed, as they are bogus. The pinside users that submit these ratings should be banned from pinside. I know they will then just create a new user and give a new sandbag rating. To attempt to solve this, maybe ratings should only be allowed from verified logins.

    13
    #5 3 years ago

    I think that’s what the flag feature is for. Enough flags and the rating is ignored.

    29
    #6 3 years ago
    Quoted from rai:

    Might be there should be a way to toss out any votes of 1.0 or 2.0 or so on as bogus.

    Good morning. You bring up great points. That said, for obvious reasons we do not state exactly how the top 100 system works (we do not want to aid in people working the system). You can always see how someone rates if a game if they have their personal settings fixed to be seen by everyone...and being a “ratings expert” simply means the person rated a certain number of games...neither of those means their rating necessarily impacted a cumulative top 100 rating (but again, we will not give any details on that point...but know that Robin is pretty smart about how he approaches the top 100 ratings).

    As a related aside, you’ll note from another thread recently that robin has been considering some changes to the top 100. I’m super enthusiastic about what he has been talking about and can’t wait to it when he gets a chance to implement it.

    #7 3 years ago
    Quoted from chuckwurt:

    I think that’s what the flag feature is for. Enough flags and the rating is ignored.

    Is it ignored or just not shown on the ratings page? I’m not clear about that.

    I only feel honest to rate a pin that I’ve owned or know 100% and played different examples. I mean I read a book review on Amazon and some will say 1 star the book had a torn page. That’s not the way to rate a book it’s supposed to be regarding the book contents maybe how it is constructed but a shipping damage should not be tabulated in the ratings.

    Likewise some will play a game on location and maybe a feature didn’t work like I played JP premium and the Trex mouth was broken. I wouldn’t rate the games poorly because of one bad feature that was broken or maybe the game was not set up correctly or working etc.

    #8 3 years ago

    I think the highest and lowest scores should be tossed when figuring the average.

    Nothing is perfect and really no modern game deserves to be anything below a 5 or 6.

    There just hasn’t been anything that bad.

    #9 3 years ago
    Quoted from rai:

    I realize this has been covered before but just had to say the rating should be honest and for fun.

    I've rated a ton of games and wish I never had. I think the whole top 100 should just go away. Its like movie reviews or music reviews, they suck.

    #10 3 years ago
    Quoted from Hazoff:

    I've rated a ton of games and wish I never had. I think the whole top 100 should just go away. Its like movie reviews or music reviews, they suck.

    It’s a popularity contest but often I like to read the comments to see why people like or dislike the game.

    I like reading movie reviews too but usually after I have watched the movie and have my own opinion.

    #11 3 years ago
    Quoted from Hazoff:

    I've rated a ton of games and wish I never had. I think the whole top 100 should just go away. Its like movie reviews or music reviews, they suck.

    A rating system is fine as long as there are controls to eliminate all the troll and fanboy ratings.

    It can very easily be done and would give the top 100 some credibility.

    #12 3 years ago
    Quoted from rai:

    It’s a popularity contest but often I like to read the comments to see why people like or dislike the game.

    Hands down. The rating system may be flawed due to various biases of the people rating it, but once you find a half dozen or so Pinsiders who have similar taste to yours you can tell a lot about a game by how they review a game in their comments. I always enjoy reading the comments from players I know or have interacted with for years on the site.

    #13 3 years ago
    Quoted from Psw757:

    I think the highest and lowest scores should be tossed when figuring the average.
    Nothing is perfect and really no modern game deserves to be anything below a 5 or 6.
    There just hasn’t been anything that bad.

    I disagree on this because that would just be silly to have a swath of middle-of-the-road modern games clumped together.

    Highs and lows have their place. Like some people genuinely hate Baby Pac-Man. Personally it’s one of my top 5 greats. But I understand the hate because the first few times you play it that game is ridiculously brutal. Lots of places have the settings totally wrong so it’s not fun at all.

    On the opposite end, some moderns have been incredible. WOZ/JP jump out off the top of my head. They really deserve to stand up taller than their contemporaries.

    Quoted from pinballmike150:

    "I realize this has been covered before but just had to say the rating should be honest and for fun. I know we all like different. But it’s also clear when a game is being rated just to lower the overall rating."
    I agree with you. It is obvious that some users try to inflate the ratings of new pins that they are trying to flip. 1.0 ratings should never be allowed, as they are bogus. The pinside users that submit these ratings should be banned from pinside. I know they will then just create a new user and give a new sandbag rating. To attempt to solve this, maybe ratings should only be allowed from verified logins.

    Not necessarily. It just needs a mystical behind-the-scenes weighting system.
    Some of my ideas would be:
    Respectable, long time Pinsiders ratings should probably be treated with more respect versus the newbie OP shared.
    Post history and possible achievement points also play into rating weights. Comments that are well thought out also give legitimacy to reviews.

    Also people USE THE FLAG SYSTEM! Flag bogus reviews.
    However there are some games that suck, AC/DC Pro is so ugly and unfun that I can’t imagine giving it anything more than a 4 on a good day.
    The premium is so much better it’s ridiculous. It’s practically a different game. That ugly big face they put instead of the mini playfield is unbelievably awful.

    46
    #14 3 years ago

    If you give any game a 1, you probably shouldn't be in this hobby.

    #15 3 years ago

    People seemed to manipulate the scores to keep the Big 4 at the top (MM,TZ,AFM, and MB) for a really long time in my opinion. Now it seems at least some of the newer better games are staying there longer but it seems to be more volatile than in years past. It was interesting to me SW popped up to number two. Probably the last game to go in my collection ,but, did not realize it had that many fans and it has been out for a while. For me the rankings are fun. I most likely like to read the reviews. On games I am thinking on buying, I look for certain Pinsiders take on a particular title. I find my opinion aligns with some versus others. As I read reviews if I see a score that seems to be a manipulation,one way or the other, I Flag it.
    If someone believes a game is a ten and there no words explaining why, its out. Pretty much any DMD is rated a 1, I think is pretty bogus as well.

    #16 3 years ago

    I think the top 100 needs some work. I have not looked at it in some time but everytime I do, I see flaws.

    Let's take for example number 84 currently on top 100. Slugfest? Really? How can you have a novelty machine that is not even a true pinball machine on top 100? The only rating I agree with is Star Wars being rated #2-bout time people came around.

    #17 3 years ago
    Quoted from beelzeboob:

    If you give any game a 1, you probably shouldn't be in this hobby.

    I dunno...I've played Punchy The Clown.

    #18 3 years ago

    If I am looking for an honest review of a game, I just read @caucasian2step's review. Seems like he has reviewed just about every game and his comments are very detailed. Maybe he should determine the Top 100

    #19 3 years ago
    Quoted from TigerLaw:

    Hands down. The rating system may be flawed due to various biases of the people rating it, but once you find a half dozen or so Pinsiders who have similar taste to yours you can tell a lot about a game by how they review a game in their comments. I always enjoy reading the comments from players I know or have interacted with for years on the site.

    To this point, I think if robin put in an option to ignore reviewers and/or reviews that are ridiculously high or low or from people who we seldom agree with, it would actually allow you to shape overall reviews to give each Pinsider a more realistic example of how they might view the game.

    I know I'm not explaining it well, but I think if implemented correctly, it would be awesome.

    #20 3 years ago
    Quoted from PismoArcade:

    I dunno...I've played Punchy The Clown.

    Still better than no game at all.

    Maybe not.

    #21 3 years ago
    Quoted from Psw757:

    I think the highest and lowest scores should be tossed when figuring the average.
    Nothing is perfect and really no modern game deserves to be anything below a 5 or 6.
    There just hasn’t been anything that bad.

    Well hold on. Let's not be a bunch of Greg Bones about rating things, either. On a 10 point scale, 5 should be average. So I think 5s are still fine. 1? Obviously not. 10? Probably not as well. But don't just throw out anything 5 or 6 and lower. A

    #22 3 years ago

    Another major flaw is many peoples ratings are old and need to be updated. Have you ever written off a machine-only to dig deeper and find out your opinions have changed?

    #23 3 years ago

    I love how people think throwing out 1-6 star reviews and 10 star reviews would fix the problem

    #24 3 years ago

    ffs get over the ratings system already....it means NOTHING.

    #25 3 years ago
    Quoted from Hazoff:I think the whole top 100 should just go away. Its like movie reviews or music reviews, they suck.

    And now you’re reviewing reviews - very meta!

    5168BF12-AC2F-4403-A3C1-C064798D3E78 (resized).jpeg5168BF12-AC2F-4403-A3C1-C064798D3E78 (resized).jpeg
    27
    #26 3 years ago

    Every once in a while this comes up. I appreciate it. Shows me that people care.

    Some random comments:

    1. Any ratings system will be abused. No matter how you implement it. No one will ever be satisfied.

    2. A lot of votes are not included in the ranking. There's a lot of code to decide which ones do. This is not visible anywhere. That's all I'm going to say about it.

    3. The current top 100 is what YOU make of it. The list itself is not something you should agree or disagree with. It's a statistical work of art but it's not meant to be YOUR list op top games. It's a starting point to dive deeper into.

    4. Read. The. Rating. Comments.

    5. A concept for a completely new Pinside ranking is aging in the barrels. Once I pour it in the glass that is Pinside, it will be... awesome... Yet still a lot of people will probably dislike it. You can never please everybody.

    6. Should Star Wars be #2? Hell no. (sorry Steve)

    #27 3 years ago
    Quoted from robin:

    . Should Star Wars be #2

    Yes, Yes it should. But instead of MM being #1 that needs to be #1 STTNG!

    The 'King' is worthy of not only #1 but #2 as well

    #28 3 years ago

    His highest rated pin is MM so he's obviously a nutball.

    #29 3 years ago

    If I look at ratings I put more weight on those that actually have thoughtful comments. Those with little or no data to support why don't add much value. I do like the comments good/bad about each game to learn about new machines. I'll also consider that the machine they had may not have been setup properly or had electronic or mechanical issues. Also, was the game running the latest code? How were each of the settings configured for a solid state that has settings? Some people initially complain a game is too easy yet some simple tweaks to some settings can make a world of difference!

    The way a game is dialed in can help. I initially played some games that weren't setup or adjusted properly and it was a bit frustrating. Later I ended up falling into a deal on the same title that wasn't working, fixed it, and once adjusted it is a great machine.

    What I would like to see on the ratings is adding another view by date ranges. The top 100 is a nice guide for overall but can be skewed by people that may just be familiar with newer games and rating older games against those. Maybe some date ranges for like machines (1977-1984) (1985-1991), etc. Ranges can be set to get the most similar groups of games together. Then see the top games in those groups.

    #30 3 years ago
    Quoted from Psw757:

    Nothing is perfect and really no modern game deserves to be anything below a 5 or 6.
    There just hasn’t been anything that bad.

    Disagree. What’s the point of a 10 pt scale if no one is willing to use the whole scale. Youre saying it should basically be a 5 pt scale plus 5 because...well all pinball is fun. Shrug

    #31 3 years ago
    Quoted from rai:

    Is it ignored or just not shown on the ratings page? I’m not clear about that.

    It’s not factored into the rating calculation at all.

    #32 3 years ago
    Quoted from pinlink:

    If I am looking for an honest review of a game, I just read caucasian2step's review. Seems like he has reviewed just about every game and his comments are very detailed. Maybe he should determine the Top 100

    He’s also good at updating and dating the updates as well. Definitely great reviews.

    #33 3 years ago
    Quoted from pinmister:

    Another major flaw is many peoples ratings are old and need to be updated. Have you ever written off a machine-only to dig deeper and find out your opinions have changed?

    I partly blame Stern. Like I gave a bad review to Deadpool when it first came out, but that’s because I legit thought it stunk. Then they actually finished coding the thing and now it’s one of my favorites of recent years. Batman 66 is an even more extreme example of this.

    Video game reviews also really struggle with this. Games are never “done” and reviews are like historical snapshots of where the game was.

    #34 3 years ago
    Quoted from TreyBo69:

    I partly blame Stern. Like I gave a bad review to Deadpool when it first came out, but that’s because I legit thought it stunk. Then they actually finished coding the thing and now it’s one of my favorites of recent years. Batman 66 is an even more extreme example of this.
    Video game reviews also really struggle with this. Games are never “done” and reviews are like historical snapshots of where the game was.

    This is why I wait until a game is at 1.00 at least.

    #35 3 years ago
    Quoted from Doctor6:

    Well hold on. Let's not be a bunch of Greg Bones about rating things, either. On a 10 point scale, 5 should be average. So I think 5s are still fine. 1? Obviously not. 10? Probably not as well. But don't just throw out anything 5 or 6 and lower. A

    You may want to not get so pantsed up and read what I said, I didn’t say throw 5 and 6 out but there is no modern pin that is a 4 or below, period.

    #36 3 years ago
    Quoted from TreyBo69:

    Disagree. What’s the point of a 10 pt scale if no one is willing to use the whole scale. Youre saying it should basically be a 5 pt scale plus 5 because...well all pinball is fun. Shrug

    Come on man, statistical averages use algorithms like this all the time to circumvent shit like this.

    I believe it’s called the trimmed mean and you have to have a way to not count outlier votes that affect the average.

    Star Wars premium a 1.0? Yeah...sure.

    Whether you flag it or not it affects the average from what I can see on other games.

    -3
    #37 3 years ago
    Quoted from pinlink:

    I love how people think throwing out 1-6 star reviews and 10 star reviews would fix the problem

    Comprehension my friend, that wasn’t what I said.

    #38 3 years ago
    Quoted from Zablon:

    ffs get over the ratings system already....it means NOTHING.

    #39 3 years ago
    Quoted from Psw757:

    Come on man, statistical averages use algorithms like this all the time to circumvent shit like this.
    I believe it’s called the trimmed mean and you have to have a way to not count outlier votes that affect the average.
    Star Wars premium a 1.0? Yeah...sure.
    Whether you flag it or not it affects the average from what I can see on other games.

    If someone doesn’t like something then they don’t like it. They’re not wrong because no one is right. If they hate a game that’s their prerogative.

    A single person making multiple reviews to skew the overall rating one way or another is an issue of course. That’s abusing the system. Giving a single bad review is using the system.

    #40 3 years ago
    Quoted from TreyBo69:

    If someone doesn’t like something then they don’t like it. They’re not wrong because no one is right. If they hate a game that’s their prerogative.
    A single person making multiple reviews to skew the overall rating one way or another is an issue of course. That’s abusing the system. Giving a single bad review is using the system.

    We can agree to disagree.

    #41 3 years ago

    I do have difficulty justifying games below the 5. Most people probably struggle with this. Anything below 5 feels insulting!

    If you reworked it saying 10 is the greatest thing since Flippers were invented be that WPC games/Alien/TBL, and 1 is the worst game ever: El Toro or Big Ben, then perhaps people would conceptualize it and utilize the full scale.

    That’s the big reason why a “most favorite” to “least favorite” slider is better than 1-10 because nobody uses all the numbers.

    #42 3 years ago
    Quoted from Psw757:

    We can agree to disagree.

    Sounds like if we dont agree with you we are getting pantsed up, whatever the fuck that means.

    -2
    #43 3 years ago

    Weight ratings based off of years on pinside.

    I love the ratings system and would recommend one potential tweek. Instead of treating everyone's ratings equally, what about giving people a weight based off of the number of years they have been on pinside? This would weight users who have been around a while (and presumably know more about pinball) higher than new members. Something simple like years on pinside x score. Robin's score would count 9 times as much as mine. Maybe even give fractional ratings for those who have been here less than a year. This would have the side effect of making older more established games harder to move down the list.

    In any case. Don't forget to live comments with your ratings!

    -1
    #44 3 years ago
    Quoted from Doctor6:

    Sounds like if we dont agree with you we are getting pantsed up, whatever the fuck that means.

    Again I didnt say throw out 5/6 and below and frankly don’t give a fuck who agrees or disagreed with me on anything. It’s ok to have differing opinion.

    #45 3 years ago

    I like stack ranking things. You vote something up or down the same list and that’s your ranking.

    It’s even better when combined with a tiered distribution curve, something like you have to have at least 1 F, 2 D, 3 C, 2 B, 1 A rankings that you can give out

    That’s me personally. But I’m also the type who has no issue saying Led Zeppelin is the ugliest art package out of Stern in 5 years. Or that Munsters is the most boring/shallow rules of the last decade. But some people think that means I’m a negative Nancy that doesn’t like pinball enough.

    Whatever. I knows what I likes and what I don’t. I don’t even look at the rankings anymore. Sometimes I’ll find a particular reviewer because I think they have something interesting to say. That’s about it.

    #46 3 years ago
    Quoted from SantaEatsCheese:

    Weight ratings based off of years on pinside.
    I love the ratings system and would recommend one potential tweek. Instead of treating everyone's ratings equally, what about giving people a weight based off of the number of years they have been on pinside? This would weight users who have been around a while (and presumably know more about pinball) higher than new members. Something simple like years on pinside x score. Robin's score would count 9 times as much as mine. Maybe even give fractional ratings for those who have been here less than a year. This would have the side effect of making older more established games harder to move down the list.
    In any case. Don't forget to live comments with your ratings!

    This seems so gate keeper. It’s basically saying your opinion is invalid if you’re new to the hobby or if you just never bothered to use this particular site.

    It also assumes there aren’t some long time trolls present...

    #47 3 years ago

    Why cannot someone hate a game and give it a 1 or vice versa love a game and give it 10? If low scores are invalid then so are the high ratings. Basically you are saying we should just throw out votes because you don't agree with them.

    #48 3 years ago
    Quoted from TreyBo69:

    I like stack ranking things. You vote something up or down the same list and that’s your ranking.
    It’s even better when combined with a tiered distribution curve, something like you have to have at least 1 F, 2 D, 3 C, 2 B, 1 A rankings that you can give out

    Something like that would be better and would eliminate many outliers.

    I just was playing around and compiling a rating for a game, it is difficult to even get to 1.0 without just scoring the minimum on everything.

    Shit, even if you hate Star Wars, just giving the music or art a good score puts it well above 1.0 threshold.

    For a game like this 5.0 would be bad but I could see how you can get to that type of rating.

    #49 3 years ago

    I'm curious why the top 100 rankings when grouping ratings for a title takes the highest rated title in the group for it's combined ranking vs. the version of the game with the most ratings? For example, Star Wars is ranked #2 currently because of the comic edition and Medieval Madness is ranked #1 because of the Royal Edition. To me it would make sense to take the version of the title with the most ratings when combining the ratings and placing it in the top 100. thoughts?

    #50 3 years ago
    Quoted from Psw757:

    Something like that would be better and would eliminate many outliers.
    I just was playing around and compiling a rating for a game, it is difficult to even get to 1.0 without just scoring the minimum on everything.
    Shit, even if you hate Star Wars, just giving the music or art a good score puts it well above 1.0 threshold.
    For a game like this 5.0 would be bad but I could see how you can get to that type of rating.

    Just for fun I ranked Star Wars Pro with the system. I ended up with it being a little over 3/10. Which...yeah that’s close to my opinion of that game relative to games of the past several years. I’d probably would have gone in saying 4/10. (Well really Id say it’s a 2/5 which sounds a lot better than a 4/10 even if it’s functionally the same)

    I won’t get into why I don’t like the game but I am pretty meh to it overall. It’s not that I don’t get the game, it’s just whatever to me compared to others. But I’ll rate stuff harshly...

    It also reminds me, the top 100 should really break out the early SS games with numerical and alphanumeric scoring. It’s so hard to compare DMD/LCD games with the older ones. Heck it’s hard to compare a lot of 90s DMD games with the latest LCD games. The expectations change so much over time.

    There are 229 posts in this topic. You are on page 1 of 5.

    Reply

    Wanna join the discussion? Please sign in to reply to this topic.

    Hey there! Welcome to Pinside!

    Donate to Pinside

    Great to see you're enjoying Pinside! Did you know Pinside is able to run without any 3rd-party banners or ads, thanks to the support from our visitors? Please consider a donation to Pinside and get anext to your username to show for it! Or better yet, subscribe to Pinside+!


    This page was printed from https://pinside.com/pinball/forum/topic/abuse-of-rating-system-?hl=anago and we tried optimising it for printing. Some page elements may have been deliberately hidden.

    Scan the QR code on the left to jump to the URL this document was printed from.