(Topic ID: 50504)

Poll: Should criteria for Top 100 be changed?

By tbanthony

10 years ago


Topic Heartbeat

Topic Stats

  • 76 posts
  • 29 Pinsiders participating
  • Latest reply 10 years ago by frolic
  • No one calls this topic a favorite

You

Linked Games

No games have been linked to this topic.

    Topic poll

    “Should the requirements for posting a Pin Rating be changed for counting towards the Top 100? If so, then how?”

    • Top 100 ratings should remain "as is" 29 votes
      10%
    • Ratings should require comments to justify/explain ratings to be counted in the Top 100 scoring 19 votes
      7%
    • Ratings set as "Private" should not be included in the Top 100 scoring 50 votes
      18%
    • There should be a minimum user Karma rating required for the rating to count in Top 100 scoring 35 votes
      12%
    • You should have to be a current or previous owner to rate the game to count in Top 100 scoring 7 votes
      2%
    • Should be able to flag ratings as "bogus". More than 10 "bogus" flags would exclude the rating from Top 100 scoring 38 votes
      13%
    • Scores of all "1's" would be excluded - no Top 100 game is that bad 40 votes
      14%
    • Scores of perfect "10's" would be excluded - although good, no Top 100 game is perfect 28 votes
      10%
    • 15 worst scores and 15 best scores are omitted to remove some of the "noise" 38 votes
      13%

    (Multiple choice - 284 votes by 131 Pinsiders)

    Topic Gallery

    View topic image gallery

    gigo350.gif

    There are 76 posts in this topic. You are on page 2 of 2.
    #51 10 years ago

    I think the way the ratings are set up now (as far as categories) is just fine the way it is.

    I like that I get to decide how much "Cab Art" really means to me (not that much compared to "Game Play").

    In my rankings "Cab Art" is given a weight of 4 while "Game Play" has a weight of 8. Which is just the way I want it.

    #52 10 years ago

    Using standard deviation will be difficult for you, I think. The SD basically tells you that you have a lot of reasonable data (if it is low) or a lot of bogus data (if it is high). A typical application of SD is to give you confidence intervals - so if MM with a rating of 9 and SD of 1 means you can be 80% sure the actual value likes between 8 and 10. A 2*SD is 95% IIRC. Been awhile since I've had any stat courses.

    You could kill 2 birds with one stone, though. If you were to base the top 100 on the average - 1 or 2 SDs, you would probably have a very accurate picture of where games really belong. The main downside to this is that the value reported is much lower than its actual rating (perhaps you show both numbers - actual average and average-SD). This also solves the problem of new game wackiness until a large number of good ratings are in there, because with all the trolling that goes one right now (for both companies in both directions), you can't believe any number that is being reported until the # users gets higher.

    Another idea is to go to a system of flagging bogus reviews and review-banning said offenders. Do it like you have posts - first ban is a week, 2nd ban is 2 weeks, etc. Also, they are not automatically unbanned until they change one of their ratings.

    And by banning, I'd leave their ability to change ratings and add new ones - just that they don't count in the calculation for the ban period.

    #53 10 years ago

    i couldn't care less. I used (past tense) the ratings to seek out the games that are allegedly good and have made up my mind for myself. I don't think they should really be used for any other purpose other than to give people an idea of what games they may want to try for themselves. I know this seems hypocritical as i have mm rated something crazy like 9.9 and i own it. I'll go downvote it now.

    #54 10 years ago

    There's far too much focus on the numbers, in my opinion. If you eliminate scores of "1", then haters will rate a game a "2" - is that really better?

    The problem with the comments page (which is what people SHOULD be focusing on) is that it isn't interactive, like a forum thread. In a perfect world, there would be a forum thread statically linked to each game for rating comments. Then when person A says "feature X is broken", person B can say "Ah yes, but there's a $5 fix for that!". And so on. Plus troll ratings could be rated down, and thoughtful objective reviews rated up (just like on Amazon).

    I suggest that you add a "Game Reviews" subforum, and a way to link threads in that forum to the specific game page. So when I'm on the AC/DC Premium page, I'm just a click away from the review thread.

    Currently, the best way to evaluate a game is to search Pinside for "XYZ owner's thread" or "XYZ review". That's where you find the real scoop on possible dealbreakers.

    #55 10 years ago
    Quoted from Deaconblooze:

    I thought it would be cool if instead of giving each machine an actual 'rating', you could instead 'place' each game within a category. So right now, you have cabinet art as it's own category. Imagine you have a thumbnail of all the games, you're currently rating something, and now you have to decide where that game compares to the others you've already rated. That would provide a benchmark effect, so to speak. TOTAN may be the #1 game in the cab art category, and everything else falls in line after that.

    I think that is a cool idea to look at, but we shouldn't use the results to form the Top 100 overall.
    Say you have 9 categories (cabinet art, playfield art, lighting, rules, build quality, toys, fun, flow, dmd/lcd)
    Take WOZ for example (sorry to use this table) but it could easily win 6 of nine categories today. If all categories are weighted equally the fact that they are #1 in 6 out of 9 would make them the new Top 100 game.

    Look at it a different way for example. The band the Rolling Stones is my favorite group of all time. But take each one of their parts and they are not the best in their field.
    Keith Richards - used to be a top 10 guitarist but not top five.
    Charlie Watts - nobody considers him a top 10 drummer (maybe if it was jazz)
    Mick Jagger - while he has major stage presence, not many people would say his voice is amazing
    Ronnie Wood - plays a great guitar but I never see him in any lists
    or take Keith and the Wino's (cool sound) or Mick when he does he solo stuff (subpar), But nothing compared to the rolling stones when you put them together.

    I can go on and on with this stuff, but you get my point (i hope) on their own they don't rank as the best of the best. But when they do these surveys of the best bands of all time, Rolling Stones and the Beatles are always battling for the top spot. What does this all mean??? i have no frigging clue...Lists are tough and always changing with times, styles and people's wants

    #56 10 years ago
    Quoted from islandpinball:

    If all categories are weighted equally

    No, I agree they should not be weighted equally. Much in the same way they are currently, and available for users to change according to their own preferences.

    I'm not one of the people that has strong feelings towards the top 100, I don't mind it, but what I posted is just my favorite way to do it if we started from scratch. It's pretty much how I rate things now. I just try to be honest within the game I'm rating, think of the best game in each category and ask myself how the game I'm rating compares to the best.

    #57 10 years ago
    Quoted from Deaconblooze:

    No, I agree they should not be weighted equally.

    I feel there needs to be more adjustability of the weightings. I'd like to throw 80% of my rating towards fun & lastability and let artwork only account for 5% of my rating. I know that doesn't make 100%...

    #58 10 years ago

    I'd eliminate LE's, Pros, Emerald Editions, etc... as different machines. Just have one rating for Tron, etc (like ipdb)... Seems silly that 2 versions of the same game can be in the top 20. It's like saying a 2013 Mustang with a V8 and a 2013 Mustang with a V8, a hood scoop, and LEDs are both counted as different in a list of top ten sports cars or something.

    #59 10 years ago
    Quoted from LouMatt:

    I'd eliminate LE's, Pros, Emerald Editions, etc... as different machines. Just have one rating for Tron, etc (like ipdb)... Seems silly that 2 versions of the same game can be in the top 20. It's like saying a 2013 Mustang with a V8 and a 2013 Mustang with a V8, a hood scoop, and LEDs are both counted as different in a list of top ten sports cars or something.

    Totally disagree. As long as the playfield and game play are different there should be two ratings; one for Premium/LE and one for Pro.

    #60 10 years ago
    Quoted from LouMatt:

    I'd eliminate LE's, Pros, Emerald Editions, etc... as different machines. Just have one rating for Tron, etc (like ipdb)... Seems silly that 2 versions of the same game can be in the top 20. It's like saying a 2013 Mustang with a V8 and a 2013 Mustang with a V8, a hood scoop, and LEDs are both counted as different in a list of top ten sports cars or something.

    100% agree.

    #61 10 years ago

    Voted for allowing others to flag public specific rating as bogus. Agree private ratings should always excluded. My number one suggestion would be requirement to incorporate date stamp of ratings and anyone that has not been active on Pinside for twelve months, has all their ratings excluded (only for period they are gone). This weeds out fake accouts and legacy voting. Forth idea that makes sense is to not list games in list until they have been out in public for six months. Votes can be accrued, they just don't post in the list. Finally, don't list LE and regular editions as separate. They can be combined under generic LE rating. It is ridiculous the TAF rates so much higher than TAFG when they are exact same game. It shouldn't take up two spots.

    #62 10 years ago

    I hope the guys/gals who are looking for the cure for cancer are as persistent as some of you ratings guys.

    -3
    #63 10 years ago

    Here's an idea: Get rid of the pinside ratings altogether. BOOM!

    #64 10 years ago

    There are some unbelievable responses to this thread. The idea here is to weed out the bogus votes. I'm just dumbfounded by the lack of logic in a lot of cases.

    Quoted from jeffspinballpalace:

    allowing others to flag public specific rating as bogus

    This might help, though it will often be done out of spite against a user someone does not like rather than any real reason. As long as the moderators use this wisely it is not a big deal though. The voting should not be an automatic ban though; just a flag to the moderators to check it out.

    Quoted from jeffspinballpalace:

    private ratings should always excluded.

    This makes no sense. How is a private rating any less relevant than a public one? I have done lots of surveys for my previous company and I can tell you that the most accurate voting is anonymous. Either way bogus votes are what we are looking to rule out and private voting is not an automatic source of bogus votes. Don't presume guilt and eliminate legit votes; some people like to remain anonymous. (BTW I tried to set my ratings public and it didn't seem to work).

    Quoted from jeffspinballpalace:

    requirement to incorporate date stamp of ratings and anyone that has not been active on Pinside for twelve months, has all their ratings excluded. This weeds out fake accouts and legacy voting.

    I am really scratching my head here. I have been a pinside member for MANY years and never read the forums once until a year or so ago. I probably went at least 5 years without even going to the main website or rating a pin. How is my rating of MM that I put up years ago after having owned it for a couple of years any less relevant now than it was then? I can see not allowing votes until the software is relatively complete, but legacy votes should NEVER expire. Fake accounts are another problem to think about (good point).

    Quoted from jeffspinballpalace:

    don't list LE and regular editions as separate. They can be combined under generic LE rating. It is ridiculous the TAF rates so much higher than TAFG when they are exact same game. It shouldn't take up two spots.

    Partially agree here. Games with significant different gameplay should be separate. Premiums and LE are close enough to lump them together, but pros are different enough to not be. TAF and TAFG is a different story. Tough call there.

    Karma rating should not affect ability to vote or the weight of the vote. Why should someone who is a lurker but nonetheless and expert player/collector be excluded while some blabbermouth troll who knows nothing but arguing about pins gets counted? If anything karma should be a reverse indicator in my opinion!

    Having to be a current or previous owner is another rule that makes no sense. Some people can't afford games but are still good players and should be given a vote just like the next person.

    Lets face it, there are only a handful of games that are going to receive significant bogus votes (recent Sterns and any game JJP ever releases). The moderators should be able to go in and find those bogus votes easily and delete them. At that point repeat offenders can be warned and removed as required. Creating rules that eliminate even a small percentage of legitimate voters just because they have low karma, private settings, don't update their votes every month or any other silly reason is simply not fair and more importantly will not produce the most accurate numbers based on the full diversity of the pinball community.

    #65 10 years ago

    I voted for (and would strongly support implementing the first two of)
    Ratings should require comments to justify/explain ratings to be counted in the Top 100 scoring
    Ratings set as "Private" should not be included in the Top 100 scoring
    There should be a minimum user Karma rating required for the rating to count in Top 100 scoring

    I also voted for
    Should be able to flag ratings as "bogus". More than 10 "bogus" flags would exclude the rating from Top 100 scoring
    - however, I think it should be a thumbs-up / thumbs-down system, where if a review gets more than, say, 60% thumbs-down, the rating associated would be excluded. (Reviews/rating comments would be mandatory.)

    #66 10 years ago
    Quoted from John_I:

    I am really scratching my head here. I have been a pinside member for MANY years and never read the forums once until a year or so ago. I probably went at least 5 years without even going to the main website or rating a pin. How is my rating of MM that I put up years ago after having owned it for a couple of years any less relevant now than it was then? I can see not allowing votes until the software is relatively complete, but legacy votes should NEVER expire. Fake accounts are another problem to think about (good point).

    Just to clarify one point. I am in agreement that legacy votes should be counted, it doesn't matter how old a vote is, it is still a vote. However I believe that votes by someone who joins Pinside and then disappears or by someone who never posts should not. This requirement could be a litmus of whether the voter is still around and are they contributing to the community with their thoughts and ideas. If you joined up and rated some machines and then left the hobby for five years but returned, once you started posting again, you would become "active" and the votes that you made back when, would start counting. Had you been a less scrupulous person and signed up just to create BS votes, then once you moved on, I would say good riddance to the mischief you tried to create. The system would have "cleaned up" your bad ways by ignoring you. But since you were a legit member and were just taking a hiatus, your return would automatically have triggered your votes to count.

    As to wheher you never posted previously, if you lurk in the background, then you are not adding anything to the community and your non posts should hold the same value as your vote, none. Granted, if posting was a requirement and was clearly spelled out in the voting process, you would either have not voted (since it won't be counted) or you would have posted some, thus meeting the requirement and having your vote count. Your behavior would have changed to match the rules and maybe we could have benefitted from your opinions.

    Most of the comments on how to improve voting are attempts to figure out how weed out the BS and increase the legitamacy of the ones that are counted. Some don't care and prefer the status quo while some want to keep their underhanded practices hidden.

    #67 10 years ago
    Quoted from John_I:

    This makes no sense. How is a private rating any less relevant than a public one? I have done lots of surveys for my previous company and I can tell you that the most accurate voting is anonymous. Either way bogus votes are what we are looking to rule out and private voting is not an automatic source of bogus votes. Don't presume guilt and eliminate legit votes; some people like to remain anonymous. (BTW I tried to set my ratings public and it didn't seem to work).

    Makes perfect sense.

    1. A private vote is more likely to be a bogus vote than a public one......if you can't see that, then I really don't know what else to say.

    2. They were never supposed to count in the first place.

    3. They don't count right now.

    4. There are no plans (thank goodness) to let them ever count.

    And last but not least, public voting does not make you any less anonymous.....necessarily. Who is "PT"? I have no idea. Making your votes public does not change that.

    #68 10 years ago
    Quoted from Astropin:

    I really like:

    Removing the 1's
    A minimum Karma rating
    And instituting "Bogus" flags.

    I'm also glad that hidden votes will no longer count.

    This is all a huge step in the right direction.

    #69 10 years ago

    Not sure if it's been said, but is there a way to make only the votes of "active" pinsiders count for the top 100? Like, if you have logged in within the last 3 months. Anything older than that, their votes get dropped. That will stop the legacy votes for older games that always push out the new ones.

    #70 10 years ago

    I don't see how the timing of a vote should weigh into the system at all??? Old votes are equal to new votes and just because a member chooses to use the ratings system but not post in the forum should not impact it either. Lurkers are people to!

    A good vote is a good vote, plain and simple!

    The whole idea here is to improve the ratings system by cleaning the data of obvious bias votes.

    #71 10 years ago
    Quoted from Whysnow:

    A good vote is a good vote, plain and simple!

    The whole idea here is to improve the ratings system by cleaning the data of obvious bias votes.

    I agree with this..

    #72 10 years ago
    Quoted from Whysnow:

    I don't see how the timing of a vote should weigh into the system at all??? Old votes are equal to new votes and just because a member chooses to use the ratings system but not post in the forum should not impact it either. Lurkers are people to!

    Well it wouldn't necessarily have to be "posting in the forum". It could also simply check wether the user was logged in to the site at all in the past x months.

    So what it boils down to is wether we want to use ALL votes ever placed into the Pinside system, or only show the votes of the "currently active" Pinside community. One reason to choose the latter could be that when some one rated Twilight Zone back in 2004, that person may have considered it the best ever made. But now it's 2013 and there may be newer games better than that. The rating may no longer ring true for that person?

    #73 10 years ago

    I'd replace the whole numeric rating system with a simpler one that has three possible votes:

    * Thumbs-up
    * Meh, just average
    * Thumbs-down

    That shifts the focus to the rater's comments.

    #74 10 years ago
    Quoted from swampfire:

    I'd replace the whole numeric rating system with a simpler one that has three possible votes:
    * Thumbs-up
    * Meh, just average
    * Thumbs-down
    That shifts the focus to the rater's comments.

    Not trying to be contentious, but I think the base of our rating system is excellent; being able to break down a game's merits into artistic, audio, and gameplay features in a quantifiable way is great. I actually wish we had a 1-10 scale, instead of 1-6.

    #75 10 years ago
    Quoted from jeffspinballpalace:

    As to wheher you never posted previously, if you lurk in the background, then you are not adding anything to the community and your non posts should hold the same value as your vote, none. Granted, if posting was a requirement and was clearly spelled out in the voting process, you would either have not voted (since it won't be counted) or you would have posted some, thus meeting the requirement and having your vote count. Your behavior would have changed to match the rules and maybe we could have benefitted from your opinions.

    Bowen and KPJ don't post very much here and neither does Lyman or Steve Ritchie or a friend of mine who has been collecting since the 60's and has about 300 machines. Why on earth would anyone want to create rules that exclude them? I value their opinions a lot more than a lot of the people who post here constantly. The people who post here all the time are generally more biased and more likely to be a fan boy in one camp or another.

    I guess I am missing the point. I always thought that this was to be the defacto top 100 pins as voted by a broad pinball community. Instead what I am hearing that it is ratings based on a small subset of people who post here all the time and generally think alike. Might as well make donations to pinside a requirement to vote.

    Once again I think this is getting way too complicated. There are only a handful of games that will be getting bogus votes. The moderators can easily keep track of them, delete them and warn the offenders without changing the rules to exclude many opinions that *should* matter.

    Quoted from Astropin:

    Who is "PT"? I have no idea.

    Haha, me too. That is a name I used on RGP a long time ago when I signed up for Pinside. I signed up by the way in order to be able to vote. I had no interest in a forum. Not even sure there was a forum back then? Anyway, I have requested to have my name changed to "John I" a couple of times, but have not had my wish granted.

    #76 10 years ago
    Quoted from robin:

    Well it wouldn't necessarily have to be "posting in the forum". It could also simply check wether the user was logged in to the site at all in the past x months.

    Exactly. They were logged in once to vote, all you need to do is check to see if they've ever come back again for any reason, nothing to do with posting.

    The longer pinside is around, the longer these legacy votes will skew the results. Robin, you're exactly right, someone votes a game #1, but 10 years pass, and it is not relative to those new games.

    It would give the charts more volatility, but that would be a good thing. It would be like music charts, where titles work their way up, and then down again.

    There are 76 posts in this topic. You are on page 2 of 2.

    Reply

    Wanna join the discussion? Please sign in to reply to this topic.

    Hey there! Welcome to Pinside!

    Donate to Pinside

    Great to see you're enjoying Pinside! Did you know Pinside is able to run without any 3rd-party banners or ads, thanks to the support from our visitors? Please consider a donation to Pinside and get anext to your username to show for it! Or better yet, subscribe to Pinside+!


    This page was printed from https://pinside.com/pinball/forum/topic/poll-should-criteria-for-top-100-be-changed/page/2 and we tried optimising it for printing. Some page elements may have been deliberately hidden.

    Scan the QR code on the left to jump to the URL this document was printed from.