Try cleaning the contacts with a business card or other stiff paper. Sometimes the clean spot is just so small that it'll only register when the switch is closed in a certain way

Whatever you do do not use sandpaper or anything abrasive, Those contacts have a minuscule amount of gold plating that can wear off and make the switch unreliable. And yes, those switches have no clue as to what direction the ball is traveling. However the press angle changes suggesting a portion of the contact has worn off. Replace the rollover switch if cleaning does not help.