Quoted from Marten:Screen Shot 2015-11-13 at 18.58.38.png
I felt the same and then discovered you can uncheck the box at the bottom of the pop-up.
This has got the be the most customisable website on the internet!
Thanks Robin!
I mean the banner down at the bottom of the screen that you see when you upvote directly: