Seems to me that a lot of the conversation about this is in the irreconcilable perceptions of what the upvoting and downvoting means.
1) some people says it's a comment about the message
2) some people says it's a comment about the messenger
I think that is assisted largely because the caret symbols: ^ and V are not words.
Look what else the symbols do. They connote up and down, and so we call them upvoting and downvoting, which further connotes the thumbs-up and thumbs-down judgments which in ancient Rome meant one is good, one is bad.
Disagreement does not have to denote anything bad. Disagreement is not automatically bad. Disagreement is not bad. So why employ symbolism that encourages these tired associations and the useless product of them?
Ditch the ambiguous carets and their front-loaded judgments and replace them with the words Agree and Disagree. Less ambiguity and no further promotion of the association with thumbs-down negativity. Then, with that done, stop fretting about exactly what it is that the person Agrees or Disagrees with.
Or, don't try this idea and continue with what you have.