This is a feature request. A proper answer should be provided by the webmasters or so. Any opinion in regards with this feature request is just an opinion, be it pro or against, but not an answer. Also, maybe the answers from this question may reflect the webmaster's vision, but the questions are different, so this feature-request is not a duplicate question. Thank you.
I've seen someone who answered a question down-voting my answer and the next answer of another member was also immediately down-voted and I suspected that someone is trying to influence the OP's "Best answer" mark by down-voting everybody's else answers.
Since the down-voter didn't provide any explanation I demanded one and the response was pure subjective and personal.
Therefore, I think it will be a good idea that the people who answered a question shouldn't be able to down-vote other answers. Also, if they down-vote someone and then post an answer that their down-votes should be ignored (with a notice, eventually).
In this way abuses would be avoided and the general idea behind the site will not be affected because there are enough readers to express their disagreement.