Is this something we should be concerned about?
Not unless you've identified more of a pattern than is evident from your post. People are free to vote however they choose with only a few limitations. Since voting is anonymous and people hardly ever justify their votes, there's no real way to tell if you've found malicious behaviour.
For instance, I'll do this very often for obviously off-topic questions, and I'm not alone, see: Should one downvote answers to off-topic questions? If people answer a low-value, off-topic or unclear question with a low-value answer that took less than a minute to write then you can bet that I'll downvote the lot of it. So that's at least one scenario where this kind of voting pattern is legitimate.
You mentioned the following in a comment: "I think it's downvoting the answers because they're not liking the questions." Keep in mind that even if that were indeed happening, I don't see how that isn't a legitimate use of a person's vote.
Full disclosure: I didn't downvote the question or answer that prompted your question and in fact had already upvoted your answer previously.