Just saying "-1" is unnecessary: the user already knows from the notification that somebody downvoted. Votes are also deliberately anonymous, so this kind of comment does nothing except rub in the disapproval. It's not constructive, which is just as good a reason to flag comments.
Saying "-1 because Thing X and Reason Z" explains why content is getting downvoted (often accounting for several downvotes, if we presume upvoting such comment reflects partially on downvoting such content), provides the user posting content a chance to inquire further, and points out the exact issue to be fixed. That is the entire point of commentary, and this kind of comment is not inherently rude unless a person thinks downvotes or explanations of downvotes themselves are rude (which is entirely unreasonable in absence of vote abuse or separately rude/offensive comment use). We simply save a few characters and seconds by typing "-1: ..." instead of "I'm downvoting this because ...".
If it would be rude without saying "-1" then it's rude. Flag it.
If it just says "-1" then it's not constructive. Flag it.
If it explains the reason for some downvotes, then it's useful, whether or not the user posting content agrees with the reason. Let it be - unless it starts a flamewar, in which case the usual process applies.