Experiment concluded: a comment by any other name

Question

This experiment has ended; you can view the report here.

---

Last year, Robert Cartaino relayed this one weird trick to fix comments:

I also flag dozens-to-hundreds of posts daily simply to remove misplaced answers and other minutia from comments which simply don't belong there. It's very time consuming, and largely ineffective.

But it never stops.

Only recently, I changed the comment prompt in Area 51 from "add comment" to "suggest improvements" (the primary use case for example questions), and that number dropped to essentially… ZERO!

He proposed a two-part change based on this observation:

• Change "add comment" under the question to "ask for clarification"
• Change "add comment" under the answer to "suggest improvements"

For the past 60 days, we've been testing a partial implementation of this over on Interpersonal Skills. Results are less dramatic than those on Area51, but still promising - so I'd like to try it here next.

To be clear: the issues this site has with comments aren't nearly as severe as that over on IPS; y'all only delete about a third of all comments posted vs. IPS's half. But... That's still a lot of comments that could be doing something better, so this seems like an obvious place to test a possible quick fix.

For the next 60 days, "add comment" will instead read "suggest improvements" on all posts for all users. Please let me know if this causes any issues!

Example:

Answer 1

Just as a quick observation anonymous users already see an "improve this question" link on the questions already instead of "edit". Seeing that above "suggest improvements" is a bit ambiguous for if there is a difference between the two.

Now this may cause problems or it may not but I think it's worth mentioning.

Answer 2

The results of the experiment

Days before the change             65    
Days after the change              65    
Posts created before the change    3655  
Posts created after the change     3708  
Comments created before the change 14029 
Comments created after the change  12003 

Description                     Before     After      PctChange  
------------------------------- ---------- ---------- ---------- 
Comments / post ratio                 3.84       3.24     -15.66 
Flags / comment ratio                 0.14       0.18      28.55 
Pct comments deleted                 34.75      34.37      -1.10 
Comment upvotes / comment ratio       2.80       3.07       9.75 
Pct comments followed by edits        8.87      10.40      17.16 

Just as we saw on Interpersonal Skills, this resulted in significantly fewer comments.

Unlike the IPS test, we didn't see fewer flags per comment; part of this may be simply that an awful lot of changes to comment flagging rolled out mid-way through the test, but given the effect wasn't very strong on IPS either it's my guess that any changes noted here are most likely just noise.

As on IPS, there was a noticeable improvement in the average utility of the comments that were posted: even as fewer comments were posted over all, y'all left slightly more comments that were followed by edits to the post being commented on. This continues to be my favorite observation, as it suggests the change in wording is helping folks to focus on activity that results in lasting improvement to the site vs. idle chit-chat.

Distribution and ongoing effectiveness

Finally, I want to address a concern I've seen raised in a few places, namely that any benefits are likely to be a temporary effect caused more by confusion than anything else.

The metrics we've been looking at thus far were picked to reduce the effects of noise and temporary variation on the results. That's useful for gauging the overall effects of a test like this, but they don't tell the whole story! As we're all painfully aware, it isn't every post that gets large amounts of tangential commentary; the norm - here and just about everywhere else - is 0-2 comments per post. That makes it kinda hard to intuit either a problem or its solution from casual observation, as problems are by definition exceptional cases and thus observing improvement means observing cases where comments might have blown up but did not.

(click for larger version)

This chart seeks to illustrate the distribution of comments posted during the first week of each post's life here over the past year and two months, grouped by every week between June 26th 2017 and last week. Weeks where the test was live are highlighted in orange; whiskers indicate the full extent of the data (so, extreme outliers are plainly visible).

A few observations:

1. The overall range has consistently dropped during the test - not a single post has garnered more than 60 comments in its first week for the duration, nor 50 during the second half of the test.

2. The median number of comments per post has been locked at 2 for the duration of the test, with 75% of posts consistently getting 4 or fewer comments.

3. There's no variation in either the median or quartile numbers week-by-week during the test, which stands in stark contrast to the year preceding it and also suggests that whatever effect this change had was consistent beyond what we could expect from novelty or confusion alone.

Summary

As with IPS, this isn't a dramatic difference. There are still posts that get too many comments, there are still plenty of flags, there's still lots of cleanup work for moderators to perform.

But, there is a difference - and it's largely positive. And for a relatively inexpensive change, that's nice to see!

Answer 3

So far, this experiment has turned out exactly as bharal predicted.

For a few minutes hours days, the change led to a drop in extraneous comments. However, once users figured out that "suggest improvements" was just "add a comment" by another name, normal service was restored comment usage patterns returned to the pre-change status.

This does not mean the change should be scrapped. It is less effective than expected, because it is a partial solution. With the following additions, this experiment would hopefully yield better results:

• Add "discuss this question" alongside "suggest improvements": This link opens the post-specific chat room where users can discuss the post.
• "Protect" commenting on protected questions: Users coming in from other sites (usually for entertainment via HNQ) who are used to "adding comments" on other sites tend to use the "suggest improvements" similarly. This can be easily resolved by disregarding the association bonus and requiring 10 reputation on this site for commenting on protected questions.

---

However, this change has also proved useful because now moderators need to spend less effort to justify deletion of comments. Consequently, complaints about comment deletion are expected to drop drastically.

---

Update: One user was recently flabbergasted by the change, and asked on meta why he wasn't able to post comments. When this experiment was explained to him, he went back to the post soon after, and posted a comment, which was partly commentary and partly semi-answer, but definitely not a suggested improvement.

Answer 4

Sounds good to me. Hopefully this will help cut down needless discussion and half baked answers in comments.

Answer 5

IMHO this is not a helpful change¹. I get that you are trying to cut down on the number of extraneous comments, but the modification attempts to classify comments into either

1. Suggestions for change
2. Everything else (which is now undesirable and should be purged)

But this is a false dichotomy. There are classes of comments that are not suggestions for improvement, are not extraneous and yet are still are germane to questions and answers. Thus I think in implementing this solution you are throwing out the baby with the bathwater.

---

1. I originally came here after noticing that I could no longer post a commented was confused as to why this was so, and what bad thing had I done to be punished so. I also checked some other SO groups and saw that I could add comments there still. So this confused me even more. I would posit that there are now a large number of people who are just as confused as me right now.

Answer 6

Interesting situation to analyze the one that this post reveals: Is it expected that I cannot comment on questions or answers, having reputation 1,986?

On that post, we can read (emphasis mine):

My reputation on workplace is currently 1,986. The commenting privilege starts at 50. However, I can't comment on any questions or answers — there is no "add a comment" link on any of them for me.

This clearly indicates that the change of wording did had an impact, at least to some users.

My hypotheses (I am not a psychologist, but I am a keen observer) is that this user was "unable to add a comment", because he was subconsciously looking for the "comment" word; when he saw the "suggest improvements" he ignored/skipped it.

Perhaps a guess from my part, but that behavior could indicate that such user was indeed not looking to suggest improvements and, perhaps, was just in for the comment or discussion...

This is the first evident sign I see that the experiment worked to some degree; besides the stats that surely can be generated or collected.

Edits:

• A second situation happened today with this post: Why can I not comment?

• A third one happened with this new user when he asked a question: Resume Summary For Senior Unix Administrator. On the edit we can see OP thought "I am editing my question because I can't reply to comments"

• A fourth incident here Took serious pay cut for new job, learning NodeJS and ReactJS, how do I approach a new contract (with different company)? OP again though they were unable to comment... now I think we can say that this definitely had some effect (but, perhaps on new users mostly?).

• Fifth case where OP doesn't know how to comment in this question.

• Another one, on the same day as the one above, on this post. Seems that new users are having a hard time finding a way to comment under their posts.

• New case here also from a "new contributor"... I am starting to see a trend here...

Answer 7

I'm not sure if this will work as you think.

At heart, I suspect that the change will work initially, while people are unsure what is going on.

After all, when you see "suggest improvement" you now think that you're not even writing a comment at all. I just assumed, initially, that the site was going to make me do a feedback faq or something.

Which means that, initially, yes, you will get rid of extraneous comments. You will also get rid of all the other comments, too. I can see you're using it only when the comment queue gets a little unwieldy - which might work.

If, I suspect, only for a little time until people work out what is going on. This will depend on the types of users making excess comments, mind. I have no idea what that is - if they're veterens this won't work, if they're normally new and don't comment much, well, great.

As an aside - I question the wisdom of nerfing the ability for this forum to work like a forum (and instead, a bunch of answers). If you sacrifice the small sense of community you already have, who is going to use this site?

Answer 8

Can't we just downvote comments instead? Some suggestions are just really bad, even if the only comments identified are suggestions - which is a clear anitpattern.

What I think downvotes should do in comments is subtract points from the user's reputation score, that way people stop doing dumb things in comments that they wouldn't do in answers because they know it would cost them reputation. I suspect that this would also stop "Me too" and "I agree" comments.

Answer 9

Please let me know if this causes any issues!

Will do. We won't know until it's been tried, so no idea why people are arguing.

Answer 10

One thing that still makes absolutely no sense to me is the fact that it seems to be disallowed (especially on IPS) to use comments to call into question the correctness of an answer.

"That's what votes are for"

Yes, but one downvote is irrelevant in the greater scheme of things, especially when no-one actually understands why the vote is there and figure the answer is good enough to deserve a counter-upvote (at which point the downvote probably helps the answer more than it hurts it).

"Votes will reflect how correct an answer is"

No. Just because there are few people who know what's wrong with the answer doesn't mean the answer isn't wrong.

"Phrase it as a suggestion"

Yes, maybe you can phrase your "this answer is so blatantly wrong because ..." comment as a "suggestion", but that shouldn't be necessary, it's somewhat dishonest to pretend to give a suggestion when you're in fact just pointing out a fundamental problem and, if they don't end up editing their answer to address the comment (which is common in the case of a fundamental problem), a suggestion will be less useful to future visitors.

"Post your own answer"

Yes, maybe you can post an answer of your own, but:

• "that answer is so blatantly wrong because ..." is not, and will never be, an answer,
• it doesn't make much sense to combine that with an answer of your own, so people must vote on both at the same time (just like you're not supposed to post multiple answers in one post),
• nor does it make sense to have something that's directly relevant to one part of the page be put somewhere else entirely.

"But this won't work since we can't downvote comments"

Well, that seems like an easy enough problem to solve.

---

The correctness of answers is too important

The correctness of answers is such a fundamental part of a Q&A that I'd have figured we could all agree that we shouldn't just sweep under the rug anyone who has something to share about that.

Basic site functionality contradicts this

How does close vote explanations fit into this? In plenty of cases they aren't suggesting an improvement at all, and every other close vote comment is, at best, doing so indirectly, which seems inconsistent with how other comments are treated - mods have absolutely no problem deleting indirect suggested improvements on answers.

---

PS: I'm not saying "suggest improvements" is a bad name, nor do I have a better suggestion, this is more about the policy driving this change.