Our Comments Problem

Question

our name is The Workplace, and we have a comment problem.

1. We definitely have a comment problem
2. We have tried to fix it, but haven't succeeded yet
3. We want your help to figure out a way to do that

We have a growing comment problem:

This is our comments by week since the Workplace started out:

As you can see, from January this year we have seen a marked increase in the number of comments we are getting. Bear in mind this does not include deleted comments, of which there are many hundred per week that are being deleted.

For a more specific example, at the time of this edit, this question is 15 hours old and already has 59 comments. Currently none are deleted. Take a look and tell me that all these comments are appropriate and useful to high-quality Q&A.

What we've tried

We have tried explaining what comments are for:

Comments are not intended for long-term storage of important information. But that transiency doesn't mean you can use comments for random, parenthetical asides. If your comment isn't likely to change the content of the post, please do not post it for someone else to clean up. Thanks.

We have tried recommending chat instead:

Chat is an underused tool on The Workplace, yet we have users in our community with over 300 comments posted in just a 30 day period. The evidence speaks for itself; people like to chat. Yet, comments on the Q&A site leads to a lot of unnecessary cleanup, not just for moderators but for all of our diligent community flaggers who work tirelessly to help keep the clutter under control.

But yet we still get a whole lot of comments, and plenty of meta posts trying to ask the community what to do about them:

• Best ways to address comments?
• Is it appropriate to leave a comment which is a general remark on the question?
• Are '+1 for [something about the post]' comments discouraged?
• Get a Room, a Chat Room!
• Why did all comments suddenly start disappearing?

These are just the posts in comments since April!

What can be done?

I would like you guys to let us know what can be done to prevent comments from drowning out the core information on the site: questions and answers. As explained in the help center for every SE site:

You should submit a comment if you want to:

• Request clarification from the author;
• Leave constructive criticism that guides the author in improving the post;
• Add relevant but minor or transient information to a post (e.g. a link to a related question, or an alert to the author that the question has been updated).

We definitely are getting far more comments that isn't in one of those categories than in it, and that is creating noise for people seeking our high-quality answers, and adding additional burdens to the community on flagging and cleanup.

How can we solve this problem short of turning off commenting?

Answer 1

I second what Joe Strazzere said. Or reiterate what I said on the other post. Whichever.

You don't have a comments problem; you have a UX problem.

If you have to explain to people "what [Feature X] is for", then [Feature X] is broken. If [Feature X] is intended to be used in a way that is orthogonal to the way people intuitively use it, then [Feature X] is broken.

Software interfaces need to be intuitive. They need to clearly convey to the user what's expected of them (graphically, without using a wall of text), and they should conform to the expectations created. Having explicit instructions/a manual is nice, but it's not reasonable/practical to expect that end-users will read and/or follow it. The acronym 'RTFM' exists for a reason, and that reason is not because people are good at reading or following instructions.

So the comments problem (if there is one; as Jon Ericson very correctly points out the metric you want to look at is comments per post/answer, not the absolute number of comments made, and by that metric the commenting rate is holding flat) is merely a symptom of an underlying UX problem. And instead of attacking the symptom (which on a personal level I find to be very inappropriate, as I have seen a number of useful, helpful, and insightful comments get deleted under the guise of "improving the QA site"; and this is also not something I've seen happen on other SE sites), you should go after its root cause.

As I see it, the root cause begins and propagates roughly like:

1. You have a feature in the interface that calls itself 'add comment'.
2. The concept of a 'comment' is well defined, both linguistically and within the web/blog/forum/general Internet context.
3. People use the 'add comment' feature to make comments, according to their intuitive understanding of what a 'comment' is.
4. While the comments made generally abide by the same content guidelines as questions/answers (in terms of being respectful, on topic, and not vulgar/abusive), they don't fit the modified definition of a comment that has been adopted here.
5. Manual action is taken to weed out all of those comments that aren't comments.
6. Frustration ensues, on all sides.

Item #1 is the root of the issue. As long as the UI says 'add comment', #2 and #3 are going to follow as a consequence of human nature, how people interact with software, and the fact that you're trying to use 'comments' in a way that's different from how the rest of the Internet uses comments. And as long as #2 and #3 happen, #4, #5, and #6 will follow.

There are two things I can see that will break the cycle:

1. Stop saying 'comment' if you don't really mean 'comment'. Change that 'add comment' to 'suggest an improvement' or 'message the author' or something else that's more in line with what you actually want people to do.

   And then next to it perhaps put a 'discuss this question/answer' (or even a 'comment') link that opens chat (preferably a lightweight version of chat, that lets people leave their initial message in a way that's similar to how 'add comment' currently works).

2. Abandon the modified definition of 'comment'. Just let people make comments, and only hit the 'Delete' button when something truly abusive, obscene, nonsensical, or off-topic gets posted.

   I'd argue that this would be in-line with how the current Stack Exchange UI 'expects' to be used. Long strings of comments auto-collapse, so that only the highest rated ones are displayed and an extra click is needed to view the complete thread. So the feature is self-limiting, in that once a certain threshold is reached adding more comments to a post does not actually add any more clutter into the interface. Perhaps that threshold could be dialed down to just 3 or 4 comments.

   Granted not all comments will be 'useful', but a good number of them will. I often find very useful information in comments on SO, particularly in terms of updates being provided against an older answer that has been superseded or modified by new information (you might say that the author should just update the original answer, but that doesn't always happen; particularly on older answers).

   I'm grateful for the comments on SO. They've helped me many times. I don't see why comments on The Workplace need to be handled any differently, and I find it concerning that on this site I've seen many helpful comments just disappear into nothing. I don't believe such deletions improve the quality of the QA site, and I think it's doing a disservice to the people who took the time to post those comments.

Bottom line: As long as your interface says 'add comment', that's what people will do. If you don't want that, fix the interface.

Answer 2

Does The Workplace have a particular problem with comment volume?

I looked at the ratio of comments to posts (including deleted of both) and tallied up the sites on the network that average more than 2.5 comments per post:

C/P  Site
---  ----
4.08 Skeptics  
3.8  Politics  
3.06 Jewish Life and Learning
3.06 History   
2.94 Mathematica
2.87 Puzzling Stack Exchange
2.81 Linguistics
2.78 Christianity
2.75 The Workplace
2.72 Theoretical Computer Science
2.63 Philosophy
2.57 MathOverflow
2.52 Code Golf 

The list included quite a few meta sites that I removed (notably The Workplace Meta at 2.86) because most sites have more commenty meta than main sites. This isn't surprising because meta tends to be more discussion-oriented than factually-oriented. The bottom of the list is largely populated with sites that died of lack of activity or are dying and have fewer than 1 comment per post.

Considering the place workplace issues have on the scale of subjectivity, the volume of comments per post here doesn't seem too extreme. (The math-related sites are somewhat of an anomaly. They rely heavily on comments for collaboration.) It's natural that more comments will be needed to clarify questions and answers when the subject has no independent method of verifying the truth of assertions.

Is the problem growing?

Well, it depends on how you look at it. I've forked your query and added a couple more lines:

The green C_per_P_100 line is comments per post multiplied by 100 so that it will fit on the sameish scale. Comments per post has held steady while number of questions and answers has increased smartly since the beginning of the year. It's not so much that people are getting more commenty as that there are more things to comment on.

Does this mean there's no comment problem?

Not at all. There are a few things I haven't looked at that might be a problem:

1. Comment distribution

   If comments were spread evenly, 2.75 comments on each and every post seems not overwhelming. But if a few questions gather many comments (like How do you decide when to go home for the day?) the problem can be quite noticable. For moderators especially, the automated "too many comments" flag ensures that such posts are noticed.

2. Comment length

   A few short comments are less problematic (at least in terms of space used) compared to the same number of long comments. We are working on making single-line comments take just one line, which could further reduce the space consumed by short comments. (On the other hand, long comments tend to be higher quality. See the next point.)

3. Comment quality

   I tend to think that comments on Skeptics, Mi Yodeya, Christianity, History, and Philosophy (the sites on the 2.5+ comments per post list that I'm familiar with) are fairly constructive on the whole. There aren't many +1 comments as I recall. If comments are mostly of the chatty sort on The Workplace and not helpful in clarifying the post they are attached to, there could very well be a growing problem that's not reflected in any statistic. Given my reading of the linked meta posts, it sounds as if the quality of comments, rather than their volume per se, is the problem here.

Summary

In order to solve a problem with comments, we need to clearly identify what, precisely, the problem is. My look at the data suggests that The Workplace includes more prolific commentary than most sites, but not excessively more. On the other hand, if comments distract from rather than augment the prime mission of a Q&A site, we may need to explore aggressive corrections.

Answer 3

I completely agree with @aroth.

The label "add comment" causes people to react the same way they do at every other site - they leave a comment. If this were happening only rarely, then it would be reasonable to blame the writer. But since this happens continuously, you must blame the interface.

If you really want to limit this to leaving clarifying questions or suggestions for improving the Question (or Answer) - choose a more descriptive label. Don't use the word "comment" since that word clearly has implications that don't match the field's intent.

Otherwise, it comes across as over-moderation, and sometimes as arbitrary moderation.

This isn't a "naughty user" issue - it's a "bad UI" issue. It can be fixed, rather than trying to be taught.

Answer 4

Since users appear to want to be chatty, make it easier for them to chat.

Right now, users are forced to leave the Question/Answer UI and figure out how the chat system works. Too much work; not worth the effort for many.

Instead, consider adding a link within the main UI, labeled something like "Discuss this Question/Answer in more detail" which leads the user directly to a Chat Room.

If you make the access to chat as easy as clicking the "add comment" link, you'll have more chatters.

Answer 5

Since the problem isn't the average number of comments but that there are some big, fast-growing clumps of them, would it be possible to raise the bar for entry on those posts? It's far better to prevent chatty/non-constructive/argumentative comments from getting there in the first place than to clean them up later, if we can manage it.

How might we do that? Here are some possibilities:

1. Lower the number of comments needed before you get the "let's continue this in chat" suggestion. I don't know what that threshold is right now, but it seems to allow two people to go back and forth three or four times right now. And when that happens it's usually an argument, and arguments have high likelihood of not being useful.

2. Have something like protection that restricts commenting. This would allow moderators (or the community, preferably) to prevent drive-by comments on posts that are getting a lot of activity. Like protection, it should require a certain amount of reputation on this site to be able to comment.

3. When comment velocity reaches a certain threshold (N comments/hour), either temporarily prevent further comments (for some number of hours) or queue comments for review. (That last probably won't happen as it's major new work, but comment review could help with other situations too, like new users' first comments.)

An aside: Requesting clarification to a post is one of the primary purposes of comments. So long as the issue remains we should keep those comments. So it needs to remain possible to leave them; turning off all commenting would be too extreme. It's most of the other uses of comments that cause problems, and that my suggestions above are intended to counter.

Answer 6

gnat asked:

could you get similar data for ratio of comments to posts but limited only to posts having comments? I mean, like, if there are for example 99 posts without comments at all and one with 100 comments, saying "average 1 comment per post" wouldn't be very informative, I'd prefer a more detailed breakdown, like, using same example, "1% posts has comments and posts having comments get 100 comments average", something like that

I ran a query to get the number of threads with certain bins of comments. I realized as I was doing the analysis, that part of the reason that Jon's analysis seems so reasonable is that it is looking at per post stats whereas as mods we look at per thread stats. After all, if you get a flag on a comment on an answer, you end up scrolling up to the question, and seeing all the comments on the way. That means that the feeling of being overwhelmed by comments is not limited to a single post, but rather a set of posts all attached to the same question.

I ran this for all (non-meta) sites on the network, and to make the data a bit more accessible, I took the biggest commenting sites from Jon's post, and SO as a comparison, and charted the "# of comments curve", or cumulatively what % of posts have under that amount of comments:

So basically, the further left the curve starts, the larger % of post threads have fewer comments. So in Stack Overflow's case (the blue line) about 95% of posts have under 10 comments in the Q&A thread. This may be a bit difficult to grok, and that's okay, it's a ton of data and a poor attempt to make it simple, so what I invite you to look at is the far right of the chart, where The Workplace is dramatically lower than any other site on the list.

2.1% of all TWP post threads get over 50 comments -- that is absolutely massive, and a significant distraction from the Q&A. Since the dawn of time, that is 112 different threads with over 50 comments, and of those, 54 (48.2%) have happened since we graduated. Here are the 9 threads since graduation that have gotten over 100 comments:

• 2014-03 - Project Manager asks for complete 100% confidence everytime committing code (108)
• 2014-03 - When should I include information in my resume to indicate I have a high IQ? (101)
• 2014-04 - I received a written warning for my performance, how can I save my job? (113)
• 2014-04 - How can I communicate better with a co-worker who is not a good listener? (103)
• 2014-04 - Do I have to relinquish my PC password to my former boss? (131)
• 2014-05 - How should I deal with bullying while looking for a new job? (103)
• 2014-06 - I'm not being hired, presumably because I have Asperger's. Is there anything I can do about it? (144)
• 2014-06 - Administrator thinks I, an IT volunteer student, am hacking the school network (102)
• 2014-07 - Is there a dress code for women in software industry? (191)

Let's take that last one. There were 15 posts totalling roughly 29,000 characters (at 6 characters per word, that's roughly 5,000 words). There were additionally 191 comments totalling roughly 51,000 characters. That's roughly 8,500 words using the same math. The reading volume of the comment exceeded the reading volume of all the posts combined. When comments balloon, they create a significant additional burden to the reader to slog through for people who are coming in from a search engine looking for a straight answer.

At an average reading speed of 250 words per minute, it would take 20 minutes to get through all the posts. To dig through the comments would be an additional 34 minutes. That is a significant additional burden to ask our visitors. That's why comments are intended to be temporary, so that they don't distract. But 191 comments means someone has to go through and clean up 191 comments -- that someone is the mod team. And at the very least, the mod team has to read through those 34 minutes of comments to decide what to keep, and usually several times over several days as the flags roll in.

We are definitely getting a lot more comments grouped up than we used to, and we are definitely pretty unique among the network in the number of extremely-high-volume comment threads we get (even when compared to places with higher numbers of comments per post).

Answer 7

Short answer: Yes, comments are continuing to climb.

I've grouped these by month, but you can see a steady trend since December last year that comments are becoming more frequent with the only drop in comment count being last month (and it's minor).

So yes, after a bit of a ragged decline from March 2014 to December 2014, comments are still an ongoing problem and we need to try to keep a lid on them.

Let's do some other quick analyses. Firstly, let's look at the total number of questions and answers (# answers is on the secondary axis):

There is a corresponding increase in the number of questions being asked each month to go with the increase in comments. However, let's trend comments against comments per question (# comments per question is on the secondary axis):

As you can see, there has been very general trend towards the number of comments per question increasing, albeit erratically.

Last one! Let's see how we're going with number of answers per question (on secondary axis):

This figure, while a little erratic, is not increasing. So we're getting a fairly flat number of answers per question. Note that we protect questions, so this figure can be capped from "fly ins" who offer an answer.

Summary

So while there has been an increase in the number of questions being asked, there is also a trend towards the number of comments per question increasing. It's this figure we need to try to manage :)

Answer 8

Let me see if I understand all this. This is a community driven site, correct? The community finds some questions/answers uninteresting and posts few comments. The moderator looks down upon this and declares it is good. The community finds some questions and answers interesting and helpful, and posts multiple comments. The moderator looks down upon this and declares it bad. Perhaps your problem could be solved by prohibiting interesting questions and helpful answers.

Answer 9

In you other post, you made my point completely. Perhaps I was too subtle. If this is a community driven forum, it should be driven in the direction indicated by the community, not by your personal wants and opinions. If it's a moderator driven forum, that's not necessarily bad. Just give up the "community driven" claim. If the community wants to make 50 comments on a post, that means that there's great interest in the subject, and lots of people have something, if not a complete answer to contribute.

I spent 20 years as Professor of Computer Science, hopefully enlightening such as yourself. Very early I learned that the most efficient means of education was not lecture, but group interaction. I also learned patience. I learned that everyone has something to contribute, and when I was facilitating discussion groups, it was my role to seek out those that were a little reticent and enable them to contribute. Aren't the comments on this forum much the same thing?

If your purpose was, as you claim, education, you'd value comments, rather than trying to find a way to discourage their contribution.

I'd guess the root of the problem is perception. Most people understand the word "comment" to mean something other than what you'd have it mean. You could perhaps satisfy your need for control by weighting comments. You could post the one's that meet your criteria at the top, and relegate the ones of which you don't approve, or perhaps don't understand to the bottom. That way, people could stop reading when the value fell their level of interest.

By the way, messing with my profile was really, mature.