The results of the experiment
Days before the change 65
Days after the change 65
Posts created before the change 3655
Posts created after the change 3708
Comments created before the change 14029
Comments created after the change 12003
Description Before After PctChange
------------------------------- ---------- ---------- ----------
Comments / post ratio 3.84 3.24 -15.66
Flags / comment ratio 0.14 0.18 28.55
Pct comments deleted 34.75 34.37 -1.10
Comment upvotes / comment ratio 2.80 3.07 9.75
Pct comments followed by edits 8.87 10.40 17.16
Just as we saw on Interpersonal Skills, this resulted in significantly fewer comments.
Unlike the IPS test, we didn't see fewer flags per comment; part of this may be simply that an awful lot of changes to comment flagging rolled out mid-way through the test, but given the effect wasn't very strong on IPS either it's my guess that any changes noted here are most likely just noise.
As on IPS, there was a noticeable improvement in the average utility of the comments that were posted: even as fewer comments were posted over all, y'all left slightly more comments that were followed by edits to the post being commented on. This continues to be my favorite observation, as it suggests the change in wording is helping folks to focus on activity that results in lasting improvement to the site vs. idle chit-chat.
Distribution and ongoing effectiveness
Finally, I want to address a concern I've seen raised in a few places, namely that any benefits are likely to be a temporary effect caused more by confusion than anything else.
The metrics we've been looking at thus far were picked to reduce the effects of noise and temporary variation on the results. That's useful for gauging the overall effects of a test like this, but they don't tell the whole story! As we're all painfully aware, it isn't every post that gets large amounts of tangential commentary; the norm - here and just about everywhere else - is 0-2 comments per post. That makes it kinda hard to intuit either a problem or its solution from casual observation, as problems are by definition exceptional cases and thus observing improvement means observing cases where comments might have blown up but did not.
(click for larger version)
This chart seeks to illustrate the distribution of comments posted during the first week of each post's life here over the past year and two months, grouped by every week between June 26th 2017 and last week. Weeks where the test was live are highlighted in orange; whiskers indicate the full extent of the data (so, extreme outliers are plainly visible).
A few observations:
The overall range has consistently dropped during the test - not a single post has garnered more than 60 comments in its first week for the duration, nor 50 during the second half of the test.
The median number of comments per post has been locked at 2 for the duration of the test, with 75% of posts consistently getting 4 or fewer comments.
There's no variation in either the median or quartile numbers week-by-week during the test, which stands in stark contrast to the year preceding it and also suggests that whatever effect this change had was consistent beyond what we could expect from novelty or confusion alone.
As with IPS, this isn't a dramatic difference. There are still posts that get too many comments, there are still plenty of flags, there's still lots of cleanup work for moderators to perform.
But, there is a difference - and it's largely positive. And for a relatively inexpensive change, that's nice to see!