Testing proposal: automating review of answers that fail to meet site-specific guidelines

Question

^{Posted on behalf of jmac.}

The idea to test automating review of answers that fail to meet site-specific guidelines has been proposed by SE Community manager here:

let's test this first, see where it works and where it falls apart, and then implement the system that emerges. Pick a site you're active on and propose a campaign based on this process but using the existing tools: flag and ask for a notice to be added, flag again after n days and ask for the post to be removed. Track the results.

Can we please run proposed testing at Workplace?

As far as I understand, for this we would need to establish a list of predefined flag messages / templates and agree upon how many days to wait prior to re-flagging for removal if answer hasn't been improved.

---

Supplementary materials:

• Improving the System for Dealing with Poor Answers
  • MSO follow-up - Improving the System to Deal with Bad Answers
  • Water Cooler follow-up - post notices experiment: what and how to test
• FAQ proposal: Back It Up and Don't Repeat Others
• Community moderation comments template
• Is it worth creating a meta.SO proposal for custom flags?

Answer 1

As pointed out, this can't be done with the Data Explorer unfortunately. We will need some community manager help to get the data, but we can lay out what sort of data we want.

What are we trying to find out?

"Are post notices successful?" is the question we're trying to answer.

What is success?

This is where things get sticky, but I'd advocate the following measures of success for post notices:

1. Post notices result in more successful edits of answers with them
2. Post notices clearly indicate poor posts that should be deleted if not edited

Like with putting question 'on hold', our goal for putting post notices would be to give the answer a chance to be improved, and if not, indicate that it is a good candidate to be removed.

How can we measure that?

We should gather the following information:

1. # of answers
2. # of edited answers
3. # of successful edits
4. # of edited answers deleted
5. # of non-edited answers deleted

This will allow us to see if we are improving the rate of edits, the quality of edits, and whether the edits are capable of saving the posts from deletion or not.

We need to compare to a baseline. I suggest comparing the following three categories:

1. Answers with a post notice (test case)
2. Answers with >=1 downvote, negative score, and a comment
3. Answers with >=1 downvote, negative score

Summary

Get a table like this:

              # answers | # edited | # successful edits | # edited/del | # non-edited/del
Post Notice
DV + Comment
All Answers

Answer 2

Data we need to collect:

• What happens to answers that get this post notice? Possible values: edited and notice removed, edited but insufficient (notice not removed), notice removed sans edit (successfully contested), answer deleted by owner, answer deleted by mod/community after the waiting period, answer deleted by mod/com early. Did I miss any? Or do we only care about "fixed" and "gone", without the detailed breakdown?

• Of the flags requesting this notice (how do we identify those?), how many were accepted (notice added) and how many dismissed? In other words, how good is the community at identifying problem answers in the first place?

• Maybe: how many answers are deleted (by non-owners) without going through this process. These are the ones that are considered to be unredeemable.

Things we need to do to prepare:

• Decide how we'll collect the data. If we're going to use the Data Explorer, we need to make sure everything we'll need is queryable there. Or we could decide to do it manually (which means, in practice, mods doing it). This depends in part on how many of these we expect to have, which depends on:

• Decide how long to run this experiment.

• Build the queries we'll use to collect the data.

• Decide on the community workflow. By what means (what type of flag?) do community members nominate a post for a post notice, within the current tool set? (Need to find out, if using SEDE: is the text of custom flags available in the data explorer, or only that they were mod flags?)

• Decide on the moderator workflow. Moderators need to review the requests, add notices, and perhaps do something to set a reminder (which could be low-tech, like a list on meta, or developing the right query). When the waiting period is up, moderators need to either remove the notices (if the post was improved) or delete the posts (otherwise).

• Decide how we decide. Once a post has a notice and gets an edit, how does the community decide if it's good enough to remove the notice? Moderators shouldn't have to read minds or decide on their own. During the experiment, should we just plan to use a chat room for reaching informal consensus, or what? (We don't have the equivalent of "reopen" votes for answers, so there's no review-queue support here.)

• Decide what the waiting period is. How long do we allow for edits (or challenges) before acting on answers with notices?

Did I miss anything?