5

Posted on behalf of jmac.

The idea to test automating review of answers that fail to meet site-specific guidelines has been proposed by SE Community manager here:

let's test this first, see where it works and where it falls apart, and then implement the system that emerges. Pick a site you're active on and propose a campaign based on this process but using the existing tools: flag and ask for a notice to be added, flag again after n days and ask for the post to be removed. Track the results.

Can we please run proposed testing at Workplace?

As far as I understand, for this we would need to establish a list of predefined flag messages / templates and agree upon how many days to wait prior to re-flagging for removal if answer hasn't been improved.


Supplementary materials:

  • 3
    To be clear, you should be looking for two things here: constructive edits from the author or greater community in response to the notice, and justified deletions of noticed posts when edits are not forthcoming. If neither occurs with any frequency, back to the drawing board. – Shog9 Mar 20 '14 at 21:14
  • @Shog9 agree, that's a very important point, a great deal of flags I currently cast as follow up to mod notices are like "please take a look: mod notice at this answer appears to be obsoleted by an edit done in rev 2..." – gnat Mar 21 '14 at 6:03
4

As pointed out, this can't be done with the Data Explorer unfortunately. We will need some community manager help to get the data, but we can lay out what sort of data we want.

What are we trying to find out?

"Are post notices successful?" is the question we're trying to answer.

What is success?

This is where things get sticky, but I'd advocate the following measures of success for post notices:

  1. Post notices result in more successful edits of answers with them
  2. Post notices clearly indicate poor posts that should be deleted if not edited

Like with putting question 'on hold', our goal for putting post notices would be to give the answer a chance to be improved, and if not, indicate that it is a good candidate to be removed.

How can we measure that?

We should gather the following information:

  1. # of answers
  2. # of edited answers
  3. # of successful edits
  4. # of edited answers deleted
  5. # of non-edited answers deleted

This will allow us to see if we are improving the rate of edits, the quality of edits, and whether the edits are capable of saving the posts from deletion or not.

We need to compare to a baseline. I suggest comparing the following three categories:

  1. Answers with a post notice (test case)
  2. Answers with >=1 downvote, negative score, and a comment
  3. Answers with >=1 downvote, negative score

Summary

Get a table like this:

              # answers | # edited | # successful edits | # edited/del | # non-edited/del
Post Notice
DV + Comment
All Answers
  • 1
    "We will need some community manager help to get the data" -- interesting that since this was posted, "we" (you:) don't need this anymore, as you became community manager yourself. Consider editing the answer to account for this, otherwise it might look confusing for readers – gnat Sep 26 '14 at 15:35
  • 1
    Multiple sites are interest in feedback from this proposal. Any updates? – rolfl Oct 1 '14 at 2:20
7

Data we need to collect:

  • What happens to answers that get this post notice? Possible values: edited and notice removed, edited but insufficient (notice not removed), notice removed sans edit (successfully contested), answer deleted by owner, answer deleted by mod/community after the waiting period, answer deleted by mod/com early. Did I miss any? Or do we only care about "fixed" and "gone", without the detailed breakdown?

  • Of the flags requesting this notice (how do we identify those?), how many were accepted (notice added) and how many dismissed? In other words, how good is the community at identifying problem answers in the first place?

  • Maybe: how many answers are deleted (by non-owners) without going through this process. These are the ones that are considered to be unredeemable.

Things we need to do to prepare:

  • Decide how we'll collect the data. If we're going to use the Data Explorer, we need to make sure everything we'll need is queryable there. Or we could decide to do it manually (which means, in practice, mods doing it). This depends in part on how many of these we expect to have, which depends on:

  • Decide how long to run this experiment.

  • Build the queries we'll use to collect the data.

  • Decide on the community workflow. By what means (what type of flag?) do community members nominate a post for a post notice, within the current tool set? (Need to find out, if using SEDE: is the text of custom flags available in the data explorer, or only that they were mod flags?)

  • Decide on the moderator workflow. Moderators need to review the requests, add notices, and perhaps do something to set a reminder (which could be low-tech, like a list on meta, or developing the right query). When the waiting period is up, moderators need to either remove the notices (if the post was improved) or delete the posts (otherwise).

  • Decide how we decide. Once a post has a notice and gets an edit, how does the community decide if it's good enough to remove the notice? Moderators shouldn't have to read minds or decide on their own. During the experiment, should we just plan to use a chat room for reaching informal consensus, or what? (We don't have the equivalent of "reopen" votes for answers, so there's no review-queue support here.)

  • Decide what the waiting period is. How long do we allow for edits (or challenges) before acting on answers with notices?

Did I miss anything?

  • deleted posts are not in Data Explorer (and unlikely will be there, at least in sufficient detail), we may need help from dev team to run queries straight from database, or study stuff manually somehow – gnat Mar 22 '14 at 7:36

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .