10

On the advice of Shog, this has been reposted on Meta, slightly simplified

So it's no shock that we have a lot of poor answers here. We are having issues with understanding the current flagging, and problems with old questions with lots of bad answers.

Kismet brought jmort, Shog9, and myself in to chat at the same time, where we had a discussion on the issue of poor answers (note: pretty long, but enlightening discussion).

Fundamental Problem

When someone asks a bad question we can put it [On Hold] to allow the poster to improve the post. If it doesn't get improved, then it will be deleted in a regular cleanup and our site is a better place for it.

There is no such option for answers.

Since graduation, we have lost a lot of people able to delete answers. That means the mods have to get involved. We are limited to voting (which doesn't have much impact when a question gets popular with lots of sympathy upvotes), or flagging (which many people seem to think we aren't using properly). This is less than ideal.

The community needs to have a way to deal with poor answers in the same way that it deals with poor questions, preferably without moderator intervention, and still giving people a chance to improve their answer prior to deletion.

Solution

We already have a great tool similar to putting a question on hold in post notices like this:

This post does not cite any references or sources. Please help improve this post by adding citations to reliable sources. Unsourced material may be challenged and removed.

Right now, these post notices can only be added by moderators, meaning that while any 3k user can vote to close a question, we are incredibly limited in our ability to do the same to answers.

Let's change that.

Proposed Workflow

  1. If a question receives answers over a certain threshold, those answers are tossed in a queue for review
  2. For users over a certain reputation threshold, one of the options for that queue can be to add a post notice
  3. If a certain amount of users vote to add a post notice, it gets automatically added to the post without moderator action required
  4. Once a post is given a post notice, if it isn't edited in a certain number of days (like the 5 for a question to go from [On Hold] to [Closed]), it is added to the Low Quality Review Queue and can be deleted regardless of score

Proposed Post Notices

I think the following post notices would be good for our site:

Insufficient Explanation

When someone just gives an answer without explaining why or how it is the appropriate answer, we can use this existing post notice:

We're looking for long answers that provide some explanation and context. Don't just give a one-line answer: please explain why you're recommending it as a solution. Answers that don't explain anything will be deleted. See Good Subjective, Bad Subjective for more information.

Does not Cite Sources

When someone posts something about statutory or regulatory requirements (for instance, they say something is not permitted under the Americans with Disability Act) but doesn't actually cite the relevant portion of the text to allow people to confirm the answer, we can use this existing post notice:

This post does not cite any references or sources. Please help improve this article by adding citations to reliable sources. Unsourced material may be challenged and removed.

Me Too Answers

Often we have answers that don't actually add any value. They contain the same advice, same reasoning, same explanation, and just end up taking up space rather than providing value. We don't have a post notice for this, but I think we should request one be added with a notice like this:

Information in this post has already been provided in other answers. Please help improve this answer by adding additional information not already given in other answers. Answers duplicating content may be removed.

  • 4
    This sounds like a great alternative to either our current use of flags or needing the mods to do everything. Let the community help, instead of just asking for mods to help. I get that post notices are supposed to be rare and there is risk of overuse, but what we're doing now isn't working so let's try this for a few months and then re-evaluate. – Monica Cellio Feb 27 '14 at 3:48
  • "answers over a certain threshold" -- amount of answers at half of CW threshold looks worth considering for that – gnat Feb 27 '14 at 7:48
  • @gnat, for reference, here is the rough cumulative % of questions by number of answers from this chart: 1, 11%;2,27%;3,45%;4,59%;5,69%;6,80%;7,85%;8,90%;9,92%;10,94%;15,99% So 8-answers (half of CW) means reviewing 10% of answers, and 5-answers would be 30%. – jmac Feb 27 '14 at 8:01
  • as a reviewer I certainly prefer 10% over 30; in my experience it takes quite a lot of effort to do thoroughly enough - no matter how much I would want to review 30 (50, 99%), I can realistically commit to only 5-10% (been there done that) – gnat Feb 27 '14 at 10:07
  • Regarding steps #3 and #4: I think this might be useful even if the rest of the system isn't. I touch on this briefly here: meta.stackexchange.com/questions/223462/… – Shog9 Feb 27 '14 at 22:14
  • @Shog, if you're going to implement #3, I think you need to implement #2 (since regular users can't actually add post notices right now). But otherwise agreed. The goal is to have poor posts: 1) be identified, 2) be given a chance to be improved by the poster/community, 3) be able to be removed if they remain poor and can't be improved – jmac Feb 27 '14 at 23:41
9

Two Stage Workflow

I like the workflow because it breaks things out into stages. It takes more than one person to make a call to add the post notice, then it takes more than one person to delete, with deletion only being allowed on positively scored posts if the answer had previously been reviewed and left unimproved.

The two stage approach gives answerers some time to get feedback through comments before the answers are ultimately removed. It also gives other community members a chance to edit in citations. I've put citations in other answers before to avoid deleting something that I personally thought had value.

This approach, if executed and implemented with care, could lessen the chance of abuse, while also keeping things from slipping through the cracks. There are currently 31 answers with post notices, and some are fairly old, having flown just below the radar. This plan would help ensure they're dealt with in a timely manner while still giving the community time to collaborate and resolve issues.

Are there different types of posts with different answering requirements?

The other important point that you've brought up is that there are different types of posts. On some, it makes sense to require citations and official references and for those posts to be removed if citations and references can't be produced. But there are other types of posts where only real experiences, backed up with explanations of why and how, can validate them, since there is no "official" solution to that problem.

The key is to be able to differentiate between the types of posts that need the citations and the types of posts that need the full how and why explanations. Perhaps we can break down certain tags into these two categories. Categorizing types of answers makes it more clear what guidance to give the folks answering as well as what guidance to give folks making decisions about the posts.


Proposed Moderator Workflow to Test the Proposed Review Changes:

Here's a proposed plan for the moderator workflow:

  • On newly posted answers, community members flag as "needs citation" or "needs explanation" post notice. If you have suggestions for improving the post, please leave a helpful comment.

  • Moderators evaluate and add the post notice and leave a comment, if necessary. (We pretty much do this anyway)

  • If the flag is helpful, we mark it as helpful and clear it off our plate.

  • Once per week, we drop the post notice answers search, sorted from oldest to newest into the Water Cooler or meta where community members review, attempt to improve, and "re-flag" the post as either "improved - remove post notice" or "very low quality" or "not an answer"see updates to flagging meaning it hasn't met the guidelines for a good Workplace answer and needs to be evaluated for removal. This last part is where we typically lose sight of things and where this proposal may prove to be helpful.

At this time, what I've written is just a proposed idea. Let's hear some more feedback on this before making these adjustments.


  • UPDATE: For the most recent flagging guidance, please see this meta post.
  • If we can do this manually for a few weeks, it will give great data on how well/poorly the general flow works, and give us better insight in how to implement it on a wider scale when we bring it to meta.so. The only question is whether or not it will significantly (and perhaps excessively) raise the moderation burden prior to new mods being elected. Speaking of which...any news on the timing of the election? – jmac Feb 27 '14 at 6:29
  • One thought on this workflow is that is is sort of what happens anyway. We see a flagged post that we think could be improved, we leave a comment, and mark it as helpful, and if it isn't improved, sometimes the post gets re-flagged and we then go back and clean it up. By using the post notice filter, we're able to "mark" these posts outside of the flag queue and more easily track them as a community, which means we can come back and review them later but without leaving a pile of flags in the queue and without just hoping someone will accidentally trip over the post again. – jmort253 Feb 27 '14 at 6:35
  • 1
    Can we add the 'me too' post notice? Or is there no current functionality for custom notices? As for dividing the difference between citations and experiences by tag, I think that is an entirely separate discussion that needs to be had. The issue is that our tags don't fit in as nicely as those related to programming or the like, so we have a serious categorization issue. If we want to be able to make that sort of distinction, we need to find a good way to categorize our questions, and then figure out how to educate users on them. – jmac Feb 27 '14 at 6:38
  • The post notices are canned notices. A good feature request would be to either add to them or edit them, similar to close reasons. – jmort253 Feb 27 '14 at 6:41
  • Feature-request submitted to meta.so here to allow the post notices to be customized for each community. – jmac Feb 27 '14 at 6:58
  • consider tweeting this question, it looks sufficiently important to make an exception (I know meta posts aren't usually tweeted). I've seen this done to some meta posts at Programmers - you can check with their mods or in Teachers Lounge for how this could be done – gnat Feb 27 '14 at 7:50
  • @gnat - twitter.com/jmort253/status/439047215068098560. The "share" links appear below meta posts too, so they can be shared publicly just like any other post. – jmort253 Feb 27 '14 at 14:40
  • I meant tweeting from Workplace.SE account, like this P.Se meta post was tweeted from Programmers - chat.stackexchange.com/transcript/message/13956936#13956936 (though any form of promotion is welcome actually) – gnat Feb 27 '14 at 14:44
  • That happens automatically, @gnat, and besides, the Twitter account has 83 followers, I have about 630. How many do you have? How many are on Stack Exchange? People are more interesting than automated bots. Plus there's more of us. The best way to spread the word about anything is with multiple, networked sharing. – jmort253 Feb 27 '14 at 15:07
  • I see, that makes sense (I for one already promoted this post at Whiteboard:) – gnat Feb 27 '14 at 15:13
  • 2
    I think there should be a threshold where 3 10K+ users can apply the notices, then the mods can come through review those notices and remove them if they were applied inappropriately. Otherwise I like it. – IDrinkandIKnowThings Feb 27 '14 at 16:49
  • 1
    @Chad - That's sort of the plan, but the idea is we'd use features we have available to us first to try out the idea and see if it has merit before pushing for official development time. – jmort253 Feb 27 '14 at 20:36
  • @Jmort253- OK that I can be on board with... I was thinking that was the end goal... – IDrinkandIKnowThings Feb 27 '14 at 20:41
  • This is a perfect example: workplace.stackexchange.com/a/20103/16 - The answer is actually answering the question asked but does not meet our standards since the question is basically soliciting opinion the answer provides what is asked for. The original question could potentially be answered objectively with with facts references and experiences. But this answer just provides the users opinion not even based on their experience just their ideology. – IDrinkandIKnowThings Mar 5 '14 at 16:54
  • 1
    @jmort, in my experience, SE tends to make things easily tweakable regarding numbers. So rather than worry about the nitty gritty details, I would rather outline the system itself and let them implement it based on their infinite experience, and then we can tweak it when we see it in action and figure out what feels right or not. In the meta.so request, I set the post notice privilege at the same level as close votes (as they are analogous), and being flags would be reviewed by 10k+ users. – jmac Mar 11 '14 at 3:37

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .