6

Another question asks about opting out of the Hot Network Questions list for a few months to see if that's better or worse for us (or ends up not mattering). See the other (unresolved!) question for details on that.

If we decide to try the experiment, we'll need a basis for evaluating the results that's better than everybody's subjective gut feelings. What factors should we measure? Assume we'd collect before-and-after stats, seasonally adjusted or year-over-year.

The following methods of collecting information are available to us:

  • Public on-site information: close/reopen stats, advanced search, site anlytics, etc.

  • Stack Exchange Data Explorer (SEDE): SEDE doesn't have everything, but it has a lot. Think about SQL queries that would be useful.

  • On-site moderator tools: search of deleted posts, total flags (but no breakdowns), number of deleted comments (but not reasons), maybe other stuff. The flag and comment stats are crude; at any given time we can see totals for the last month, quarter, or year, but we can't get those stats for specific date ranges.

  • Things that require database access: in the past, CMs have sometimes been willing to query the live database directly for information not exposed through the UI. We might be able to get a breakdown of flag types in a date range, for example.

Some feel that having our questions on the HNQ list is harmful for us. What is "harmful" and how do we measure it? And what is beneficial about being on it and how do we measure that?

7
  • In the original proposal you wrote "Specifically, questions -- usually sensationalist or controversial questions, because that's what HNQ optimizes for -- get onto the list that's shown in the right sidebar all across the network, attracting vastly disproportionate voting and lots of comments. We also get more than a few provocative questions here that seem designed to try to get onto that list." Thus, if there are queries which could record a before and after of those, that could generate appropriate measures. Seems unlikely.
    – Joe Strazzere
    Commented Nov 27, 2018 at 12:35
  • 3
    Frankly, I prefer a "gut feel" approach. A Meta poll asking "is it better now?" would be my preference.
    – Joe Strazzere
    Commented Nov 27, 2018 at 12:36
  • this post at IPS meta looks relevant: On the HNQ and its effects on the site stats. Also I would support "gut feel" evaluation proposed by @JoeStrazzere - I think it is a must to have it as a part of evaluation (maybe even main part of it)
    – gnat
    Commented Nov 27, 2018 at 12:46
  • 5
    Is there a way to determine which questions would have ended up on HNQ if we wouldn't have opted out? That way we could look at statistics for HNQ-questions specifically.
    – David K
    Commented Nov 27, 2018 at 13:02
  • 1
    I don't think it can be emphasized enough that the most important part of your question is the last paragraph. It's impossible to discuss metrics without trying to understand what you're measuring. And, based on scanning the responses to the question you referenced, there are a lot of people with very different ideas about this. If we can't agree as a "community" as to what our goals are, I don't think we're going to be able to agree how to measure them.
    – dwizum
    Commented Nov 27, 2018 at 16:57
  • 1
    @DavidK we can't even reliably find out which questions are on the hot list -- there's no way to ask "has this question ever been hot?" or get a list of all such questions. So I don't know if we can figure out which ones would have been, but we can ask if there's some way to run the calculation but not add the questions to the list (and just tell us instead).
    – Monica Cellio Mod
    Commented Nov 27, 2018 at 17:12
  • Monica, wrt list of past hot questions, it is possibly to get it now thanks to community efforts in project HNQ Scraper. As for determining which questions would get to it the closest thing is this user script but it's not quite there because it only shows current hotness score and can't tell what it was in the past (that also seems to be doable technically but more complicated)
    – gnat
    Commented Nov 27, 2018 at 17:52

3 Answers 3

Reset to default
7

Here's a list of statistics that I think would speak to overall community health, not just to the HNQ problem. I don't know if we have the ability to get all of these, but I think the information would be very interesting.

  • Overall site traffic (page views?)
  • Number of unique visitors to the site
  • Total percentage of new questions closed
  • Percentage of new answers deleted
  • Percentage of new comments deleted
  • Total number of flagged posts
  • Number of new accounts created
  • Number of questions asked by new users (including % closed)
  • Number of answers from new users (including % deleted)
  • Number of comments from new users (including % deleted)
3
  • 3
    Could be also interesting to see how much of those new accounts created actually stay and become more active members, versus how much of those are only 101's looking for some "entertainment"... only wish I knew how that could be quantified.
    – DarkCygnus Mod
    Commented Nov 27, 2018 at 15:49
  • I would like to see stats for before and after
    – SaggingRufus
    Commented Nov 27, 2018 at 16:11
  • You can also look at the percentages of positive / negative / zero scoring questions and answers (although many of the negative-scoring ones presumably end up closed or deleted). And just total number of questions, answers and comments (not just for new users).
    – Bernhard Barker
    Commented Nov 28, 2018 at 19:26
5

What is "harmful" and how do we measure it? And what is beneficial about being on it and how do we measure that?

I like David's list of statistics that could help give a notion on the health of the site.

I would like to complement that by including a possible interpretation of such statistics, that could better help decide on the "harmful" nature of HNQ w.r.t. TWP. Feel free to comment so we can work out this interpretation in a way that makes sense to the Community:

  • Overall site traffic (page views?): this will indicate to what extend does HNQ brings attention to the site. Some attention is OK I believe, but much attention could indicate that most of it is because of "Entertainment".
  • Number of unique visitors to the site: this could indicate if we are drawing new visitors to the site (something good?), or well suggest that the visitors are recurrent and seeking a new episode of "Entertainment"
  • Total percentage of new questions closed: given we have a strong baseline of % of closed posts, this will help us realize to what extent HNQ worsens/helps the close situation. I would expect a drop after leaving HNQ, but if it's too steep then that could suggest HNQ is in part responsible.
  • Percentage of new answers deleted: this will shed some light on the nature of the answers that HNQ invites. A considerable drop in this statistic would mean HNQ increases the incidence of low-quality answers.
  • Percentage of new comments deleted: this I believe we are all familiar with. A significant drop here would mean that HNQ brings noise and chatty comments to the site (which I can already see happening).
  • Total number of flagged posts: similar to comments, this will indicate how much "nastiness" does HNQ brings in comments.

Now, the following statistics I would analyze in a bundle:

  • Number of new accounts created
  • Number of questions asked by new users (including % closed)
  • Number of answers from new users (including % deleted)
  • Number of comments from new users (including % deleted)

I believe that having new accounts is something good, and most likely HNQ increases the number of accounts created... however, it stops being good when those new accounts don't ask nor answer, and just stick to comments. It is even worse if we also happen to have high increase in flags and comments deleted.

So, if HNQ brings new users but they never contribute and just stick to comments and nastiness, that would mean HNQ is undesirable to some extent. If we don't get new users, but still have nastiness it would mean the problematic users are recurrent. After that, we can further analyze the results to draw more conclusions and decide.

2
  • 3
    Great descriptions, thanks! I would say that for the first two (traffic and unique visitors), we can expect that those numbers will go down when we get rid of HNQ, since the whole point of HNQ is to attract people who wouldn't otherwise be here. I think the question will be how far those numbers drop and whether they drop to a point that is harmful to the site.
    – David K
    Commented Nov 28, 2018 at 13:02
  • yup, as long as the changes are "significant" or "considerable" that would mean something. For that we will have to wait for the stats to make some analysis :)
    – DarkCygnus Mod
    Commented Nov 28, 2018 at 16:30
-5

Flip a coin, spin a bottle.............

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .