Background
Last week we promoted the Weighted Query Strategy in order to test variations on the weights to see what results in more relevant feeds.
This time around, we decided to test a few variations of giving comment count more weight. They hypothesis is that posts with more comment weight are more relevant on the basis of demonstrating community congregation. Relevance can speak to topical relevance, quality relevance, and general community relevance.
We made a couple notable feed-related adjustments this week outside of the test:
- Turn featured back on as query scoring weight: Gives admins back the capacity for a curation-oriented nudge.
- Change weighted query limit from 50 (default) to 25: Adjusts the amount of results, with 50 having been more than we needed.
Results
We tested two variations of more comment weight testing the hypothesis: "Slightly more" and "Much more".
| Scenario | Incumbent Conversion | "Slightly more" Conversion | "Much more" Conversion | Likely Winner | Probability of Winner |
|---|---|---|---|---|---|
| Creates a comment. | 4.3% | 4.48% | 4.02% | "Slightly More" | 55.66% |
| Creates comments on at least 4 different days within a week. | 0.3% | 0.22% | 0.33% | "Much More" | 65.11% |
| Views pages on at least 4 different days within a week. | 21.22% | 22.14% | 20.28 | "Slightly More" | 90.93% |
| Views pages on at least 4 different hours within a day. | 10.03% | 10.99% | 10.27 | "Slightly More" | 80.78% |
| Views pages on at least 9 different days within 2 weeks. | 9.47% | 10.38% | 8.26 | "Slightly More" | 97.66% |
| Views pages on at least 12 different hours within five days. | 2.02% | 2.42% | 1.75% | "Slightly More" | 94.03% |
Conjecture
"Slightly more" was our clear goldilocks winner here, with one of the new variations winning all the tests. We are still — and will be for a while — in the"low-hanging fruit" stage of this process.
We have gotten some anecdotal feedback that the feed is becoming more relevant, and need to pair that with continued examination of what is bringing people back and keeping them involved in the community.
Next steps
The next test has been proposed and is pending review. It adds a "final ordering" step to the query strategy in order to take the 25 most relevant results, as currently returned by the query, and presenting them in an order that is most useful and relevant to the end user.
Currently, among the 25 results we return, it is equally possible that the most useful one be the first result, or the 24th, as it is now.
The way the algorithm is constructed, we get the opportunity to add a "final ordering stage", which gets to operate on just the ones we have deemed relevant in the big picture — so this stage can purely focus on sifting through these results and ensuring that they are presented in an effective order.
I have a lot of confidence that this final step will be a big step up from the current results, and will be a step we can continue to hone over time. For now, since this is new, we will assign a variety of possibilities to this step for maximum learnings. I suspect that even the runner-up candidates could give us information to combine some of these in the future.
Top comments (0)