Forem Creators and Builders 🌱

Ben Halpern for Forem Core Team

Posted on

Feed test experiment 5 results: Tag point summing leads to mixed results.

Background

We assign weights to tags members follow based on what they explicitly set, and adjustments based on implicit activity. In the weighted algorithm, we had been taking into account whether people followed tags, but not the weights.

This experiment tested whether putting these points back in had a positive effect.

We allowed this test to run for two weeks because there was no runaway winner.

Results

Here were the possibilities:

  • incumbent
  • tag_follow_points_maximum_spectrum
  • tag_follow_points_medium_spectrum
  • tag_follow_points_minimum_spectrum
Scenario Incumbent ranking 2nd Place Likely Winner
Creates a comment. 2nd incumbent tag_follow_points_minimum_spectrum
Creates comments on at least 4 different days within a week. 3rd tag_follow_points_minimum_spectrum tag_follow_points_maximum_spectrum
Views pages on at least 4 different days within a week. 2nd incumbent tag_follow_points_maximum_spectrum
Views pages on at least 4 different hours within a day. 1st tag_follow_points_maximum_spectrum incumbent
Views pages on at least 9 different days within 2 weeks. 3rd tag_follow_points_minimum_spectrum tag_follow_points_maximum_spectrum
Views pages on at least 12 different hours within five days. 1st tag_follow_points_minimum_spectrum incumbent

The results were fairly mixed, and there is no clear winner, but tag_follow_points_maximum_spectrum won the most tests with three wins.

Conjecture

This could arguably be an opportunity to keep the incumbent, given that it is not abundantly clear that tag_follow_points_maximum_spectrum should be the winner. However, given that without tallying points, we are not really respecting the functionality we offer to users to have their points accounted for — tie should go to the test that allows for this. Among the challengers, "max spectrum" was the winner. This will also allow us to fiddle with the points that tags are allotted independently to adjust the score in a different test.

There is a lot of benefit to declaring tag_follow_points_maximum_spectrum the winner.

Next steps

We have a new test scheduled for review and some adjustments coming down the pike to allow more flexibility for how other Forems might adjust their own feed strategies.

Top comments (0)