There are a few checks in place that are factoring in making this decision - the change described here removed one favoring code tags as DEV specific, and unified the two magic numbers which had been 3 and 5 for the sitemap and robots meta, as well as making it tunable in admin.
If an article has a negative score (more spam reactions than votes) it will not be indexed. If an article is a draft (not published) it's not indexable. If an article was published before July 13th, 2017 (unlikely for you, this shows as the featured_number in the code and seems DEV specific) it is not indexable.
The last condition requires all of the following to be true and is an anti-spam filter:
score less than the minimum (this had been a constant 5 before, the default now is 0), the idea being low quality posts would be excluded
no comments from this user ("active" community members participating in discussions didn't fit the spam pattern)
post is not featured (any featured post should be indexable and not include that robots meta)
When a new article is created - the score is lower than the previous "magic" number 5, but should be higher than the current default 0 (if you haven't set this in admin).
I checked the two top articles for Jay-Z on 1vibe (1vibe.com/ildi/the-blueprint-chang... and 1vibe.com/1vibeteam/why-jay-z-s-co...) and didn't see the robots meta there now - the more recent one was published Sept 11th (before the changes were made) so it's possible at the time it was marked noindex and no longer is.
I am wondering what this value means exactly and how posts get assigned that number?
The article score is the sum of user reactions (hearts and unicorns from the left sidebar), with each counting 1 point (and an automatic like reaction created when you author the post). The author's reactions' scores (reactions on the user, rather than the article) are also factored in - this is normally used by moderators to downvote an abusive or spammy user. The actual code setting the score is in github but effectively the scoring is "more hearts means a higher score".
The old way could have marked an article no-index if the article had too few heart reactions, was not featured, and contained no code tag. The new way should remove that minimum popularity threshold at the default value of zero, which is reached for all published articles automatically.
Thank you for breaking this down for me. It’s very interesting to understand the initial logic for when the “noindex” rule only applied to DEV vs all the Forem communities today.
The Jay-Z deepfake article is indeed now indexable by Google. Based on your explanation, the cause of the “noindex” was most likely due to the author of that article having had 0 comments on 1VIBE when posting the article, even though the article was published on behalf of an organization, the 1VIBE Team in this case.
It makes sense that these rules were put into place considering the size of DEV and how many posts users must be making on the site every 24hrs. I’m glad that these settings can now be customized, they will be useful for SEO and dealing with spam.
@lee
yes, that issue with articles published but not search indexed had been observed and reported in a few cases (including Ildi's case above), the decision to include or exclude those meta tags is done "live" when you view the page, or when you generate the sitemap, so sites that have updated in the past week should already be seeing the change and more articles get picked up by search engines, If Forem hosts your site, these updates happen about twice per week, if you're self-hosted then you'd want to update to a current container image to get this change.
For further actions, you may consider blocking this person and/or reporting abuse
Hi Ildi,
There are a few checks in place that are factoring in making this decision - the change described here removed one favoring
codetags as DEV specific, and unified the two magic numbers which had been 3 and 5 for the sitemap and robots meta, as well as making it tunable in admin.If an article has a negative score (more spam reactions than votes) it will not be indexed. If an article is a draft (not published) it's not indexable. If an article was published before July 13th, 2017 (unlikely for you, this shows as the
featured_numberin the code and seems DEV specific) it is not indexable.The last condition requires all of the following to be true and is an anti-spam filter:
When a new article is created - the score is lower than the previous "magic" number 5, but should be higher than the current default 0 (if you haven't set this in admin).
I checked the two top articles for Jay-Z on 1vibe (1vibe.com/ildi/the-blueprint-chang... and 1vibe.com/1vibeteam/why-jay-z-s-co...) and didn't see the robots meta there now - the more recent one was published Sept 11th (before the changes were made) so it's possible at the time it was marked noindex and no longer is.
The article score is the sum of user reactions (hearts and unicorns from the left sidebar), with each counting 1 point (and an automatic like reaction created when you author the post). The author's reactions' scores (reactions on the user, rather than the article) are also factored in - this is normally used by moderators to downvote an abusive or spammy user. The actual code setting the score is in github but effectively the scoring is "more hearts means a higher score".
The old way could have marked an article no-index if the article had too few heart reactions, was not featured, and contained no code tag. The new way should remove that minimum popularity threshold at the default value of zero, which is reached for all published articles automatically.
Hey Daniel,
Thank you for breaking this down for me. It’s very interesting to understand the initial logic for when the “noindex” rule only applied to DEV vs all the Forem communities today.
The Jay-Z deepfake article is indeed now indexable by Google. Based on your explanation, the cause of the “noindex” was most likely due to the author of that article having had 0 comments on 1VIBE when posting the article, even though the article was published on behalf of an organization, the 1VIBE Team in this case.
It makes sense that these rules were put into place considering the size of DEV and how many posts users must be making on the site every 24hrs. I’m glad that these settings can now be customized, they will be useful for SEO and dealing with spam.
Great explanation, thanks for this. Does this mean that articles on new Forem pre the update would have noindex applied if the score was low?
@lee yes, that issue with articles published but not search indexed had been observed and reported in a few cases (including Ildi's case above), the decision to include or exclude those meta tags is done "live" when you view the page, or when you generate the sitemap, so sites that have updated in the past week should already be seeing the change and more articles get picked up by search engines, If Forem hosts your site, these updates happen about twice per week, if you're self-hosted then you'd want to update to a current container image to get this change.