Forem Creators and Builders

Akhil Naidu
Akhil Naidu

Posted on

Unable to submit sitemap.

When I tried to submit my sitemap, after a quick look at the robots.txt file I realised that it was sitemap.xml.gz file. I also connected my Forem to s3, where the sitemap.xml.gz of my Forem was located.

So, I quickly learned few facts from Google documentation and realised that this is one of the standard method of indexing URLs and the Google search engine console supports it.

In order for this to work, I followed few guides, and the simplified procedure was this:

  1. Add you bucked URL within Search Console
  2. Make the sitemap.xml.gz available for public
  3. Update the robots.txt to point out the sitemap, Forem already did this.
  4. Paste the link in Console.

But for some reason, google was unable to fetch and index my website.

  • click here to view my robots.txt
  • click here to check the public access of my sitemap
  • Also attaching a screenshot of my search engine console, which verifies that I attached my bucket and uploaded the correct URL.

Search Console Error

Discussion (14)

Collapse
akhil profile image
Akhil Naidu Author

I would like to wait for a response from the @forem team before updating it in the Known Issues of Self Hosting Forem

Collapse
jamie profile image
Jamie Gaskins

I made a change to sitemap generation last week to fix a memory leak, but it shouldn't have had any effect if you had your AWS credentials in place to upload it to S3. I'll give that another look this morning.

Collapse
akhil profile image
Akhil Naidu Author

I'm attaching the link to my auto generated robots.txt file: app.leewardslope.com/robots.txt

As it is autogenerated you can confirm that I have successful enabled S3.

Also, I disabled my cloudinary api, so that you can also confirm it by inspecting any image within my Forem.

Rest of the information are clearly mentioned in the above article, be it realted to making the S3 bucket public, adding my bucket to search console and other requirements.

Thread Thread
ben profile image
Ben Halpern

Hey folks, we at DEV were actually given the instruction from our SEO consultant to submit monthly sitemaps directly via the Search Console and in that context they need to be off the root.

So you may do that using this format:

app.leewardslope.com/sitemap-Jun-2...

There is some additional logic in SitemapsController around how this is generated and what the options are that currently have some "magic number syndrome" that applies too much to just DEV.

TL;DR the robots.txt one is not necessarily best practice anyway, but the other functionality is not entirely fleshed out and is not appropriately documented. I'll follow up on these problems as an issue, but @ellativity and co. heads up on the state of this. We could immediately adjust some of the docs to address this.

Thread Thread
akhil profile image
Akhil Naidu Author

So is there any criteria for a post to be eligible to stay in sitemap?; because I was not able to see all of my posts in the sitemap.

Anyways something is always better than nothing, but just curious to know about it; as sitemap submission is one of the essential step for SEO.

Thread Thread
lee profile image
Lee

Good question. @ben do we need to do a manual sitemap submission per month?

Thread Thread
manuel profile image
Manuel

I have the same issue that the sitemap wasnt updating, so i need to do it manually with

./rake sitemap:create

Offcourse the best solution is to have this automated, not sure if this already part of Forem and could be an issue with our setup.

Thread Thread
lee profile image
Lee

I didn’t read Ben’s post on this properly. All the monthly sitemaps are automated (i think) and then you manually submit then like this:

So you should add sitemap-Feb-2021.xml and sitemap-Mar-2021.xml and so-forth to your search console.

Thread Thread
akhil profile image
Akhil Naidu Author

Yup that's right.

Collapse
manuel profile image
Manuel

Hi Akhil,

Last month i was able to add my sitemap.xml.gz to the search console, I think the only difference is that I added my site hackerss.com to the search console and I exposed my sitemap via nginx:

hackerss.com/sitemap.xml.gz

hackerss sitemap

Collapse
akhil profile image
Akhil Naidu Author

Hello @manuel

There is slight difference in the way we host our Forem. My Forem is based on dokku(Opensource Heroku), so it can be seen as a clone of DEV(at the level of skeleton)

According to my opinion, your setup might be based on Docker/Podman or Native Linux installation. If this is not the case, you should probably check your s3 integration.

I came to this conclusion based on few findings on your server.

  1. You have a robots.txt file which is not pointing any s3 bucket
  2. Inspecting an image from you Forem, shows that the image is located within your VPS(Uploads Folder)

Also, in my opinion, if your Forem is not based on dokku or Heroku; your setup might be somewhat similar to the way Forem.dev is up and running.


The issue I was pointing here is for those people who are hosting their database in remote locations like s3.(Because Heroku don't provide persistent storage, while we can configure dokku to have persistent storage, but I was not intended to use my VPS as a storage block)


Also, I would like to know about your setup configuration, it might help me understand other possible ways of hosting Forem.

Collapse
manuel profile image
Manuel

I installed forem using Docker and just put a nginx to handle the static files and reverse proxy for forem.

I think the issue with your setup in the search console is that you have your domain as amazonws and not app.leewardslope.com As Ben already gave us an example with one sitemap for June.

Thread Thread
akhil profile image
Akhil Naidu Author

No, I have both the domains configured from past 2 years. And Forem sutap is working for me according to Ben's reply.

Collapse
ellativity profile image
Ella Ang (she/her/elle)

@akhil thanks for bringing up this issue! I don't have personal experience of submitting sitemaps but I will flag this with some teammates and see if anyone else has any suggestions.