When I tried to submit my sitemap, after a quick look at the robots.txt file I realised that it was sitemap.xml.gz file. I also connected my Forem to s3, where the sitemap.xml.gz of my Forem was located.
So, I quickly learned few facts from Google documentation and realised that this is one of the standard method of indexing URLs and the Google search engine console supports it.
In order for this to work, I followed few guides, and the simplified procedure was this:
- Add you bucked URL within Search Console
- Make the
sitemap.xml.gzavailable for public - Update the robots.txt to point out the sitemap, Forem already did this.
- Paste the link in Console.
But for some reason, google was unable to fetch and index my website.
- click here to view my robots.txt
- click here to check the public access of my sitemap
- Also attaching a screenshot of my search engine console, which verifies that I attached my bucket and uploaded the correct URL.
Top comments (14)
I would like to wait for a response from the @forem team before updating it in the Known Issues of Self Hosting Forem
I made a change to sitemap generation last week to fix a memory leak, but it shouldn't have had any effect if you had your AWS credentials in place to upload it to S3. I'll give that another look this morning.
I'm attaching the link to my auto generated
robots.txtfile: app.leewardslope.com/robots.txtAs it is autogenerated you can confirm that I have successful enabled S3.
Also, I disabled my cloudinary api, so that you can also confirm it by inspecting any image within my Forem.
Rest of the information are clearly mentioned in the above article, be it realted to making the S3 bucket public, adding my bucket to search console and other requirements.
Hey folks, we at DEV were actually given the instruction from our SEO consultant to submit monthly sitemaps directly via the Search Console and in that context they need to be off the root.
So you may do that using this format:
app.leewardslope.com/sitemap-Jun-2...
There is some additional logic in
SitemapsControlleraround how this is generated and what the options are that currently have some "magic number syndrome" that applies too much to just DEV.TL;DR the robots.txt one is not necessarily best practice anyway, but the other functionality is not entirely fleshed out and is not appropriately documented. I'll follow up on these problems as an issue, but @ellativity and co. heads up on the state of this. We could immediately adjust some of the docs to address this.
So is there any criteria for a post to be eligible to stay in sitemap?; because I was not able to see all of my posts in the sitemap.
Anyways something is always better than nothing, but just curious to know about it; as sitemap submission is one of the essential step for SEO.
Good question. @ben do we need to do a manual sitemap submission per month?
I have the same issue that the sitemap wasnt updating, so i need to do it manually with
./rake sitemap:create
Offcourse the best solution is to have this automated, not sure if this already part of Forem and could be an issue with our setup.
I didn’t read Ben’s post on this properly. All the monthly sitemaps are automated (i think) and then you manually submit then like this:
So you should add sitemap-Feb-2021.xml and sitemap-Mar-2021.xml and so-forth to your search console.
Yup that's right.
Hi Akhil,
Last month i was able to add my sitemap.xml.gz to the search console, I think the only difference is that I added my site hackerss.com to the search console and I exposed my sitemap via nginx:
hackerss.com/sitemap.xml.gz
Hello @manuel
There is slight difference in the way we host our Forem. My Forem is based on dokku(Opensource Heroku), so it can be seen as a clone of DEV(at the level of skeleton)
According to my opinion, your setup might be based on Docker/Podman or Native Linux installation. If this is not the case, you should probably check your s3 integration.
I came to this conclusion based on few findings on your server.
Also, in my opinion, if your Forem is not based on dokku or Heroku; your setup might be somewhat similar to the way Forem.dev is up and running.
The issue I was pointing here is for those people who are hosting their database in remote locations like s3.(Because Heroku don't provide persistent storage, while we can configure dokku to have persistent storage, but I was not intended to use my VPS as a storage block)
Also, I would like to know about your setup configuration, it might help me understand other possible ways of hosting Forem.
I installed forem using Docker and just put a nginx to handle the static files and reverse proxy for forem.
I think the issue with your setup in the search console is that you have your domain as amazonws and not app.leewardslope.com As Ben already gave us an example with one sitemap for June.
No, I have both the domains configured from past 2 years. And Forem sutap is working for me according to Ben's reply.
@akhil thanks for bringing up this issue! I don't have personal experience of submitting sitemaps but I will flag this with some teammates and see if anyone else has any suggestions.