Hello everyone,
I'm trying to up a self-hosted Forem community on Studyhab.com on DigitalOcean.
I have some problems with Forem for that project:
StudyHab contains over 5 million articles, 10 million comments, and 2 million users. Its Postgresql database is a database service with 8 GB RAM, 4 vCPU, 115 GB Disk configuration on DigitalOcean, too. As you can see here, Forem is too slow with this database, and I couldn't find a solution to improve Forem's performance.
Although I have tried to use Fastly with its API service and added the API Key and Service ID in rails.env file, Forem didn't work on Fastly.
What can I do about these problems? Do you have a suggestion for dealing with this size of a database?
Latest comments (9)
We are very much here to support this Forem and help it scale! I'm sure @jamie will help you work through the current issues, and we'll be a team going forward to make sure it keeps working.
I'm curious about the project itself: Can you tell me more about Studyhab, what its mission is, where the users come from, etc. Very exciting to support something substantial like this.
Of course, @ben ,
I'll send an email to you the mission and outline of the project in detail.
Thanks a lot, @jamie . I'm trying to analyze the indexes on Postgresql and to scale my servers effectively for today.
Thank you! ben@forem.com if you weren't aware :)
I thought to use bendhalpern@gmail.com, which is located on your profile :) However, I'll contact you at ben@forem.com as you said :)
@ce7in Can you go into more detail about how it didn't work on Fastly? What error messages was it showing, if any? That sort of thing.
Also, this is an impressive quantity of data on a Forem instance — multiple times more rows than DEV in all 3 of those tables. I'm curious how you're fitting that much data on a 115GB disk, though, because even though we have less data, we store a lot more than 115GB.
Fitting the data on a 115 GB disk wasn't a problem, but publishing them fast enough is a big challenge. I don't know how to deal with this problem because I have not needed to do that before. I have been trying to find a solution for this performance issue for a few days, but I couldn't.
Anyway, when I try to access my Forem with Fastly, it redirects too many times to the same URL. Then, it says "Backend doesn't work.", but the backend already works.
Should I need to add VCL snippets manually or should I wait for it to be completed automatically thanks to API settings?
If there were a complete guide for Fastly and Forem settings, it would be so great for me.
Understanding this kind of scale requires a lot of insight.
Is it Fastly doing this redirection? Can you provide the redirect chain? Please provide all of the details you can.
What backend are you pointing it at? Are you using a DNS name for it? Is that DNS name pointed at the right DO load balancer/droplet IP? Do you have TLS certificates setup for that DNS name? If you point your Fastly service at a different backend, does that work?
Please be very specific, this is the most likely place for this problem to be.
It should be done automatically. You can check whether the snippets are in your VCL by clicking the "Service Configuration" tab on your Fastly service page and then clicking the "VCL Snippets" entry on the side nav. If they're not there, try running
bin/rails fastly:update_configsmanually on your Forem server and make sure it exits cleanly (status code0).Keep in mind, we can't see your configuration. In order for us to be able to help here, you'll need to be very detailed so we can understand what's going on. It sounds like there's some small detail being overlooked here, but we have no way of knowing what that is unless you tell us everything — not credentials, of course, but all the other details.
It currently looks like your domain name is resolving to a DigitalOcean IP address. I don't know if it was pointed at Fastly before and you changed it back, but I point it out because it further prevents us from being able to get insight into the problem, so we have to rely solely on what you tell us.
And when in doubt, make sure you're up to speed on the Forem Fastly docs.
Edit: I have found how I can run the fastly:update-configs command. It can be run with foremctl rake because it was a rake command :)
I'm sorry about that. I'm a newbie at Rails ecosystem, and I can be confused sometimes because Rails isn't similar to PHP or Python.
No worries 😄 I'll update the docs to include using
foremctlhere. It is definitely not obvious thatrakecommands are now runnable by (and encouraged to be run by) therailscommand.