Forem Creators and Builders

Muhammed Cetin
Muhammed Cetin

Posted on

Can I use Fastly on Self-hosted Forem?

Hello everyone,

I'm trying to up a self-hosted Forem community on Studyhab.com on DigitalOcean.

I have some problems with Forem for that project:

  1. StudyHab contains over 5 million articles, 10 million comments, and 2 million users. Its Postgresql database is a database service with 8 GB RAM, 4 vCPU, 115 GB Disk configuration on DigitalOcean, too. As you can see here, Forem is too slow with this database, and I couldn't find a solution to improve Forem's performance.

  2. Although I have tried to use Fastly with its API service and added the API Key and Service ID in rails.env file, Forem didn't work on Fastly.

What can I do about these problems? Do you have a suggestion for dealing with this size of a database?

Discussion (9)

Collapse
jamie profile image
Jamie Gaskins

@ce7in Can you go into more detail about how it didn't work on Fastly? What error messages was it showing, if any? That sort of thing.

Also, this is an impressive quantity of data on a Forem instance β€” multiple times more rows than DEV in all 3 of those tables. I'm curious how you're fitting that much data on a 115GB disk, though, because even though we have less data, we store a lot more than 115GB.

Collapse
ce7in profile image
Muhammed Cetin Author

Fitting the data on a 115 GB disk wasn't a problem, but publishing them fast enough is a big challenge. I don't know how to deal with this problem because I have not needed to do that before. I have been trying to find a solution for this performance issue for a few days, but I couldn't.

Anyway, when I try to access my Forem with Fastly, it redirects too many times to the same URL. Then, it says "Backend doesn't work.", but the backend already works.

Should I need to add VCL snippets manually or should I wait for it to be completed automatically thanks to API settings?

If there were a complete guide for Fastly and Forem settings, it would be so great for me.

Collapse
jamie profile image
Jamie Gaskins

publishing them fast enough is a big challenge

Understanding this kind of scale requires a lot of insight.

  • How fast is "fast enough"? What are your throughput requirements? What is your average/median/p90 latency?
  • How frequently are posts being published?
  • How certain are you that the database is the bottleneck? Could it be the application instead? Both?
  • How much compute capacity are you running on DigitalOcean for a service that size?
    • Are you running this all on a single DigitalOcean droplet or are you running several behind a load balancer?
    • What happens if you add more compute capacity (either larger droplets or more of them)?
  • How much of the CPU and RAM are the web processes using? Ditto for the Sidekiq workers? How many of each are you running?

when I try to access my Forem with Fastly, it redirects too many times to the same URL.

Is it Fastly doing this redirection? Can you provide the redirect chain? Please provide all of the details you can.

Then, it says "Backend doesn't work.", but the backend already works.

What backend are you pointing it at? Are you using a DNS name for it? Is that DNS name pointed at the right DO load balancer/droplet IP? Do you have TLS certificates setup for that DNS name? If you point your Fastly service at a different backend, does that work?

Please be very specific, this is the most likely place for this problem to be.

Should I need to add VCL snippets manually or should I wait for it to be completed automatically thanks to API settings?

It should be done automatically. You can check whether the snippets are in your VCL by clicking the "Service Configuration" tab on your Fastly service page and then clicking the "VCL Snippets" entry on the side nav. If they're not there, try running bin/rails fastly:update_configs manually on your Forem server and make sure it exits cleanly (status code 0).


Keep in mind, we can't see your configuration. In order for us to be able to help here, you'll need to be very detailed so we can understand what's going on. It sounds like there's some small detail being overlooked here, but we have no way of knowing what that is unless you tell us everything β€” not credentials, of course, but all the other details.

It currently looks like your domain name is resolving to a DigitalOcean IP address. I don't know if it was pointed at Fastly before and you changed it back, but I point it out because it further prevents us from being able to get insight into the problem, so we have to rely solely on what you tell us.

And when in doubt, make sure you're up to speed on the Forem Fastly docs.

Thread Thread
ce7in profile image
Muhammed Cetin Author • Edited on

Edit: I have found how I can run the fastly:update-configs command. It can be run with foremctl rake because it was a rake command :)

I'm sorry about that. I'm a newbie at Rails ecosystem, and I can be confused sometimes because Rails isn't similar to PHP or Python.

Thread Thread
jamie profile image
Jamie Gaskins

No worries πŸ˜„ I'll update the docs to include using foremctl here. It is definitely not obvious that rake commands are now runnable by (and encouraged to be run by) the rails command.

Collapse
ben profile image
Ben Halpern

We are very much here to support this Forem and help it scale! I'm sure @jamie will help you work through the current issues, and we'll be a team going forward to make sure it keeps working.

I'm curious about the project itself: Can you tell me more about Studyhab, what its mission is, where the users come from, etc. Very exciting to support something substantial like this.

Collapse
ce7in profile image
Muhammed Cetin Author

Of course, @ben ,

I'll send an email to you the mission and outline of the project in detail.

Thanks a lot, @jamie . I'm trying to analyze the indexes on Postgresql and to scale my servers effectively for today.

Collapse
ben profile image
Ben Halpern

Thank you! ben@forem.com if you weren't aware :)

Thread Thread
ce7in profile image
Muhammed Cetin Author

I thought to use bendhalpern@gmail.com, which is located on your profile :) However, I'll contact you at ben@forem.com as you said :)