Forem Creators and Builders

Cover image for This is the best optimization PR.
Akhil Naidu
Akhil Naidu

Posted on • Updated on

This is the best optimization PR.

Last night, I merged the Forem main branch into my fork and pushed the code into my Forem based on dokku server. When I was awake this morning, I was surprised to see the tremendous improvement within my RAM utilisation. Thanks to this PR

To show a overview how it fixed the memory leak, I will share you two screenshots. It is visibly clear that this is an awesome optimization PR.


This below image is before PR, it is dropping back at 80% because I configured it in that way.
before optimisation


This is after the PR.
After optimisation

Previously, my workers used to restart thrice in every 12 hours, but now for the last 12 hours, the memory stayed constant without any leaks.


This is a the picture representing the overall week performance.
Week Long Data

Discussion (15)

Collapse
manuel profile image
Manuel

Ohh that explain why in my instance was getting slower after 3 or 4 days of restarting it. Thanks Akhil.

I am having an issue merging forem update since last week, there is an issue with one css and i am not able to recreate the assets :( .

Collapse
djuber profile image
Daniel Uber • Edited

Forem's main branch is seeing the same thing (postcss processing fails) - it doesn't look like it's limited to your setup (but docker requiring the asset compilation to succeed might be blocking you in ways it's not blocking us).

I opened an issue about this (and am actively working to resolve the situation)
github.com/forem/forem/issues/14052

I believe this parse error you're seeing in the postcss_error.log happens after all the css files are generated (it's a post-processing step to handle backwards compatibility with browsers, among other things). But if you're blocked merging/updating that's a problem. One "work-around" in the meantime while we get this sorted would be to edit the package.json file's "postcss" script to not try to run on the files (the assets:precompile css files would be available, but not postprocessed), something like "postcss": "postcss --version", is enough to get it passing again (while not fixing anything).

However, if it's back to working for you (you reported that below) I would not do anything in the meantime.

Collapse
akhil profile image
Akhil Naidu Author

Can I know your setup and the issue you are using and facing now, if I know somethinf about it, I can help you fix it.

Collapse
manuel profile image
Manuel

Thanks Akhil! The error is that after merging the changes from forem branch, i tried to recreate the assets with:

docker-compose exec rails bundle exec rake assets:precompile
Enter fullscreen mode Exit fullscreen mode

However there is an error and is creating a file called: postcss_error.log and the error seems to be on a css file:

Expecting "SUB", "LPAREN", "NESTED_CALC", "NUMBER", "CSS_VAR", "LENGTH", "ANGLE", "TIME", "FREQ", "RES", "EMS", "EXS", "CHS", "REMS", "VHS", "VWS", "VMINS", "VMAXS", "PERCENTAGE", "expression", "math_expression", "value", "css_variable", "css_value", got unexpected "PREFIX"
    at /opt/apps/forem/public/assets/minimal-ce170b82da2bad10bc32420e34957902affc3ef433c0f625fb31926084604802.css:1:222887
Enter fullscreen mode Exit fullscreen mode

Before this I was making merge every 3 or 4 days without and issue, but since last week i got the error.

Thread Thread
akhil profile image
Akhil Naidu Author • Edited

So, I suppose you are using Forem in development on VPS suing docker? (If this is not the case, let me know)

If this is the case, you can try these to forcefully stop all the services and start services with the updated code.

  1. docker-compose down
  2. Update your Forem folder. If you have your own fork that would be great and more preferable. git push origin main
  3. docker-compose up -d (wait for a while, as sidekiq and web workers are loading in background)
Thread Thread
manuel profile image
Manuel • Edited

Yeah thats exactly what i was doing, after the docker-compose up -d all seems to be fine however when enter the app there are some css missing and tried to recreate but got the issue with that specific css.

I also tried removing the assets folder but getting the same error as well.

Thread Thread
akhil profile image
Akhil Naidu Author • Edited

Ok, I'm not sure how to recreate this in my own development instance, but I can sure suggest you a thing.

Hard reset
Enter fullscreen mode Exit fullscreen mode

Yes, I really mean it.

As you are in development insatnce, there is no need to worry of anything. You can remove everything and start everything new in 5 minutes and within 30minutes you willl have a new insatce.

But if you have configured something and you need to take backup of them, which is not an adviceable thing to do have in development instances. But there is a way for you to backup. There are many tutorials in youtube in how to backing up and restoring a something.dump(a postgress backup file). Follow with one of them and store your data. But I'm not sure how it will work, if you are not using s3 as storage.

Thread Thread
akhil profile image
Akhil Naidu Author

Remember to do this on caution, I'm suggesting this because of few reasons.

  1. I can't recreate your situation
  2. You are using development instance
  3. You are having this issue from few days.

Consider this above comment only as a reminder, rather than the only way of solving this.

Thread Thread
manuel profile image
Manuel

Thanks Akhil! I am going to debug more on the weekend to see what could be the issue, yeah if its no other way I will do a hard reset. I have backup of the database so just need to restore in the volume of the postgress docker. Will let you know my findings! thanks again for all the help your tutorials rocks!

Thread Thread
manuel profile image
Manuel

Just for the update, today i just download the most updated changes to my branch and the issue was solved, didnt know what happened.

Thread Thread
akhil profile image
Akhil Naidu Author

That is good to know. So, I'll take a note of it.

If any kind of .css error; replacing the fork will fix it.

Collapse
akhil profile image
Akhil Naidu Author • Edited

I would also suggest you configure ENV variables of Fastly; it is such an excellent CDN. I never expected such a good delivery. My Forem's Rails server, dB server, s3, and CDN are all in India and requests from India make my app feel like a web app.

Suppose you are familiar with Gatsby or Jamstack, or Static websites. With this new PR, and configured Fastly our Forem will works exactly like a static site.

Collapse
ellativity profile image
Ella Ang (she/her/elle) • Edited
Collapse
jamie profile image
Jamie Gaskins

Thanks for the ping @ellativity !

It turns out that there was a small memory leak that only shows itself in some configurations — it still existed even in DEV, but it was such a slow leak in DEV's config (and we run it on pretty beefy instances) that it didn't cause any trouble and was honestly only detectable in certain visualizations of the graph. Self-hosters were the main ones to see this issue. Specifically, if you didn't have certain env vars configured, the recurring job that caused the leak would fail and run 10x as often.

And there was another configuration bug that compounded it by holding telemetry data in memory if there was nowhere to report it to.

We fixed one of them last week and the other this week and the default self-host config now has a constant memory profile. Here's how it all played out on one of our internal self-hosted instances in the out-of-the-box configuration:

Forem self-host memory usage in the default configuration. Memory usage first goes up very steeply with both memory leaks in play, then after the first was fixed it rose more slowly but still shows evidence of a leak. After the second leak was fixed, memory usage remains relatively constant.

Note the sharp rise in memory in the beginning, then after the first leak was fixed we noticed the second one. Then after we fixed that one, it all smoothed right out! These fixes should help self-hosted communities immensely! 💯

Collapse
akhil profile image
Akhil Naidu Author

Yes, it really helped a lot.