Has anyone else noticed Sidekiq take a lot of memory? It seems no matter how many Heroku Dynos I throw at it it's hitting 125% RAM usage.
I'm not very familiar with Sidekiq and have been trying numerous things found online to no avail, but does anyone have any ideas?
Top comments (7)
Hi Elliott!
We definitely see high memory usage with Sidekiq on production Forems. One thing to consider is what KIND of Heroku dyno's you're throwing at it: different dynos have different memory amounts.
One other thing to look at is how much swap your dyno's are using: having a high level of swap over time means you're definitely maxing our your memory usage and should add more capacity in terms of more dynos or high capacity dynos.
Interesting I'll look more in to the memory swapping and see!
I was just throwing standard dynos at it and got it up to 2GBs ram and then we'd blown past my monthly budget for this little experiment so I started looking into other solutions.
While I'm sure the performance line of dynos would have been plenty they are just out of the budget for right now
You bring up a good point: we don't really have any published minimum and recommended server specs for running a forem instance. I've added an issue to add such documentation here: github.com/forem/forem/issues/12386
Awesome thank you! Let me know if there's anything I can do to help
I don't have any experience with Sidekiq so hopefully someone else on the team will chime in but from a quick search it seems like this is a known issue. You mentioned you have tried numerous things you have seen online so if you have seen these already feel free to ignore:
Reducing Sidekiq memory useage with jamalloc
And the issue where they found that 👆 info
Thanks Christina! I'd actually found that and forgot to mention I implemented it.
After a bit more research I found it interesting how Sidekiq/Ruby handles memory (I'm a RoR newb). I noticed when Sidekiq finished a job that the memory did not get released back to the OS. And I subsequently learned that it just releases it back to Ruby. The process then just slowly increments its memory over time and blows the gasket on my dynos unless I kill them.
I've ended up implementing github.com/klaxit/sidekiq-worker-k... and it's working great! I haven't noticed any side effects, but my community is in its infant stages right now so I'm keeping a close eye on it.
@rhymes want to weigh in here on the state of this?