forem.dev Community

loading...

Sidekiq Memory Issues?

bighitbiker3 profile image Elliott McNary ・1 min read

Has anyone else noticed Sidekiq take a lot of memory? It seems no matter how many Heroku Dynos I throw at it it's hitting 125% RAM usage.

I'm not very familiar with Sidekiq and have been trying numerous things found online to no avail, but does anyone have any ideas?

Discussion (7)

pic
Editor guide
Collapse
joshpuetz profile image
Josh Puetz

Hi Elliott!

We definitely see high memory usage with Sidekiq on production Forems. One thing to consider is what KIND of Heroku dyno's you're throwing at it: different dynos have different memory amounts.

One other thing to look at is how much swap your dyno's are using: having a high level of swap over time means you're definitely maxing our your memory usage and should add more capacity in terms of more dynos or high capacity dynos.

Collapse
bighitbiker3 profile image
Elliott McNary Author • Edited

Interesting I'll look more in to the memory swapping and see!

I was just throwing standard dynos at it and got it up to 2GBs ram and then we'd blown past my monthly budget for this little experiment so I started looking into other solutions.

While I'm sure the performance line of dynos would have been plenty they are just out of the budget for right now

Collapse
joshpuetz profile image
Josh Puetz • Edited

You bring up a good point: we don't really have any published minimum and recommended server specs for running a forem instance. I've added an issue to add such documentation here: github.com/forem/forem/issues/12386

Thread Thread
bighitbiker3 profile image
Elliott McNary Author

Awesome thank you! Let me know if there's anything I can do to help

Collapse
coffeecraftcode profile image
Christina Gorton • Edited

I don't have any experience with Sidekiq so hopefully someone else on the team will chime in but from a quick search it seems like this is a known issue. You mentioned you have tried numerous things you have seen online so if you have seen these already feel free to ignore:

Reducing Sidekiq memory useage with jamalloc

And the issue where they found that 👆 info

Collapse
bighitbiker3 profile image
Elliott McNary Author • Edited

Thanks Christina! I'd actually found that and forgot to mention I implemented it.

After a bit more research I found it interesting how Sidekiq/Ruby handles memory (I'm a RoR newb). I noticed when Sidekiq finished a job that the memory did not get released back to the OS. And I subsequently learned that it just releases it back to Ruby. The process then just slowly increments its memory over time and blows the gasket on my dynos unless I kill them.

I've ended up implementing github.com/klaxit/sidekiq-worker-k... and it's working great! I haven't noticed any side effects, but my community is in its infant stages right now so I'm keeping a close eye on it.

Collapse
ben profile image
Ben Halpern

@rhymes want to weigh in here on the state of this?