It turns out that there was a small memory leak that only shows itself in some configurations β it still existed even in DEV, but it was such a slow leak in DEV's config (and we run it on pretty beefy instances) that it didn't cause any trouble and was honestly only detectable in certain visualizations of the graph. Self-hosters were the main ones to see this issue. Specifically, if you didn't have certain env vars configured, the recurring job that caused the leak would fail and run 10x as often.
And there was another configuration bug that compounded it by holding telemetry data in memory if there was nowhere to report it to.
We fixed one of them last week and the other this week and the default self-host config now has a constant memory profile. Here's how it all played out on one of our internal self-hosted instances in the out-of-the-box configuration:
Note the sharp rise in memory in the beginning, then after the first leak was fixed we noticed the second one. Then after we fixed that one, it all smoothed right out! These fixes should help self-hosted communities immensely! π―
I am an Engineering Physics graduate from IIT Guwahati, but merely not restricted to Physics itself. In the past few years, I involved in many Cybersecurity blogging and now on Open-source.
@jamie @djuber @jdoss π
Thanks for the ping @ellativity !
It turns out that there was a small memory leak that only shows itself in some configurations β it still existed even in DEV, but it was such a slow leak in DEV's config (and we run it on pretty beefy instances) that it didn't cause any trouble and was honestly only detectable in certain visualizations of the graph. Self-hosters were the main ones to see this issue. Specifically, if you didn't have certain env vars configured, the recurring job that caused the leak would fail and run 10x as often.
And there was another configuration bug that compounded it by holding telemetry data in memory if there was nowhere to report it to.
We fixed one of them last week and the other this week and the default self-host config now has a constant memory profile. Here's how it all played out on one of our internal self-hosted instances in the out-of-the-box configuration:
Note the sharp rise in memory in the beginning, then after the first leak was fixed we noticed the second one. Then after we fixed that one, it all smoothed right out! These fixes should help self-hosted communities immensely! π―
Yes, it really helped a lot.