Crossbar Memory Usage Constantly Increasing

#1

Hello,

I’m running a Crossbar instance in AWS (specifically, a t2.medium). I’ve noticed that the host keeps eating up more and more memory until it starts getting dangerously close to the memory available on the machine. After restarting the host, the same issue repeats itself. You can see a Grafana graph over the last couple of weeks below:

The dips on the above graph are dates on which I restarted the instance. On 10/13, I upgraded Crossbar from 17.6.1.post3 to the latest version 17.9.2. Crossbar is basically the only thing running on the machine. Additionally, when I run top, my top hit is the crossbar-worker:

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

1325 crossbar 20 0 1357344 1.215g 11212 S 3.0 31.5 262:40.67 crossbar-worker

So, it seems clear that Crossbar is chewing up more memory every day. I created a session and called “wamp.registration.list” and confirmed that our registrations are where I’d expect them to be (only about 160). I also checked the “wamp.registration.get”, “wamp.registration.list_callees”, and “wamp.registration.count_callees” and everything looks reasonable for each registration.

Finally, I tried connecting with Pyrasite, but was unable to get it to connect to the process.

Is anyone familiar with something I could be doing that might cause Crossbar to chew up extra memory? Does anyone have any other suggestions for how I might debug this issue?

Thanks,

Ryan

0 Likes

#2

I believe that I tracked down the memory leak: https://github.com/crossbario/crossbar/issues/1202

···

On Monday, October 16, 2017 at 7:03:02 PM UTC-7, Ryan Rueth wrote:

Hello,

I’m running a Crossbar instance in AWS (specifically, a t2.medium). I’ve noticed that the host keeps eating up more and more memory until it starts getting dangerously close to the memory available on the machine. After restarting the host, the same issue repeats itself. You can see a Grafana graph over the last couple of weeks below:

The dips on the above graph are dates on which I restarted the instance. On 10/13, I upgraded Crossbar from 17.6.1.post3 to the latest version 17.9.2. Crossbar is basically the only thing running on the machine. Additionally, when I run top, my top hit is the crossbar-worker:

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

1325 crossbar 20 0 1357344 1.215g 11212 S 3.0 31.5 262:40.67 crossbar-worker

So, it seems clear that Crossbar is chewing up more memory every day. I created a session and called “wamp.registration.list” and confirmed that our registrations are where I’d expect them to be (only about 160). I also checked the “wamp.registration.get”, “wamp.registration.list_callees”, and “wamp.registration.count_callees” and everything looks reasonable for each registration.

Finally, I tried connecting with Pyrasite, but was unable to get it to connect to the process.

Is anyone familiar with something I could be doing that might cause Crossbar to chew up extra memory? Does anyone have any other suggestions for how I might debug this issue?

Thanks,

Ryan

0 Likes

#3

Just as a matter of interest, which flavour and version of Python are you using?
(Specifically, are you using pypy?)

0 Likes

#4

We are running Python v3.5.2. The output of “crossbar version” is attached in the Github issue if you’re curious about any other packages that Crossbar uses.

···

On Tuesday, October 17, 2017 at 6:58:00 PM UTC-7, Gareth Bult wrote:

Just as a matter of interest, which flavour and version of Python are you using?
(Specifically, are you using pypy?)

0 Likes

#5

Ah, sorry, should’ve read more closely.

Now I am looking, are you actively using LMDB? I think I read recently Crossbar is using this for maintaining clustering information (?) I’m specifically using LMDB in place of Mongo (!) and I have found it does tend towards eating up all available memory … however I had thought this was more by design than a leak. i.e. if you set (for example) a 2G map size then as time goes by and data usage increases, more and more of that 2G will be mapped to RAM (subject to paging). As I type I’m wondering if there’s a way to force it to ‘unmap’ memory it’s not using, or whether I have an errant transaction somewhere I’m not closing … may be a red herring, but it might be interesting to check the map size of any LMDB’s you’re opening (?)

···

On Wednesday, 18 October 2017 04:17:05 UTC+1, Ryan Rueth wrote:

We are running Python v3.5.2. The output of “crossbar version” is attached in the Github issue if you’re curious about any other packages that Crossbar uses.

On Tuesday, October 17, 2017 at 6:58:00 PM UTC-7, Gareth Bult wrote:

Just as a matter of interest, which flavour and version of Python are you using?
(Specifically, are you using pypy?)

0 Likes

#6

I don’t believe we use LMDB, but Crossbar may be using it for something. I did find one memory leak in the linked GitHub issue, but it’s hard to say if that’s the only issue or not.

···

On Wednesday, October 18, 2017 at 12:42:33 AM UTC-7, Gareth Bult wrote:

Ah, sorry, should’ve read more closely.

Now I am looking, are you actively using LMDB? I think I read recently Crossbar is using this for maintaining clustering information (?) I’m specifically using LMDB in place of Mongo (!) and I have found it does tend towards eating up all available memory … however I had thought this was more by design than a leak. i.e. if you set (for example) a 2G map size then as time goes by and data usage increases, more and more of that 2G will be mapped to RAM (subject to paging). As I type I’m wondering if there’s a way to force it to ‘unmap’ memory it’s not using, or whether I have an errant transaction somewhere I’m not closing … may be a red herring, but it might be interesting to check the map size of any LMDB’s you’re opening (?)

On Wednesday, 18 October 2017 04:17:05 UTC+1, Ryan Rueth wrote:

We are running Python v3.5.2. The output of “crossbar version” is attached in the Github issue if you’re curious about any other packages that Crossbar uses.

On Tuesday, October 17, 2017 at 6:58:00 PM UTC-7, Gareth Bult wrote:

Just as a matter of interest, which flavour and version of Python are you using?
(Specifically, are you using pypy?)

0 Likes