Debug memory leak

Hi,

I think we see a memory leak in crossbar:

I think it is a bit strange that I have not found any other report regarding that. So it could be that it is only affecting us? I would be glad if you could confirm that: it would help us tracking down the issue (could it be a configuration issue?). Sad thing: I can not say how long that already exists. We somehow never looked at that metric during the last few months. Another bad thing is that we can only see that behavior on our production system which has a much higher load than our test/stage env’s.

The commit from which crossbar is running is the following one: https://github.com/rentouch/crossbar/commit/aeb756183226162747ebaffdc554d23f255ed473
In that we fixed https://github.com/crossbario/crossbar/issues/1243 and added a few lines which are helping us to prevent brute force attacks. But I can not imagine that those lines are the reason for the leek?

Update:


On the top graph you can see the spikes of authentications (wamp cra). These are happening when our ingress reloads (and all connections break - which is not ideal, but another issue). At each spike we see that the memory of crossbar increases rapidly.

I would appreciate any help (guidance on where to look) on this very much.

Hi,

just quickly:

  • in general, I can’t comment on anything but the official latest master https://github.com/crossbario/crossbar
  • if you have a fix for #1243, a PR would be highly welcome! that way, your changes will be on the official master branch, tested and published

having said that, from the information you posted, there are some crucial bits missing:

  • the exact platform (eg OS, Python flavor and version, TCP settings, …)
  • the full node configuration and the output of “crossbar version”

further, as you say, it seems memory jumps whenever a new chunk of 400-800 connections (all authenticated over WAMP-CRA? what type? static or dynamic? if dynamic, we need the code of the custom dynamic authenticator to check for leaks there).

however, missing in the charts are the total number of current connections. eg if the chunks of incoming connections will not be closed, then it is expected to see the memory consumption increase monotonically in time. etc

Cheers,
/Tobias