Crossbar.io memory footprint

#1

Hi Tobias,

I’m trying to evaluate Crossbar at work; we are currently designing a new multi-process software architecture for our embedded systems (X86 & ARM based).

2 weeks ago, I have installed the Crossbar packages on Win7+Vmware with Linux Ubuntu 14.04 image and it ran smoothly.
The memory footprint for a demo based on hello:cpp template was ~10MB for the controller and additional ~10MB RAM for the router. Cpp workers < 1MB RAM. The memory footprint was stable even with additionnal cpp worker instances.

During the last days I’ve installed the latest version of crossbar (0.9.7-3 with Twisted 14.00) on a Linux Ubuntu 14.04 X86 target in order to measure the latencies.

During the test, I have checked the memory footprint and now it uses ~25MB RAM for controller and ~25MB for the router !

Do you think it has increased with the latest modifications of Crossbar or Twisted 14.00 ? Or is it a regression ?
50MB is definitely not possible for my ARM equipment since it is fitted with 128MB RAM only… that’s why 20MB for controller+router would be great !

Could you have a look and tell me if it is normal or not ? It is very important for me to know if you can decrease this memory footprint. Crossbar.io looks awesome and I would like to use it !

Precision: I use CPython and not Pypy (Pypy uses even more memory).

Last remark:
Initial latency was about 40ms (!) for 1 RPC call with AutobahnCpp client. I had to configure the boost socket with the “tcpNoDelay” option in the AutobahnCpp example, in order to fall between 1 and 2ms. May be the template could be updated accordingly.

socket.setTcpNoDelay(true); // to add just after the connection establishment

Thanks in advance for your help.

Julien

0 Likes

#2

Hi Julien,

Hi Tobias,

I'm trying to evaluate Crossbar at work; we are currently designing a
new multi-process software architecture for our embedded systems (X86 &
ARM based).

Great! Exactly one use case we designed Crossbar.io for.

2 weeks ago, I have installed the Crossbar packages on Win7+Vmware with
Linux Ubuntu 14.04 image and it ran smoothly.
The memory footprint for a demo based on hello:cpp template was ~10MB
for the controller and additional ~10MB RAM for the router. Cpp workers
< 1MB RAM. The memory footprint was stable even with additionnal cpp
worker instances.

Cpp workers (actually, any non-native worker, that is non-Python) is just "run as is".

Means: the memory overhead of Crossbar.io for running such a worker is just a couple of KB in the controller process.

During the last days I've installed the latest version of crossbar
(0.9.7-3 with Twisted 14.00) on a Linux Ubuntu 14.04 X86 target in order
to measure the latencies.

During the test, I have checked the memory footprint and now it uses
~25MB RAM for controller and ~25MB for the router !

Mmh. This sounds a little dubious to be honest;)

Try checking memory when just starting Python interactive prompt.

I get something like 35MB VSS and 5MB RES on 14.04/64bit. That is _Python_ without anything else.

VSS is "virtual memory", and practically nearly meaningless. You can have (almost) any amount of _virtual_ mem. "consumed".

RSS is the resident memory consumption of the process. That is closer to "actual" memory use. However, it doesn't take into account sharing of pages accross processes.

What you can however also try is: build Python _shared_:

./configure --prefix=$HOME/python278 --enable-shared

This will build Python as a SO that is shared between all processes using that Python.

Do you think it has increased with the latest modifications of Crossbar
or Twisted 14.00 ? Or is it a regression ?

Neither nor. How did you measure?

50MB is definitely not possible for my ARM equipment since it is fitted
with 128MB RAM only... that's why 20MB for controller+router would be
great !

Measuring memory use on a x86-64 host when the actual device is ARM 32 bit (I guess): not recommended;) For 2 reasons:

- "measuring" _actual_ memory consumption is tricky
- the difference in architecture can make a huge difference (64 bit vs 32 bit)

For these reasons, I'd strongly encourage to actual test this stuff on your real hardware device. Fire up Crossbar with 10 Py workers. _Test_ what happens.

Could you have a look and tell me if it is normal or not ? It is very
important for me to know if you can decrease this memory footprint.
Crossbar.io looks awesome and I would like to use it !

Sure. Let's track that down. For a start, pls tell us how you "measure" memory consumption .

Precision: I use CPython and not Pypy (Pypy uses even more memory).

While it is true in general that PyPy uses more memory, I wouldn't rule it out from the beginning. _Test_ it. On a ARM board like the Pi, it runs very well:

http://tavendo.com/blog/post/pypy-on-the-pi/

Btw, here are latency numbers:

http://tavendo.com/blog/post/autobahn-pi-benchmark/

Note: above was still done with plain AutobahnPython with a test router, but it'll be very similar for Crossbar.io.

Last remark:
Initial latency was about 40ms (!) for 1 RPC call with AutobahnCpp
client. I had to configure the boost socket with the "tcpNoDelay" option
in the AutobahnCpp example, in order to fall between 1 and 2ms. May be
the template could be updated accordingly.

socket.setTcpNoDelay(true); // to add just after the connection
establishment

You mean adding

socket.set_option(boost::asio::ip::tcp::no_delay(true));

here

https://github.com/crossbario/crossbar/blob/master/crossbar/crossbar/templates/hello/cpp/hello.cpp#L67

?

It's a tradeoff between CPU load and latency. Nagle will result in lower latency, but higher CPU load (roughly). It'll reduce the chances of coalescing outgoing bytes written by the app in different calls to socket.send() - hence higher CPU.

Cheers,
/Tobias

···

Am 26.07.2014 02:32, schrieb Julien Martin:

Thanks in advance for your help.

Julien

--
You received this message because you are subscribed to the Google
Groups "Autobahn" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to autobahnws+...@googlegroups.com
<mailto:autobahnws+...@googlegroups.com>.
To post to this group, send email to autob...@googlegroups.com
<mailto:autob...@googlegroups.com>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/autobahnws/bd00a236-e2ef-4d9b-9359-09e5f9040367%40googlegroups.com
<https://groups.google.com/d/msgid/autobahnws/bd00a236-e2ef-4d9b-9359-09e5f9040367%40googlegroups.com?utm_medium=email&utm_source=footer>.
For more options, visit https://groups.google.com/d/optout.

0 Likes

#3

Hi Tobias,
Hope you are well.
I’ve made the measure thanks to System Monitor. It measures real memory usage and not the virtual memory.
I’ve attached a screenshot where you can see that controller and router uses respectively 10 and 13 MB.
The file Crossbar 0.9.6-2 10MB.txt contains Crossbar’s traces .
The following version was used for the test: Ubuntu 14.04 LTS 32 bit with VMWare (I followed the “Installation On Linux Guide”, I used the Python which was already provided with Ubuntu)
Crossbar.io software versions:
Crossbar.io : 0.9.6-2
Autobahn : 0.8.10
Twisted : 14.0.0-EPollReactor
Python : 2.7.6
UTF8 Validator : wsaccel-0.6.2
XOR Masker : wsaccel-0.6.2
I made the same test with the latest version 0.9.7-3 of Crossbar. I’ve followed the “Install from Sources Guide” (build my own Python, get Crossbar from Git, etc…)
In this case I’ve 20 and 25MB of memory usage: May be it is linked to the different way of installing Crossbar or additionnal Python module that are imported ?
Then I’ve tried to update the 0.9.6-2 version (the one which takes only 10+13MB) by taping pip install -U crossbar.
Everything has been updated correctly, and the memory consumption is the same (10+13MB).
But now when I try to launch a hello:cpp demo (using AutobahnCpp) I get an error when the Cpp worker tries to connect to the Router (see detailed traces in file Crossbar 0.9.7-3 update + HelloCpp Error.txt
2014-07-31 07:22:12-0700 [Router 3835] CrossbarWampRawSocketServerProtocol instance has no attribute '_cbtid’
Any idea ? Is it a regression or I’ve missed something ? Can you help me ?
My ARM target is not available yet (only powered by WinCE OS at that time…) I will make some test with a Raspberry in september…
Regarding the latencies, yes, I mean adding socket.set_option(boost::asio::ip::tcp::no_delay(true)); in the hello.cpp. I agree with you that it is a compromise (In my use case I need very low latencies…)
Another thing: I think there’s a small bug in https://github.com/crossbario/crossbar/blob/master/crossbar/crossbar/templates/hello/cpp/web/index.html
Replace “com.example.add2” by “com.myapp.cpp.add2”, else the RPC call fails.

Thank you in advance if you can check the no attribute ‘_cbtid’ error. I’m stucked…

Cheers,
Julien

Crossbar 0.9.6-2 10MB.txt (7.18 KB)

Crossbar 0.9.7-3 update + HelloCpp Error.txt (8.19 KB)

0 Likes

#4

Hi Julien,

> Then I've tried to update the 0.9.6-2 version (the one which takes only

10+13MB) by taping pip install -U crossbar.
Everything has been updated correctly, and the memory consumption is the
same (10+13MB).

Ok. So the latest Crossbar on your system Python has the same memory footprint than the old one on that system Python?

Probably the system Python is stripped in some way or whatever. Hard to say without access to that box. But anyway: the real test rgd memory is running on real hardware.

But now when I try to launch a hello:cpp demo (using AutobahnCpp) I get
an error when the Cpp worker tries to connect to the Router (see
detailed traces in file /*Crossbar 0.9.7-3 update + HelloCpp Error.txt*/
*/2014-07-31 07:22:12-0700 [Router 3835]
CrossbarWampRawSocketServerProtocol instance has no attribute '_cbtid'/*

Any idea ? Is it a regression or I've missed something ? Can you help me ?

Yes, this is fixed on trunk.

Regarding the latencies, yes, I mean adding
socket.set_option(boost::asio::ip::tcp::no_delay(true)); in the
hello.cpp. I agree with you that it is a compromise (In my use case I
need very low latencies...)

Another thing: I think there's a small bug in
https://github.com/crossbario/crossbar/blob/master/crossbar/crossbar/templates/hello/cpp/web/index.html
Replace "/*com.example.add2*/" by "*/com.myapp.cpp.add2/*", else the RPC
call fails.

Done:

https://github.com/crossbario/crossbar/commit/b7459fe98cbbe4866a59a76506c412050b4ce841

Thank you in advance if you can check the no attribute '_cbtid' error.
I'm stucked...

Try running from latest GitHub .. should work.

Cheers,
/Tobias

0 Likes