Large number of stacks with callee Autobahn-cpp example

#1

I’m looking at using Autobahn-cpp in an embedded arm system with bonefish as the WAMP router.

I built the callee example, but was surprised to see it has a very large memory foot print (a number of processes have been removed from this listing):

Mem: 61968K used, 177892K free, 0K shrd, 748600K buff, 235092K cached

CPU: 1% usr 9% sys 0% nic 88% idle 0% io 0% irq 0% sirq

Load average: 0.19 0.28 0.31 1/126 2641

PID PPID USER STAT VSZ %MEM %CPU COMMAND

2622 1031 root S 54500 23% 0% ./callee -r map300 -p 8888

1025 1024 root S 8104 3% 0% ./bonefish --realm map300 --websocket-port 8080 --rawsocket-port 8888

This so I dug in a bit, and pmap shows 6 “stack” signatures (we have 8K stacks on this box):

2622: ./callee -r map300 -p 8888

00010000 336K r-x-- callee

00073000 4K r---- callee

00074000 4K rw— callee

00075000 148K rw— [ anon ]

b3a00000 4K ----- [ anon ]

b3a01000 8188K rw— [ anon ]

b4200000 4K ----- [ anon ]

b4201000 8188K rw— [ anon ]

b4a00000 4K ----- [ anon ]

b4a01000 8188K rw— [ anon ]

b5200000 4K ----- [ anon ]

b5201000 8188K rw— [ anon ]

b5a00000 4K ----- [ anon ]

b5a01000 8320K rw— [ anon ]

b6221000 892K ----- [ anon ]

b63a8000 4K ----- [ anon ]

b63a9000 8188K rw— [ anon ]

b6ba8000 412K r-x-- libm.so.6

b6c0f000 60K ----- libm.so.6

b6c1e000 4K r---- libm.so.6

b6c1f000 4K rw— libm.so.6

but it only has one thread (from cat /proc/XXXX/status):

VmPMD: 0 kB

VmSwap: 0 kB

Threads: 1

SigQ: 0/1742

SigPnd: 0000000000000000

ShdPnd: 0000000000000000

Historically I’ve seen this when threads are not allowed to properly terminate, the stack memory gets orphaned and is not released until the program exits.

Any suggestions as this doesn’t scale well enough for me to use Autobahn-cpp?

Thanks!

0 Likes

#2

Specifically, this looks like stack leaks where threads haven’t been joined.

0 Likes

#3

This appears to be a consequence of the boost futures. Is there anyway to avoid the high number of stacks required?

I’ll have to look for another library if I can’t reduce the number of stacks as this will quickly exhaust our available memory and makes it unusable.

Thanks in advance for any suggestions!

0 Likes

#4

Hi Chris,

This so I dug in a bit, and pmap shows 6 "stack" signatures (we have 8K
stacks on this box):

..

but it only has one thread (from cat /proc/XXXX/status):

Right, this seems weird/wrong.

Any chance you could dig deeper by running the client under valgrind memcheck to possibly find out more?

Another test: rip out everything from the client but the establishment of the connection (up to onjoin) and let the client just sit still.

Yet another: do you see the same when the caller/publisher/subscriber (rather the callee)?

Cheers,
/Tobias

0 Likes

#5

Any chance you could dig deeper by running the client under valgrind
memcheck to possibly find out more?

It is a bit tricky to do this on the target, but I’ll spend a bit of time and see if I can get some results. I had little success with valgrind with this type of problem (detecting leaked stacks) in the past, but will see if I can get additional details.

I’m able to reproduce the same problem running callee in Ubunto, so I’ll try getting Valgrind results from there first.

Another test: rip out everything from the client but the establishment
of the connection (up to onjoin) and let the client just sit still.

I commented out the providers and the stacks shrunk by two. There appears to be a one to one correspondence to the boost future and the stack. Here is the pmap result:

~/bin/pmap 21682

21682: ./callee -r myrealm -p 8888

00010000 316K r-x-- callee

0006e000 4K r---- callee

0006f000 4K rw— callee

00070000 148K rw—
[ anon ]

b4a00000 4K -----
[ anon ]

b4a01000 8188K rw—
[ anon ]

b5200000 4K -----
[ anon ]

b5201000 8188K rw—
[ anon ]

b5a00000
4K ----- [ anon ]

b5a01000 8320K rw—
[ anon ]

b6221000 892K -----
[ anon ]

b6389000 4K -----
[ anon ]

b638a000 8188K rw—
[ anon ]

b6b89000 412K r-x-- libm.so.6

I also ran callee under Ubuntu (x86_64) and it shows the same stack pattern and some other very large memory requirement (this is the original callee source code):

0000000000400000
492K r-x-- callee

000000000067a000
4K r---- callee

000000000067b000
4K rw— callee

000000000067c000
4K rw— [ anon ]

0000000002506000
200K rw— [ anon ]

00007fd216ffe000
4K ----- [ anon ]

00007fd216fff000
8192K rw— [ anon ]

00007fd2177ff000
4K ----- [ anon ]

00007fd217800000
8192K rw— [ anon ]

00007fd218000000
132K rw— [ anon ]

00007fd218021000
65404K ----- [ anon ]

00007fd21c186000
4K ----- [ anon ]

00007fd21c187000
8192K rw— [ anon ]

00007fd21c987000
4K ----- [ anon ]

00007fd21c988000
8192K rw— [ anon ]

00007fd21d188000
4K ----- [ anon ]

00007fd21d189000
8192K rw— [ anon ]

00007fd21d989000
1056K r-x-- libm-2.23.so

Yet another: do you see the same when the caller/publisher/subscriber
(rather the callee)?

I originally encountered the problem using publisher. I was using publisher.cpp to setup scalibility testing with our target when I noted the large memory foot print. I haven’t tried with caller/subscriber, but I suspect they will have the same problem.

Cheers,

/Tobias

I appreciate you looking into this and will try to get information from valgrind for you.

0 Likes

#6

I’ve attached the output of two Valgrind runs:

valgrind --tool=memcheck --leak-check=full --log-file=vlogcallee.txt --show-leak-kinds=all ./callee -h 10.14.20.87 -r myrealm -p 8888

valgrind --tool=memcheck --leak-check=full --log-file=vlogpublisher.txt --show-leak-kinds=all ./publisher -h 10.14.20.87 -r myrealm -p 8888

These were run in x86_64 Ubunto with 1.58.0 Boost to a bonefish WAMP router running on another machine. I didn’t setup any suppressions so the callee log is quite verbose. callee was terminated using ^C on the command line. publisher of course runs and exits so no manual termination was required.

callee and publisher are the original code I got from the Git repository commit 2bea540078979bfc60b5fda66fca9422aa2a1f99

vlogcallee.txt (133 KB)

vlogpublisher.txt (1.55 KB)

0 Likes

#7

I’ve started working with cppWamp and it doesn’t have stack problems, but it isn’t using boost futures which I think is the source of the stack problem.

I would prefer to use Autobahn-cpp so let me know if there is additional information I can provide to help with the investigation.

0 Likes

#8

Hi Chris,

right now, I am at a conference (until monday), so unfort. wont be able to look. Maybe you could file an issue and ping/CC the Bonefish guys within the issue? They are also very familiar with the AutobahnCpp codebase (contributed lots of code) …

Cheers,

/Tobias

···

Am 18.11.2016 6:56 nachm. schrieb “Chris Welch” ve3...@gmail.com:

I’ve started working with cppWamp and it doesn’t have stack problems, but it isn’t using boost futures which I think is the source of the stack problem.

I would prefer to use Autobahn-cpp so let me know if there is additional information I can provide to help with the investigation.

You received this message because you are subscribed to the Google Groups “Autobahn” group.

To unsubscribe from this group and stop receiving emails from it, send an email to autobahnws+unsubscribe@googlegroups.com.

To post to this group, send email to autob...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/autobahnws/d538488b-9b25-40d6-892a-133f208da2de%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

0 Likes

#9

- right now, I am at a conference (until monday), so unfort. wont be able to look. Maybe you could file an issue and
- ping/CC the Bonefish guys within the issue? They are also very familiar with the AutobahnCpp codebase (contributed
- lots of code) ..

Will do Tobias, like yourself though, I probably won't be able to get to it until Monday.

0 Likes

#10

I’ve filed an issue report for it and sent an email to the folks at Topology/bonefish.

Let me know if I can be of any further assistance.

0 Likes