Open file descriptor limits being ignored?

#1

Hi,

I am running Crossbar under an unprivileged user:

ubuntu@ip-XX-XX-XX-XX:~$ ps aux | grep crossbar
ubuntu 3782 0.2 4.4 238288 180540 ? Ss 13:54 0:04 crossbar-controller
ubuntu 3824 0.4 4.8 254740 198004 ? S 13:54 0:07 crossbar-worker [router]
ubuntu 7272 0.0 0.0 10460 940 pts/4 S+ 14:25 0:00 grep --color=auto crossbar

``

These are the limits at the filesystem level for that use:

ubuntu@ip-XX-XX-XX-XX:~$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 31538
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1048576
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 10000
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

``

as soon as the file descriptors opened by the application go over 1024, I am getting “Could not accept new connection” like the below and the server stops accepting new connections:

2016-01-11T14:19:22+0000 [Router 3824 log_legacy] Could not accept new connection (EMFILE)

``

Is there a possibility that there is some other (hardcoded?) limit that is being hit?

Here is my version output

ubuntu@ip-XX-XX-XX-XX:~$ /opt/crossbar/bin/crossbar version
Automatically choosing optimal Twisted reactor
Running on Linux and optimal reactor (epoll) was installed.
__ __ __ __ __ __ __ __
/ |__)/ \/__/`|) /\ |) |/
__,| \
/././|__)/~~| . |__/

Crossbar.io : 0.11.1
Autobahn : 0.10.9
UTF8 Validator : autobahn
XOR Masker : autobahn
JSON Codec : stdlib
MsgPack Codec : msgpack-python-0.4.6
Twisted : 15.4.0-EPollReactor
Python : 2.7.10/PyPy-2.6.1
OS : Linux-3.13.0-48-generic-x86_64-with-debian-jessie-sid
Machine : x86_64

``

Any help would be greatly appreciated

0 Likes

#2

This was in actual fact more of a systems issue, since I was using upstart and it ignores limit.conf, I have added the below to the upstart job to get it fixed:

limit nofile 1048576 1048576

``

BR

Jonathan

···

On Monday, 11 January 2016 15:28:37 UTC+1, Jonathan Bugeja wrote:

Hi,

I am running Crossbar under an unprivileged user:

ubuntu@ip-XX-XX-XX-XX:~$ ps aux | grep crossbar
ubuntu 3782 0.2 4.4 238288 180540 ? Ss 13:54 0:04 crossbar-controller
ubuntu 3824 0.4 4.8 254740 198004 ? S 13:54 0:07 crossbar-worker [router]
ubuntu 7272 0.0 0.0 10460 940 pts/4 S+ 14:25 0:00 grep --color=auto crossbar

``

These are the limits at the filesystem level for that use:

ubuntu@ip-XX-XX-XX-XX:~$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 31538
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1048576
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 10000
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

``

as soon as the file descriptors opened by the application go over 1024, I am getting “Could not accept new connection” like the below and the server stops accepting new connections:

2016-01-11T14:19:22+0000 [Router 3824 log_legacy] Could not accept new connection (EMFILE)

``

Is there a possibility that there is some other (hardcoded?) limit that is being hit?

Here is my version output

ubuntu@ip-XX-XX-XX-XX:~$ /opt/crossbar/bin/crossbar version
Automatically choosing optimal Twisted reactor
Running on Linux and optimal reactor (epoll) was installed.
__ __ __ __ __ __ __ __
/ |__)/ \/__/`|) /\ |) |/
__,| \
/././|__)/~~| . |__/

Crossbar.io : 0.11.1
Autobahn : 0.10.9
UTF8 Validator : autobahn
XOR Masker : autobahn
JSON Codec : stdlib
MsgPack Codec : msgpack-python-0.4.6
Twisted : 15.4.0-EPollReactor
Python : 2.7.10/PyPy-2.6.1
OS : Linux-3.13.0-48-generic-x86_64-with-debian-jessie-sid
Machine : x86_64

``

Any help would be greatly appreciated

0 Likes

#3

This was in actual fact more of a systems issue, since I was using upstart
and it ignores limit.conf, I have added the below to the upstart job to get
it fixed:

limit nofile 1048576 1048576

Good that you solved it. Be aware however that there's a system-wide
limit as well, which applies to all processes combined:

root@c4b278f184da:/# sysctl fs.file-max
fs.file-max = 797774

That was from an Ubuntu 14.04 docker container. So you may want to
increase that one as well.

Elvis

···

2016-01-11 15:54 GMT+01:00 Jonathan Bugeja <xad...@gmail.com>:

BR
Jonathan

On Monday, 11 January 2016 15:28:37 UTC+1, Jonathan Bugeja wrote:

Hi,

I am running Crossbar under an unprivileged user:

ubuntu@ip-XX-XX-XX-XX:~$ ps aux | grep crossbar
ubuntu 3782 0.2 4.4 238288 180540 ? Ss 13:54 0:04
crossbar-controller
ubuntu 3824 0.4 4.8 254740 198004 ? S 13:54 0:07
crossbar-worker [router]
ubuntu 7272 0.0 0.0 10460 940 pts/4 S+ 14:25 0:00 grep
--color=auto crossbar

These are the limits at the filesystem level for that use:

ubuntu@ip-XX-XX-XX-XX:~$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 31538
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1048576
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 10000
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

as soon as the file descriptors opened by the application go over 1024, I
am getting "Could not accept new connection" like the below and the server
stops accepting new connections:

2016-01-11T14:19:22+0000 [Router 3824 log_legacy] Could not accept
new connection (EMFILE)

Is there a possibility that there is some other (hardcoded?) limit that is
being hit?

Here is my version output

ubuntu@ip-XX-XX-XX-XX:~$ /opt/crossbar/bin/crossbar version
Automatically choosing optimal Twisted reactor
Running on Linux and optimal reactor (epoll) was installed.
     __ __ __ __ __ __ __ __
    / `|__)/ \/__`/__`|__) /\ |__) |/ \
    \__,| \\__/.__/.__/|__)/~~\| \. |\__/

Crossbar.io : 0.11.1
   Autobahn : 0.10.9
     UTF8 Validator : autobahn
     XOR Masker : autobahn
     JSON Codec : stdlib
     MsgPack Codec : msgpack-python-0.4.6
   Twisted : 15.4.0-EPollReactor
   Python : 2.7.10/PyPy-2.6.1
OS :
Linux-3.13.0-48-generic-x86_64-with-debian-jessie-sid
Machine : x86_64

Any help would be greatly appreciated

--
You received this message because you are subscribed to the Google Groups
"Crossbar" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to crossbario+...@googlegroups.com.
To post to this group, send email to cross...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/crossbario/8fa5ec36-922c-4a4a-a981-f3bb8522c390%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

0 Likes

#4

Hi Jonathan,

http://crossbar.io/docs/Network-Tuning/

Cheers,
/Tobias

···

Am 11.01.2016 um 15:28 schrieb Jonathan Bugeja:

Hi,

I am running Crossbar under an unprivileged user:

>
ubuntu@ip-XX-XX-XX-XX:~$ ps aux |grep crossbar
ubuntu 37820.24.4238288180540?Ss13:540:04crossbar-controller
ubuntu 38240.44.8254740198004? S 13:540:07crossbar-worker [router]
ubuntu 72720.00.010460940pts/4 S+14:250:00grep --color=autocrossbar
>

These are the limits at the filesystem level for that use:

>
ubuntu@ip-XX-XX-XX-XX:~$ ulimit -a
core file size (blocks,-c)0
data seg size (kbytes,-d)unlimited
scheduling priority (-e)0
file size (blocks,-f)unlimited
pending signals (-i)31538
max locked memory (kbytes,-l)64
max memory size (kbytes,-m)unlimited
open files (-n)1048576
pipe size (512bytes,-p)8
POSIX message queues (bytes,-q)819200
real-time priority (-r)0
stack size (kbytes,-s)8192
cpu time (seconds,-t)unlimited
max user processes (-u)10000
virtualmemory (kbytes,-v)unlimited
file locks (-x)unlimited
>

as soon as the file descriptors opened by the application go over 1024,
I am getting "Could not accept new connection" like the below and the
server stops accepting new connections:

>
2016-01-11T14:19:22+0000[Router3824log_legacy]Couldnotaccept
newconnection (EMFILE)
>

Is there a possibility that there is some other (hardcoded?) limit that
is being hit?

Here is my version output

>
ubuntu@ip-XX-XX-XX-XX:~$ /opt/crossbar/bin/crossbar version
Automaticallychoosing optimal Twistedreactor
Runningon Linuxandoptimal reactor (epoll)was installed.
      __ __ __ __ __ __ __ __
/`|__)/ \/__`/__`|__) /\ |__) |/ \
     \__,| \\__/.__/.__/|__)/~~\| \. |\__/

  Crossbar.io : 0.11.1
    Autobahn : 0.10.9
      UTF8 Validator : autobahn
      XOR Masker : autobahn
      JSON Codec : stdlib
      MsgPack Codec : msgpack-python-0.4.6
    Twisted : 15.4.0-EPollReactor
    Python : 2.7.10/PyPy-2.6.1
  OS : Linux-3.13.0-48-generic-x86_64-with-debian-jessie-sid
  Machine : x86_64
>

Any help would be greatly appreciated

--
You received this message because you are subscribed to the Google
Groups "Crossbar" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to crossbario+...@googlegroups.com
<mailto:crossbario+...@googlegroups.com>.
To post to this group, send email to cross...@googlegroups.com
<mailto:cross...@googlegroups.com>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/crossbario/f71bd674-854d-4acf-8cb7-0441befa602e%40googlegroups.com
<https://groups.google.com/d/msgid/crossbario/f71bd674-854d-4acf-8cb7-0441befa602e%40googlegroups.com?utm_medium=email&utm_source=footer>.
For more options, visit https://groups.google.com/d/optout.

0 Likes

#5

Hi Tobias,

Yes I did configure limits correctly. They are however ignored by upstart so thought it was worth pointing that out for anyone who's using upstart to manage their crossbar setup.

Regards,
Jonathan

···

On 13 Jan 2016, at 22:18, Tobias Oberstein <tobias.o...@gmail.com> wrote:

Hi Jonathan,

http://crossbar.io/docs/Network-Tuning/

Cheers,
/Tobias

Am 11.01.2016 um 15:28 schrieb Jonathan Bugeja:
Hi,

I am running Crossbar under an unprivileged user:

>
ubuntu@ip-XX-XX-XX-XX:~$ ps aux |grep crossbar
ubuntu 37820.24.4238288180540?Ss13:540:04crossbar-controller
ubuntu 38240.44.8254740198004? S 13:540:07crossbar-worker [router]
ubuntu 72720.00.010460940pts/4 S+14:250:00grep --color=autocrossbar
>

These are the limits at the filesystem level for that use:

>
ubuntu@ip-XX-XX-XX-XX:~$ ulimit -a
core file size (blocks,-c)0
data seg size (kbytes,-d)unlimited
scheduling priority (-e)0
file size (blocks,-f)unlimited
pending signals (-i)31538
max locked memory (kbytes,-l)64
max memory size (kbytes,-m)unlimited
open files (-n)1048576
pipe size (512bytes,-p)8
POSIX message queues (bytes,-q)819200
real-time priority (-r)0
stack size (kbytes,-s)8192
cpu time (seconds,-t)unlimited
max user processes (-u)10000
virtualmemory (kbytes,-v)unlimited
file locks (-x)unlimited
>

as soon as the file descriptors opened by the application go over 1024,
I am getting "Could not accept new connection" like the below and the
server stops accepting new connections:

>
2016-01-11T14:19:22+0000[Router3824log_legacy]Couldnotaccept
newconnection (EMFILE)
>

Is there a possibility that there is some other (hardcoded?) limit that
is being hit?

Here is my version output

>
ubuntu@ip-XX-XX-XX-XX:~$ /opt/crossbar/bin/crossbar version
Automaticallychoosing optimal Twistedreactor
Runningon Linuxandoptimal reactor (epoll)was installed.
     __ __ __ __ __ __ __ __
/`|__)/ \/__`/__`|__) /\ |__) |/ \
    \__,| \\__/.__/.__/|__)/~~\| \. |\__/

Crossbar.io : 0.11.1
   Autobahn : 0.10.9
     UTF8 Validator : autobahn
     XOR Masker : autobahn
     JSON Codec : stdlib
     MsgPack Codec : msgpack-python-0.4.6
   Twisted : 15.4.0-EPollReactor
   Python : 2.7.10/PyPy-2.6.1
OS : Linux-3.13.0-48-generic-x86_64-with-debian-jessie-sid
Machine : x86_64
>

Any help would be greatly appreciated

--
You received this message because you are subscribed to the Google
Groups "Crossbar" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to crossbario+...@googlegroups.com
<mailto:crossbario+...@googlegroups.com>.
To post to this group, send email to cross...@googlegroups.com
<mailto:cross...@googlegroups.com>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/crossbario/f71bd674-854d-4acf-8cb7-0441befa602e%40googlegroups.com
<https://groups.google.com/d/msgid/crossbario/f71bd674-854d-4acf-8cb7-0441befa602e%40googlegroups.com?utm_medium=email&utm_source=footer>.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "Crossbar" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/crossbario/EqScWNAKEbQ/unsubscribe.
To unsubscribe from this group and all its topics, send an email to crossbario+...@googlegroups.com.
To post to this group, send email to cross...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/crossbario/5696BF30.2090109%40gmail.com.
For more options, visit https://groups.google.com/d/optout.

0 Likes

#6

Hi Jonathan,

do you have

Modify /etc/security/limits.conf for the following

# wildcard does not work for root, but for all other users
* soft nofile 1048576
* hard nofile 1048576

for running under non-root?

also rgd running under unpriv. user: http://crossbar.io/docs/Running-on-privileged-ports/

Cheers,
/Tobias

···

Am 13.01.2016 um 22:24 schrieb xad...@gmail.com:

Hi Tobias,

Yes I did configure limits correctly. They are however ignored by upstart so thought it was worth pointing that out for anyone who's using upstart to manage their crossbar setup.

Regards,
Jonathan

On 13 Jan 2016, at 22:18, Tobias Oberstein <tobias.o...@gmail.com> wrote:

Hi Jonathan,

http://crossbar.io/docs/Network-Tuning/

Cheers,
/Tobias

Am 11.01.2016 um 15:28 schrieb Jonathan Bugeja:
Hi,

I am running Crossbar under an unprivileged user:

>
ubuntu@ip-XX-XX-XX-XX:~$ ps aux |grep crossbar
ubuntu 37820.24.4238288180540?Ss13:540:04crossbar-controller
ubuntu 38240.44.8254740198004? S 13:540:07crossbar-worker [router]
ubuntu 72720.00.010460940pts/4 S+14:250:00grep --color=autocrossbar
>

These are the limits at the filesystem level for that use:

>
ubuntu@ip-XX-XX-XX-XX:~$ ulimit -a
core file size (blocks,-c)0
data seg size (kbytes,-d)unlimited
scheduling priority (-e)0
file size (blocks,-f)unlimited
pending signals (-i)31538
max locked memory (kbytes,-l)64
max memory size (kbytes,-m)unlimited
open files (-n)1048576
pipe size (512bytes,-p)8
POSIX message queues (bytes,-q)819200
real-time priority (-r)0
stack size (kbytes,-s)8192
cpu time (seconds,-t)unlimited
max user processes (-u)10000
virtualmemory (kbytes,-v)unlimited
file locks (-x)unlimited
>

as soon as the file descriptors opened by the application go over 1024,
I am getting "Could not accept new connection" like the below and the
server stops accepting new connections:

>
2016-01-11T14:19:22+0000[Router3824log_legacy]Couldnotaccept
newconnection (EMFILE)
>

Is there a possibility that there is some other (hardcoded?) limit that
is being hit?

Here is my version output

>
ubuntu@ip-XX-XX-XX-XX:~$ /opt/crossbar/bin/crossbar version
Automaticallychoosing optimal Twistedreactor
Runningon Linuxandoptimal reactor (epoll)was installed.
      __ __ __ __ __ __ __ __
/`|__)/ \/__`/__`|__) /\ |__) |/ \
     \__,| \\__/.__/.__/|__)/~~\| \. |\__/

  Crossbar.io : 0.11.1
    Autobahn : 0.10.9
      UTF8 Validator : autobahn
      XOR Masker : autobahn
      JSON Codec : stdlib
      MsgPack Codec : msgpack-python-0.4.6
    Twisted : 15.4.0-EPollReactor
    Python : 2.7.10/PyPy-2.6.1
  OS : Linux-3.13.0-48-generic-x86_64-with-debian-jessie-sid
  Machine : x86_64
>

Any help would be greatly appreciated

--
You received this message because you are subscribed to the Google
Groups "Crossbar" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to crossbario+...@googlegroups.com
<mailto:crossbario+...@googlegroups.com>.
To post to this group, send email to cross...@googlegroups.com
<mailto:cross...@googlegroups.com>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/crossbario/f71bd674-854d-4acf-8cb7-0441befa602e%40googlegroups.com
<https://groups.google.com/d/msgid/crossbario/f71bd674-854d-4acf-8cb7-0441befa602e%40googlegroups.com?utm_medium=email&utm_source=footer>.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "Crossbar" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/crossbario/EqScWNAKEbQ/unsubscribe.
To unsubscribe from this group and all its topics, send an email to crossbario+...@googlegroups.com.
To post to this group, send email to cross...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/crossbario/5696BF30.2090109%40gmail.com.
For more options, visit https://groups.google.com/d/optout.

0 Likes