How I can establish 5000 web scoket connections concurrently with echosever in python scripting?

#1

Hi,

I want to establish lot of websocket connections with autobahn echoserver in python language.

I have used subproccess module in python to do this. But after 1000-1500 connections my machine is going down.

I am calling a subscript from my main script.Here subsript is acting as subproccess.

Is there any good solution to do this?

My target is to support 5000 connections.

0 Likes

#2

Hi,

  I want to establish lot of websocket connections with autobahn
echoserver in python language.

I have used subproccess module in python to do this. But after 1000-1500
connections my machine is going down.

Using subprocess is the wrong approach (this will create many many OS level processes). Instead, the Twisted way is like this:

https://github.com/tavendo/AutobahnTestSuite/blob/master/autobahntestsuite/autobahntestsuite/massconnect.py

We have successfully tested up to 64k client connections from 1 AutobahnPython client process.

64k outgoing TCP connections is the maximum for 1 IP. To do more, you will need multiple IP addresses bound to your interface.

/Tobias

···

Am 13.08.2013 12:01, schrieb Shyam Biradar:

I am calling a subscript from my main script.Here subsript is acting as
subproccess.

Is there any good solution to do this?

My target is to support 5000 connections.

--
You received this message because you are subscribed to the Google
Groups "Autobahn" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to autobahnws+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

0 Likes

#3

Hi Tobias,

Thanks for the answer.

I am too on the similar boat with Shyam. So not starting a new thread.

I could establish the websockets connections using massconnect.py. But can you please give me some pointers how to send payload to those established websockets connections via massconnect?

Regards,

Vinay Bhalerao

···

I want to establish lot of websocket connections with autobahn

echoserver in python language.

I have used subproccess module in python to do this. But after 1000-1500

connections my machine is going down.

Using subprocess is the wrong approach (this will create many many OS
level processes). Instead, the Twisted way is like this:

https://github.com/tavendo/AutobahnTestSuite/blob/master/autobahntestsuite/autobahntestsuite/massconnect.py

We have successfully tested up to 64k client connections from 1
AutobahnPython client process.

64k outgoing TCP connections is the maximum for 1 IP. To do more, you
will need multiple IP addresses bound to your interface.

/Tobias

I am calling a subscript from my main script.Here subsript is acting as

subproccess.

Is there any good solution to do this?

My target is to support 5000 connections.

You received this message because you are subscribed to the Google

Groups “Autobahn” group.

To unsubscribe from this group and stop receiving emails from it, send

an email to autobahnws+...@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

0 Likes

#4

Hi Tobias,

Thanks for the answer.

I am too on the similar boat with Shyam. So not starting a new thread.

I could establish the websockets connections using massconnect.py. But
can you please give me some pointers how to send payload to those
established websockets connections via massconnect?

You can send stuff after the WebSocket connection has been established .. add sendMessage(..) after

https://github.com/tavendo/AutobahnTestSuite/blob/master/autobahntestsuite/autobahntestsuite/massconnect.py#L32

/Tobias

···

Am 16.08.2013 21:23, schrieb vinay bhalerao:

Regards,
Vinay Bhalerao

     > I want to establish lot of websocket connections with autobahn
     > echoserver in python language.
     >
     > I have used subproccess module in python to do this. But after
    1000-1500
     > connections my machine is going down.

    Using subprocess is the wrong approach (this will create many many OS
    level processes). Instead, the Twisted way is like this:

    https://github.com/tavendo/AutobahnTestSuite/blob/master/autobahntestsuite/autobahntestsuite/massconnect.py
    <https://github.com/tavendo/AutobahnTestSuite/blob/master/autobahntestsuite/autobahntestsuite/massconnect.py>

    We have successfully tested up to 64k client connections from 1
    AutobahnPython client process.

    64k outgoing TCP connections is the maximum for 1 IP. To do more, you
    will need multiple IP addresses bound to your interface.

    /Tobias

     >
     > I am calling a subscript from my main script.Here subsript is
    acting as
     > subproccess.
     >
     > Is there any good solution to do this?
     >
     > My target is to support 5000 connections.
     >
     > --
     > You received this message because you are subscribed to the Google
     > Groups "Autobahn" group.
     > To unsubscribe from this group and stop receiving emails from it,
    send
     > an email to autobah...@googlegroups.com <javascript:>.
     > For more options, visit https://groups.google.com/groups/opt_out
    <https://groups.google.com/groups/opt_out>.
     >

--
You received this message because you are subscribed to the Google
Groups "Autobahn" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to autobahnws+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

0 Likes

#5

Hi Tobias,

You need to make sure that you have the relevant limits raised on both client and server side. Which limits apply and what to change is specific to OS.

E.g. on Linux client, not only number of open files limit needs to be raised, but also ephemeral port range.

echo "1024 65535" > /proc/sys/net/ipv4/ip_local_port_range

Will allow for (65535-1024) outgoing TCP connections (from 1 IP).

On Windows, you need to edit a Registry key:

Computer/HKEY_LOCAL_MACHINE\SYSTEM\CurrenControlSet\Services\Tcpip\Parameters\MaxUserPort

=> "DWORD" with value "65534"

The error message "could not accept new connection" suggests that the settings on the server side are still lacking ..

/Tobias

···

Am 14.08.2013 14:07, schrieb Shyam Biradar:

I have modified my ulimit settings to following:
# ulimit -a
core file size (blocks, -c) unlimited
data seg size (kbytes, -d) unlimited
scheduling priority (-e) unlimited
file size (blocks, -f) unlimited
pending signals (-i) unlimited
max locked memory (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files (-n) 65000
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) unlimited
real-time priority (-r) unlimited
stack size (kbytes, -s) unlimited
cpu time (seconds, -t) unlimited
max user processes (-u) unlimited
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

-------------------------------------------------------------

Maximum number of open FDs

# cat /proc/sys/fs/file-max
785344
------------------------------------------------------------

Despite of these settings if I hit massconnect command by following options:
"connections": 2000,
       "batchsize": 100,
       "batchdelay": 2000,
       "retrydelay": 2000

On server side (echoserver) I am getting error:
"could not accept new connection"

I am missing any other settings?

Please have a look into this.

0 Likes