Autobahn with twisted in detail

#1

Hi,

As Autobahn is developed over twisted. Twisted runs on single threaded. Thats make us think about following scenarios.

  1. Can Twisted/Autobahn use multiple core of CPUs, as it seems it’s a single threaded lib and it will only use one thread and one core cpu. In other words, it does not matter how many cores your have in your server system, it will just use one. So, may be the multiple systems with single or dual core are better than one systems with 10-12 core. Correct me if I am wrong?
  2. Can we run two or more different server applications (developed with twisted/autobahn) at one machine. Or they both will point to same lib and it will be a problem later on. Or we can with different virtual environments (python virtualenv and pip).
  3. Is there any issue/harm running autobahn/twisted with virtualenv and pip?
  4. Is it not recommended to use multiprocessing python package with Twisted/Autobahn?
  5. Is there completely no issues/harm listening or connecting with multiple type of servers using Autobahn/Twisted in one application? For example: Write an application using Autobahn/Twisted with following functionality (all in one server app):
  • Running two web socket server (using autobahn) (listening on port 9000 and 9200) for JS/Flash clients. (Expecting 10000 connections).
  • Running a smtp server (using twisted) to send mail report all critical issues, start, stop logs and application stats.
  • Running a TCP or UDP server (using twisted) for clustering (connected with each other servers to send the messages to all connected clients)
  • Running a TCP or UDP or Websocket client (using twisted or autobahn for WS) for clustering
  • Running a different process using multiprocess python package (no code for twisted/autobahn) to write log files for all the messages and client connection/disconnections.
  • Can we use Autobahn or Twisted without their services and application package and there is no harm using it just little extra code to start/stop the server?
  • What we have used in crossbar.io autobahn with twisted or asyncio?

To be honest, i am afraid running a complete server listening/connecting multiple ports writing logs all of that in one thread. :frowning:

The multiprocessing package I mentioned above: http://docs.python.org/2/library/multiprocessing.html

0 Likes

#2

Hi,

     As Autobahn is developed over twisted. Twisted runs on single
threaded. Thats make us think about following scenarios.

1. Can Twisted/Autobahn use multiple core of CPUs, as it seems it's a
    single threaded lib and it will only use one thread and one core
    cpu. In other words, it does not matter how many cores your have in
    your server system, it will just use one. So, may be the multiple
    systems with single or dual core are better than one systems with
    10-12 core. Correct me if I am wrong?

Twisted/Python (similar to NodeJS) runs the _networking_ on a single thread.

If you have _compute_ intensive work to do, you can run that on a pool of background worker threads (using Twisted deferToThread). This allows you to use all CPU cores.

If you want to use multiple cores for _networking_, you will need to run multiple processes and coordinate appropriately.

Scale-up (multi-core) and scale-out (multi-node) for networking are features that Crossbar.io adds over "raw Autobahn|Python".

2. Can we run two or more different server applications (developed with
    twisted/autobahn) at one machine. Or they both will point to same

Yes. No problem.

    lib and it will be a problem later on. Or we can with different
    virtual environments (python virtualenv and pip).

Not needed.

3. Is there any issue/harm running autobahn/twisted with virtualenv and
    pip?

No. Note the "pip" is an install tool. I guess the question more refers to virtualenv. And the answer regarding this is: no problem at all, and in fact often recommended.

4. Is it not recommended to use multiprocessing python package with
    Twisted/Autobahn?

No, the multiprocessing Python package is _not_ recommended. The recommended way of interprocess communication between multiple Twisted-based processes would be: Unix/Windows pipes, Unix domain sockets or loopback TCP/IP.

5. Is there completely no issues/harm listening or connecting with
    multiple type of servers using Autobahn/Twisted in one application?

No. Don't worry. This is exactly one strength of Twisted: multiple services in 1 server.

    For example: Write an application using Autobahn/Twisted with
    following functionality (all in one server app):
      * Running two web socket server (using autobahn) (listening on
        port 9000 and 9200) for JS/Flash clients. (Expecting 10000
        connections).
      * Running a smtp server (using twisted) to send mail report all
        critical issues, start, stop logs and application stats.
      * Running a TCP or UDP server (using twisted) for clustering
        (connected with each other servers to send the messages to all
        connected clients)
      * Running a TCP or UDP or Websocket client (using twisted or
        autobahn for WS) for clustering

All of above in one server programm, running as one or multiple process instances is no problem at all.

      * Running a different process using multiprocess python package
        (no code for twisted/autobahn) to write log files for all the
        messages and client connection/disconnections.

multiprocess package is not recommended.

6. Can we use Autobahn or Twisted without their services and
    application package and there is no harm using it just little extra
    code to start/stop the server?

How you start your application, both from OS startup sequence, and inside Python is totally up to you.

7. What we have used in crossbar.io autobahn with twisted or asyncio?

Crossbar.io is based on Autobahn|Python/Twisted.

Autobahn>Python supports asynchio currently only for raw WebSocket.

It _will_ support WAMP also, so that you can write any application components using both Twisted and asyncio.

Crossbar.io _itself_ (the router core) to which your application components connect, will probably stay Twisted only. However, this is totally irrelevant for any application component development.

In fact, we might opt to reimplement Crossbar.io router core in a competely different language, without any impact on application components.

To be honest, i am afraid running a complete server listening/connecting
multiple ports writing logs all of that in one thread. :frowning:

You should _measure_ before worrying;)

In my experience, a modern single Xeon CPU core has incredible performance. E.g. you can saturate a 10GbE Ethernet NIC easily with _1_ core!

In many cases, the same applies to disk/mass storage .. in fact to any I/O bound program.

/The multiprocessing package I mentioned above:
http://docs.python.org/2/library/multiprocessing.html/

Yes. Not recommended.

Cheers,
/Tobias

···

Am 07.02.2014 18:46, schrieb Chuck James:

/

--
You received this message because you are subscribed to the Google
Groups "Autobahn" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to autobahnws+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

0 Likes

#3

Okie information Thanks Tobias.

So far I understood that with autobahn/twisted, we can not take vantage for multi-core processor. But the answer of point 2, made us think about a possible solution.

*> 2. Can we run two or more different server applications (developed with *
*> twisted/autobahn) at one machine. Or they both will point to same *

*Yes. No problem. *

Possible solution with point 2: (to use all the core of system when needed).

We are running server (listening 9000 for ws) and keep checking the CPU usages (with psutil package), when CPU is higher than 70%, run/execute a new .py file with os.system(‘newScript.py params’) to starting new server app which will listen on 9100 and then all the clients will be redirected to 9100 port, who will try to connect on 9000.

and same way after port 9100 to 9200.

No idea what challenges we will face in this. But have few questions in mind:

  • Os.systems command may not start the process on a new CPU core, if this is correct then, is there a way to tell os to execute the py file on a new CPU core?
  • Is there any other way-around of this to let the server app (written in twisted/autobahn) using all the CPU cores? As it does not make sense in this era to write an app which will use only one CPU, while all the servers are more than 4 CPU core systems. Even though it should be an operating system task, if one process getting more load on one CPU cores, os should extend it to other CPU cores, after all these all cores are part of same CPU.
    Can you share, how are you guys implementing multi-core and multi-system in crossbar.io. What is the flow behind it. I am sure, you guys have figured it out a way around it to use the autobahn without the boundaries :)?

I have searched the autobahn group posts for reconnection of tcpconnect and found the twisted way solution to overrid function and connector.connect(). Just checking is it still the only solution, or autobahn providing any api now on this?

Thank,

Chuck.

···

On Friday, February 7, 2014 3:28:10 PM UTC-5, Tobias Oberstein wrote:

Am 07.02.2014 18:46, schrieb Chuck James:

Hi,

 As Autobahn is developed over twisted. Twisted runs on single

threaded. Thats make us think about following scenarios.

  1. Can Twisted/Autobahn use multiple core of CPUs, as it seems it’s a
single threaded lib and it will only use one thread and one core
cpu. In other words, it does not matter how many cores your have in
your server system, it will just use one. So, may be the multiple
systems with single or dual core are better than one systems with
10-12 core. Correct me if I am wrong?

Twisted/Python (similar to NodeJS) runs the networking on a single thread.

If you have compute intensive work to do, you can run that on a pool
of background worker threads (using Twisted deferToThread). This allows
you to use all CPU cores.

If you want to use multiple cores for networking, you will need to run
multiple processes and coordinate appropriately.

Scale-up (multi-core) and scale-out (multi-node) for networking are
features that Crossbar.io adds over “raw Autobahn|Python”.

  1. Can we run two or more different server applications (developed with
twisted/autobahn) at one machine. Or they both will point to same

Yes. No problem.

lib and it will be a problem later on. Or we can with different
virtual environments (python virtualenv and pip).

Not needed.

  1. Is there any issue/harm running autobahn/twisted with virtualenv and
pip?

No. Note the “pip” is an install tool. I guess the question more refers
to virtualenv. And the answer regarding this is: no problem at all, and
in fact often recommended.

  1. Is it not recommended to use multiprocessing python package with
Twisted/Autobahn?

No, the multiprocessing Python package is not recommended. The
recommended way of interprocess communication between multiple
Twisted-based processes would be: Unix/Windows pipes, Unix domain
sockets or loopback TCP/IP.

  1. Is there completely no issues/harm listening or connecting with
multiple type of servers using Autobahn/Twisted in one application?

No. Don’t worry. This is exactly one strength of Twisted: multiple
services in 1 server.

For example: Write an application using Autobahn/Twisted with
following functionality (all in one server app):
  * Running two web socket server (using autobahn) (listening on
    port 9000 and 9200) for JS/Flash clients. (Expecting 10000
    connections).
  * Running a smtp server (using twisted) to send mail report all
    critical issues, start, stop logs and application stats.
  * Running a TCP or UDP server (using twisted) for clustering
    (connected with each other servers to send the messages to all
    connected clients)
  * Running a TCP or UDP or Websocket client (using twisted or
    autobahn for WS) for clustering

All of above in one server programm, running as one or multiple process
instances is no problem at all.

  * Running a different process using multiprocess python package
    (no code for twisted/autobahn) to write log files for all the
    messages and client connection/disconnections.

multiprocess package is not recommended.

  1. Can we use Autobahn or Twisted without their services and
application package and there is no harm using it just little extra
code to start/stop the server?

How you start your application, both from OS startup sequence, and
inside Python is totally up to you.

  1. What we have used in crossbar.io autobahn with twisted or asyncio?

Crossbar.io is based on Autobahn|Python/Twisted.

Autobahn>Python supports asynchio currently only for raw WebSocket.

It will support WAMP also, so that you can write any application
components using both Twisted and asyncio.

Crossbar.io itself (the router core) to which your application
components connect, will probably stay Twisted only. However, this is
totally irrelevant for any application component development.

In fact, we might opt to reimplement Crossbar.io router core in a
competely different language, without any impact on application components.

To be honest, i am afraid running a complete server listening/connecting

multiple ports writing logs all of that in one thread. :frowning:

You should measure before worrying;)

In my experience, a modern single Xeon CPU core has incredible
performance. E.g. you can saturate a 10GbE Ethernet NIC easily with 1
core!

In many cases, the same applies to disk/mass storage … in fact to any
I/O bound program.

/The multiprocessing package I mentioned above:

http://docs.python.org/2/library/multiprocessing.html/

Yes. Not recommended.

Cheers,

/Tobias

/

/

/

/

You received this message because you are subscribed to the Google

Groups “Autobahn” group.

To unsubscribe from this group and stop receiving emails from it, send

an email to autobahnws+...@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

0 Likes

#4

Can you share, how are you guys implementing multi-core and multi-system
in crossbar.io. What is the flow behind it. I am sure, you guys have
figured it out a way around it to use the autobahn without
the boundaries :)?

Yes, a scalable architecture is part of what Crossbar.io brings. Any WAMP based application can be scaled by using Crossbar.io as a core application router.

/Tobias

0 Likes