Crossbar configuration

#1

Hello,

Thanks for spending so much time on this mailing list, I know how hard it can be.

I noticed you called for help for writting documentation, but realized I needed to be able to code a full application with it before I could write anything.

So I’m going to start coding :slight_smile:

Got some questions regarding configuring the beast :

  • What’s “cpu_affinity” for and how does it compare to running several nodes if you are looking for using several CPUs ? Can you dedicate CPUs to certain components ?

  • what’s “component.program” for ?

  • which part of the conf file defines component runnings inside the node (so they must be non blocking) and component running as separate processes (so they can block) ?

I also encountered a problem doing this :

     "transports": [
        {
           "type": "web",
           "endpoint": {
              "type": "tcp",
              "port": 8080
           },
           "paths": {
              "/static/": {
                 "type": "static",
                 "directory": "../static"
              },
              "/": {
                 "type": "static",
                 "directory": "../"
              },

              "ws": {
                 "type": "websocket",
                 "url": "ws://localhost:8080/ws"
              }
           }
        }

Indeed, if I go to http://127.0.0.1:8080/static/, I will get the following error :

2014-03-24 01:03:50+0100 [Controller 11741] Unhandled error in Deferred:
2014-03-24 01:03:50+0100 [Controller 11741] Unhandled Error
Traceback (most recent call last):
Failure: autobahn.wamp.exception.ApplicationError: ApplicationError(wamp.error.runtime_error)

What can cause this problem ? How can I get more verbosity to debug ? A full stack trace ?

0 Likes

#2

Michel,

Hello,

Thanks for spending so much time on this mailing list, I know how hard
it can be.

Exactly;)

I noticed you called for help for writting documentation, but realized I
needed to be able to code a full application with it before I could
write anything.

So I'm going to start coding :slight_smile:

That is of course welcome as well! Any feedback is welcome .. as is app coding ..

Got some questions regarding configuring the beast :

- What's "cpu_affinity"for and how does it compare to running several
nodes if you are looking for using several CPUs ? Can you dedicate CPUs
to certain components ?

Yep. Setting CPU affinity allows to "bind" a process (e.g. a Crossbar worker process or others) to one (or multiple given) CPU cores.

This is useful, since it avoids "cache trashing" .. it's performance optimization. When a OS process is not bound to a CPU, the OS is free to reschedule the process to run on different cores all the time. Which invalidates caches .. bad.

Crossbar will soon have the ability to scale on multi-core not only the app workers but also the router etc. And then we want to have that...

- what's "component.program" for ?

Please have a looked at

https://github.com/crossbario/crossbar/wiki/Node.js-Application-Components#running-nodesjs-components-under-crossbario

Note: this docs are still uncomplete (probably also a little out of date) .. Crossbar is moving fast .. I try to do my best to polish up the docs ..

- which part of the conf file defines component runnings inside the node
(so they must be non blocking) and component running as separate
processes (so they can block) ?

A Crossbar node has always a "node controller process". Nothing else runs inside that.

The "node controller" spawns background processes. Those can be different things, and whether they "block" internally or not isn't affecting the controller.

Now, a Python component host process is one of background process types. And such a process will host Python app components written using AutobahnPython .. and those should be non-blocking. They'll run under Twisted.

I also encountered a problem doing this :

          "transports": [
             {
                "type": "web",
                "endpoint": {
                   "type": "tcp",
                   "port": 8080
                },
                "paths": {
                   "/static/": {
                      "type": "static",
                      "directory": "../static"
                   },
                   "/": {
                      "type": "static",
                      "directory": "../"
                   },

                   "ws": {
                      "type": "websocket",
                      "url": "ws://localhost:8080/ws"
                   }
                }
             }

The subpath should not contain leading or trailing slashes .. just tested with the attached config:

oberstet@THINKPAD-T410S /c/Temp/test2
$ find .
.
./.crossbar
./.crossbar/config.json
./moo.txt
./my_static_stuff
./my_static_stuff/foo.txt

Opening http://127.0.0.1:8080/static/ will render a 404.

What does "crossbar version" give you?

oberstet@THINKPAD-T410S /c/Temp/test2
$ crossbar version

Crossbar.io software versions:

Crossbar.io : 0.9.2
Autobahn : 0.8.6
Twisted : 13.2.0-IOCPReactor
Python : 2.7.6
UTF8 Validator : wsaccel-0.6.2
XOR Masker : wsaccel-0.6.2

Indeed, if I go to http://127.0.0.1:8080/static/, I will get the
following error :

2014-03-24 01:03:50+0100 [Controller 11741] Unhandled error in Deferred:
2014-03-24 01:03:50+0100 [Controller 11741] Unhandled Error
     Traceback (most recent call last):
     Failure: autobahn.wamp.exception.ApplicationError:
ApplicationError(wamp.error.runtime_error)

What can cause this problem ? How can I get more verbosity to debug ? A
full stack trace ?

Good point! We need to make error conditions more transparent:

https://github.com/crossbario/crossbar/issues/39

config.json (1.23 KB)

···

Am 24.03.2014 01:05, schrieb Michel Desmoulin:

--
You received this message because you are subscribed to the Google
Groups "Autobahn" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to autobahnws+...@googlegroups.com
<mailto:autobahnws+...@googlegroups.com>.
For more options, visit https://groups.google.com/d/optout.

0 Likes

#3

Great, serving static files works. Is that recommanded to let it serve files for a smal to medium web site ? What order of magnitude is it slower compared to using something like nginx to do so ?

Another thing :

If I understand well, using “component.program” would allow me to run stuff like a files watcher triggering css preprocessor, javascript mignifier, a 3rd party streaming server, etc. So no need for supervisor anymore. Crossbar seems like knocking down circus as well.

And if I declare several “type”: “component.python” in the conf file, it will spawn several processes. Could I then declare one that I dedicate to blocking operations, so the other components can delegate work to it that require unported blocking libs ? Since communication is async, it seems like it would be a good strategy to let most of the apps non blocking while still be able to use ORMs, old school drivers and the like.

···

On Monday, March 24, 2014 1:05:13 AM UTC+1, Michel Desmoulin wrote:

Hello,

Thanks for spending so much time on this mailing list, I know how hard it can be.

I noticed you called for help for writting documentation, but realized I needed to be able to code a full application with it before I could write anything.

So I’m going to start coding :slight_smile:

Got some questions regarding configuring the beast :

  • What’s “cpu_affinity” for and how does it compare to running several nodes if you are looking for using several CPUs ? Can you dedicate CPUs to certain components ?

  • what’s “component.program” for ?

  • which part of the conf file defines component runnings inside the node (so they must be non blocking) and component running as separate processes (so they can block) ?

I also encountered a problem doing this :

     "transports": [
        {
           "type": "web",
           "endpoint": {
              "type": "tcp",
              "port": 8080
           },
           "paths": {
              "/static/": {
                 "type": "static",
                 "directory": "../static"
              },
              "/": {
                 "type": "static",
                 "directory": "../"
              },

              "ws": {
                 "type": "websocket",
                 "url": "ws://localhost:8080/ws"
              }
           }
        }

Indeed, if I go to http://127.0.0.1:8080/static/, I will get the following error :

2014-03-24 01:03:50+0100 [Controller 11741] Unhandled error in Deferred:
2014-03-24 01:03:50+0100 [Controller 11741] Unhandled Error
Traceback (most recent call last):
Failure: autobahn.wamp.exception.ApplicationError: ApplicationError(wamp.error.runtime_error)

What can cause this problem ? How can I get more verbosity to debug ? A full stack trace ?

0 Likes

#4

Great, serving static files works. Is that recommanded to let it serve
files for a smal to medium web site ? What order of magnitude is it

That's fine.

slower compared to using something like nginx to do so ?

https://github.com/oberstet/scratchbox/tree/master/python/twisted/sharedsocket

https://github.com/oberstet/scratchbox/raw/master/python/twisted/sharedsocket/results.pdf

Without further tuning/hacking on Twisted Web, expect roughly 30% the performance of Nginx when serving static files.

I know how that could be pumped up ... say to 50-70% the performance of Nginx maybe. That would be a little Twisted Web hacking project. Couple of days. I am overloaded;(

Another thing :

If I understand well, using "component.program" would allow me to run
stuff like a files watcher triggering css preprocessor, javascript
mignifier, a 3rd party streaming server, etc. So no need for supervisor

Exactly. The node controller of Crossbar monitors any processes it forks. It also can dynamically start, stop and restart such processes.

The idea is: have Crossbar start at system boot. Let Crossbar start workers as needed.

Here is

https://github.com/crossbario/crossbar/wiki/Automatic-startup-and-restart

how we do it on

https://demo.crossbar.io/

anymore. Crossbar seems like knocking down circus as well.

And if I declare several "type": "component.python"in the conf file, it
will spawn several processes. Could I then declare one that I dedicate

yes.

to blocking operations, so the other components can delegate work to it
that require unported blocking libs ? Since communication is async, it

yes.

seems like it would be a good strategy to let most of the apps non
blocking while still be able to use ORMs, old school drivers and the like.

yes;)

That is by design: multi-process architecture allows a lot of flexiblity. Wanna write 1 component in JS run under Node? No problem. Other component in AutobahnPython? Sure. etc.

Plus: those components can use WAMP or not (ok, starting a worker process _not_ talking WAMP will obviously limit it's integration .. but you still get the "free" start/restart/monitor thing).

And _if_ the component talks WAMP to the router, it can do so over different transports:

- loopback TCP
- Unix domain sockets
- stdio pipes

The latter now comes in handy e.g. with AutobahnCpp (C++ WAMP) : it talks WAMP over the stdio pipes created when Crossbar spawns the C++ thing ..

···

Am 24.03.2014 12:38, schrieb Michel Desmoulin:

On Monday, March 24, 2014 1:05:13 AM UTC+1, Michel Desmoulin wrote:

    Hello,

    Thanks for spending so much time on this mailing list, I know how
    hard it can be.

    I noticed you called for help for writting documentation, but
    realized I needed to be able to code a full application with it
    before I could write anything.

    So I'm going to start coding :slight_smile:

    Got some questions regarding configuring the beast :

    - What's "cpu_affinity"for and how does it compare to running
    several nodes if you are looking for using several CPUs ? Can you
    dedicate CPUs to certain components ?

    - what's "component.program" for ?

    - which part of the conf file defines component runnings inside the
    node (so they must be non blocking) and component running as
    separate processes (so they can block) ?

    I also encountered a problem doing this :

              "transports": [
                 {
                    "type": "web",
                    "endpoint": {
                       "type": "tcp",
                       "port": 8080
                    },
                    "paths": {
                       "/static/": {
                          "type": "static",
                          "directory": "../static"
                       },
                       "/": {
                          "type": "static",
                          "directory": "../"
                       },

                       "ws": {
                          "type": "websocket",
                          "url": "ws://localhost:8080/ws"
                       }
                    }
                 }

    Indeed, if I go to http://127.0.0.1:8080/static/, I will get the
    following error :

    2014-03-24 01:03:50+0100 [Controller 11741] Unhandled error in Deferred:
    2014-03-24 01:03:50+0100 [Controller 11741] Unhandled Error
         Traceback (most recent call last):
         Failure: autobahn.wamp.exception.ApplicationError:
    ApplicationError(wamp.error.runtime_error)

    What can cause this problem ? How can I get more verbosity to debug
    ? A full stack trace ?

--
You received this message because you are subscribed to the Google
Groups "Autobahn" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to autobahnws+...@googlegroups.com
<mailto:autobahnws+...@googlegroups.com>.
For more options, visit https://groups.google.com/d/optout.

0 Likes

#5

It’s so awesome I want to cry rainbow tears.

···

On Monday, March 24, 2014 1:05:13 AM UTC+1, Michel Desmoulin wrote:

Hello,

Thanks for spending so much time on this mailing list, I know how hard it can be.

I noticed you called for help for writting documentation, but realized I needed to be able to code a full application with it before I could write anything.

So I’m going to start coding :slight_smile:

Got some questions regarding configuring the beast :

  • What’s “cpu_affinity” for and how does it compare to running several nodes if you are looking for using several CPUs ? Can you dedicate CPUs to certain components ?

  • what’s “component.program” for ?

  • which part of the conf file defines component runnings inside the node (so they must be non blocking) and component running as separate processes (so they can block) ?

I also encountered a problem doing this :

     "transports": [
        {
           "type": "web",
           "endpoint": {
              "type": "tcp",
              "port": 8080
           },
           "paths": {
              "/static/": {
                 "type": "static",
                 "directory": "../static"
              },
              "/": {
                 "type": "static",
                 "directory": "../"
              },

              "ws": {
                 "type": "websocket",
                 "url": "ws://localhost:8080/ws"
              }
           }
        }

Indeed, if I go to http://127.0.0.1:8080/static/, I will get the following error :

2014-03-24 01:03:50+0100 [Controller 11741] Unhandled error in Deferred:
2014-03-24 01:03:50+0100 [Controller 11741] Unhandled Error
Traceback (most recent call last):
Failure: autobahn.wamp.exception.ApplicationError: ApplicationError(wamp.error.runtime_error)

What can cause this problem ? How can I get more verbosity to debug ? A full stack trace ?

0 Likes

#6

Thanks;)

Getting that kind of feedback is encouraging and motivating!

/Cheers
Tobias

···

Am 24.03.2014 13:26, schrieb Michel Desmoulin:

It's so awesome I want to cry rainbow tears.

0 Likes