Does WAMP server use a message queue broker?

#1

Hi -

I am planning to design a system (over websockets) that allows browser/non-browser “worker-clients” to publish data to a “server application process” and also the “server application process” to push data to the worker-clients. Note that the requirement is that I should be able to add new server-application-processes dynamically that register unique topics that they listen on.

I like the way WAMP has been defined. I was also looking into other options such as STOMP over websockets.

It is clear that STOMP requires a messsage queue broker server to which many clients can connect to - specifically in my case, the worker-clients and server application process would be a client to the broker.

Questions:

It appears to me that the Autobahn WAMP server does not have a broker server even though it implements a pubsub pattern. Am I correct?

If so, how can I dynamically add new WAMP applications that can register topics dynamically? Should each server process run its own WAMP server in a different port?

Thanks,

PC

0 Likes

#2

Hey, think I’ve done this.

There’s a central WAMP server, which has the dispatcher (broker) overridden. Other servers, let’s call them services, because they are actually WAMP clients – so that they don’t get confused with the central server – register at this central server. They tell them what kind of services they offer, ie “service/ping” or “service/stocks”.

Then clients connect to the central server, and request interest in services by saying that they are interested in messages with certain tags, ie “service/ping” or “service/stocks”.

When the service has new data, it sends an RPC message to the server (ie “service/stock” with stock data attached), the server then checks which clients registered interest, and publishes it to them.

If clients want to interact with services, they also post an RPC message to the server, who then hands it over to a specific service, or to all, if the client wanted some general information, ie which services are avaliable, and in which configuration state they are in (ie one ping service pings every 5 seconds with a certain message, and another one another message every 10 seconds).

Clients can also modify the configuration of services, make them save or load their configuration, and request all data to dynamically create a client-side user interface for the service.

I’m a bit confused about the complexity of this system, so I haven’t touched it for a couple of weeks. I’m not sure if it’s worth it continuing to research in that direction, somehow it feels amazingly flexible, but on the other hand a slim, custom system without WAMP might be a better solution.

Tell me if it goes in that direction, I’ll free some time to post it to github.

···

On Tue, Aug 7, 2012 at 12:19 AM, PC paddy....@gmail.com wrote:

Hi -
I am planning to design a system (over websockets) that allows browser/non-browser “worker-clients” to publish data to a “server application process” and also the “server application process” to push data to the worker-clients. Note that the requirement is that I should be able to add new server-application-processes dynamically that register unique topics that they listen on.

I like the way WAMP has been defined. I was also looking into other options such as STOMP over websockets.

It is clear that STOMP requires a messsage queue broker server to which many clients can connect to - specifically in my case, the worker-clients and server application process would be a client to the broker.

Questions:

It appears to me that the Autobahn WAMP server does not have a broker server even though it implements a pubsub pattern. Am I correct?

If so, how can I dynamically add new WAMP applications that can register topics dynamically? Should each server process run its own WAMP server in a different port?

Thanks,

PC

0 Likes

#3

Hi Paddy,

Hi -
     I am planning to design a system (over websockets) that allows
browser/non-browser "worker-clients" to publish data to a "server
application process" and also the "server application process" to push
data to the worker-clients. Note that the requirement is that I should
be able to add new server-application-processes dynamically that
register unique topics that they listen on.

I like the way WAMP has been defined. I was also looking into other
options such as STOMP over websockets.

It is clear that STOMP requires a messsage queue broker server to which
many clients can connect to - specifically in my case, the
worker-clients and server application process would be a client to the
broker.

Questions:
It appears to me that the Autobahn WAMP server does not have a broker
server even though it implements a pubsub pattern. Am I correct?

No, already AutobahnPython includes a message broker. To get started,
have a look at the 2nd example here:

http://autobahn.ws/python

and the PubSub examples here

https://github.com/tavendo/AutobahnPython/tree/master/examples/wamp/pubsub

If so, how can I dynamically add new WAMP applications that can register
topics dynamically? Should each server process run its own WAMP server
in a different port?

Running on different ports is of course possible, but there is no need to .. and I'd guess it's probably more complex. You can dynamically
spawn processes from your single AutobahnPython WAMP server running
on a single port using

http://twistedmatrix.com/documents/current/core/howto/process.html

This allows to dynamically spawn new processes and talk to them,
forwarding between PubSub and the processes' stdin/stdout in both directions.

You can register new topics for PubSub now only at the beginning of
a WAMP session, but also during it's lifetime.

Since you want to do something "more" than merely have the builtin
PubSub message broker dispatch events between clients (probably
talk to your processes), you might want to use custom handlers
which you can do:

registerHandlerForPubSub
registerHandlerForSub
registerHandlerForPub

http://autobahn.ws/python/reference
https://github.com/tavendo/AutobahnPython/tree/master/examples/wamp/pubsub/custom

Using this feature allows you to do your "special stuff", but reuse
the builtin broker for all the boilerplate around PubSub.

Hope that helps,
Tobias

···

Am 07.08.2012 00:19, schrieb PC:

Thanks,
PC

0 Likes

#4

Hi Daniel,

Thanks for the reply. What you have done sounds interesting to me. Please post the code and let me know.

Thanks,

PC

···

On Monday, August 6, 2012 3:47:46 PM UTC-7, Daniel F. wrote:

Hey, think I’ve done this.

There’s a central WAMP server, which has the dispatcher (broker) overridden. Other servers, let’s call them services, because they are actually WAMP clients – so that they don’t get confused with the central server – register at this central server. They tell them what kind of services they offer, ie “service/ping” or “service/stocks”.

Then clients connect to the central server, and request interest in services by saying that they are interested in messages with certain tags, ie “service/ping” or “service/stocks”.

When the service has new data, it sends an RPC message to the server (ie “service/stock” with stock data attached), the server then checks which clients registered interest, and publishes it to them.

If clients want to interact with services, they also post an RPC message to the server, who then hands it over to a specific service, or to all, if the client wanted some general information, ie which services are avaliable, and in which configuration state they are in (ie one ping service pings every 5 seconds with a certain message, and another one another message every 10 seconds).

Clients can also modify the configuration of services, make them save or load their configuration, and request all data to dynamically create a client-side user interface for the service.

I’m a bit confused about the complexity of this system, so I haven’t touched it for a couple of weeks. I’m not sure if it’s worth it continuing to research in that direction, somehow it feels amazingly flexible, but on the other hand a slim, custom system without WAMP might be a better solution.

Tell me if it goes in that direction, I’ll free some time to post it to github.

On Tue, Aug 7, 2012 at 12:19 AM, PC wrote:

Hi -
I am planning to design a system (over websockets) that allows browser/non-browser “worker-clients” to publish data to a “server application process” and also the “server application process” to push data to the worker-clients. Note that the requirement is that I should be able to add new server-application-processes dynamically that register unique topics that they listen on.

I like the way WAMP has been defined. I was also looking into other options such as STOMP over websockets.

It is clear that STOMP requires a messsage queue broker server to which many clients can connect to - specifically in my case, the worker-clients and server application process would be a client to the broker.

Questions:

It appears to me that the Autobahn WAMP server does not have a broker server even though it implements a pubsub pattern. Am I correct?

If so, how can I dynamically add new WAMP applications that can register topics dynamically? Should each server process run its own WAMP server in a different port?

Thanks,

PC

0 Likes

#5

Hi
Tobias,

Thanks.

If I understand you right, I would have a single “main” WAMP server different server-application-processes communicating with it through IPC. I would need to override the publish/subscribe routines of the base pubsub engine (of the main WAMP server) and do my own thing. Specifically in my case, for example, my publish routine would use IPC and send the data across to the appropriate server-process. Am I correct?

How do you suggest handling RPCs in this case? Note that the routine would actually be defined in one of the server-application-processes, not in the main WAMP server.

Thanks,

PC

···

On Tuesday, August 7, 2012 1:00:52 AM UTC-7, Tobias Oberstein wrote:

Hi Paddy,

Am 07.08.2012 00:19, schrieb PC:

Hi -

 I am planning to design a system (over websockets) that allows

browser/non-browser “worker-clients” to publish data to a "server

application process" and also the “server application process” to push

data to the worker-clients. Note that the requirement is that I should

be able to add new server-application-processes dynamically that

register unique topics that they listen on.

I like the way WAMP has been defined. I was also looking into other

options such as STOMP over websockets.

It is clear that STOMP requires a messsage queue broker server to which

many clients can connect to - specifically in my case, the

worker-clients and server application process would be a client to the

broker.

Questions:

It appears to me that the Autobahn WAMP server does not have a broker

server even though it implements a pubsub pattern. Am I correct?

No, already AutobahnPython includes a message broker. To get started,

have a look at the 2nd example here:

http://autobahn.ws/python

and the PubSub examples here

https://github.com/tavendo/AutobahnPython/tree/master/examples/wamp/pubsub

If so, how can I dynamically add new WAMP applications that can register

topics dynamically? Should each server process run its own WAMP server

in a different port?

Running on different ports is of course possible, but there is no need
to … and I’d guess it’s probably more complex. You can dynamically

spawn processes from your single AutobahnPython WAMP server running

on a single port using

http://twistedmatrix.com/documents/current/core/howto/process.html

This allows to dynamically spawn new processes and talk to them,

forwarding between PubSub and the processes’ stdin/stdout in both
directions.

You can register new topics for PubSub now only at the beginning of

a WAMP session, but also during it’s lifetime.

Since you want to do something “more” than merely have the builtin

PubSub message broker dispatch events between clients (probably

talk to your processes), you might want to use custom handlers

which you can do:

registerHandlerForPubSub

registerHandlerForSub

registerHandlerForPub

http://autobahn.ws/python/reference

https://github.com/tavendo/AutobahnPython/tree/master/examples/wamp/pubsub/custom

Using this feature allows you to do your “special stuff”, but reuse

the builtin broker for all the boilerplate around PubSub.

Hope that helps,

Tobias

Thanks,

PC

0 Likes

#6

Hi Tobias,
     Thanks.
  If I understand you right, I would have a single "main" WAMP server
different server-application-processes communicating with it through
IPC. I would need to override the publish/subscribe routines of the

Assuming you actually need to run "other stuff" in separate OS processes on the same machine (as running the WAMP server), the simplest IPC mechanism to use is probably pipes (standard
input/output).

base pubsub engine (of the main WAMP server) and do my own thing.
Specifically in my case, for example, my publish routine would use IPC
and send the data across to the appropriate server-process. Am I correct?

Yes. Technically, you don't override the WampServerProtocol base class
methods, but use registerHandlerForPub to install your custom logic
into the message broker.

How do you suggest handling RPCs in this case? Note that the routine
would actually be defined in one of the server-application-processes,
not in the main WAMP server.

Now you lost me: do you want a) to have logic in your background server
processes that you can call from WAMP clients via RPC, or b) have
logic in your background processes that get called when events are published by WAMP clients?

···

Am 07.08.2012 20:29, schrieb PC:

Thanks,
PC

On Tuesday, August 7, 2012 1:00:52 AM UTC-7, Tobias Oberstein wrote:

    Hi Paddy,

    Am 07.08.2012 00:19, schrieb PC:
     > Hi -
     > I am planning to design a system (over websockets) that allows
     > browser/non-browser "worker-clients" to publish data to a "server
     > application process" and also the "server application process" to
    push
     > data to the worker-clients. Note that the requirement is that I
    should
     > be able to add new server-application-processes dynamically that
     > register unique topics that they listen on.
     >
     > I like the way WAMP has been defined. I was also looking into other
     > options such as STOMP over websockets.
     >
     > It is clear that STOMP requires a messsage queue broker server to
    which
     > many clients can connect to - specifically in my case, the
     > worker-clients and server application process would be a client
    to the
     > broker.
     >
     > Questions:
     > It appears to me that the Autobahn WAMP server does not have a
    broker
     > server even though it implements a pubsub pattern. Am I correct?

    No, already AutobahnPython includes a message broker. To get started,
    have a look at the 2nd example here:

    http://autobahn.ws/python

    and the PubSub examples here

    https://github.com/tavendo/AutobahnPython/tree/master/examples/wamp/pubsub
    <https://github.com/tavendo/AutobahnPython/tree/master/examples/wamp/pubsub>

     > If so, how can I dynamically add new WAMP applications that can
    register
     > topics dynamically? Should each server process run its own WAMP
    server
     > in a different port?

    Running on different ports is of course possible, but there is no need
    to .. and I'd guess it's probably more complex. You can dynamically
    spawn processes from your single AutobahnPython WAMP server running
    on a single port using

    http://twistedmatrix.com/documents/current/core/howto/process.html
    <http://twistedmatrix.com/documents/current/core/howto/process.html>

    This allows to dynamically spawn new processes and talk to them,
    forwarding between PubSub and the processes' stdin/stdout in both
    directions.

    You can register new topics for PubSub now only at the beginning of
    a WAMP session, but also during it's lifetime.

    Since you want to do something "more" than merely have the builtin
    PubSub message broker dispatch events between clients (probably
    talk to your processes), you might want to use custom handlers
    which you can do:

    registerHandlerForPubSub
    registerHandlerForSub
    registerHandlerForPub

    http://autobahn.ws/python/reference
    <http://autobahn.ws/python/reference>
    https://github.com/tavendo/AutobahnPython/tree/master/examples/wamp/pubsub/custom
    <https://github.com/tavendo/AutobahnPython/tree/master/examples/wamp/pubsub/custom>

    Using this feature allows you to do your "special stuff", but reuse
    the builtin broker for all the boilerplate around PubSub.

    Hope that helps,
    Tobias

     >
     > Thanks,
     > PC

0 Likes

#7

Hi Tobias,

 Thanks.

If I understand you right, I would have a single “main” WAMP server

different server-application-processes communicating with it through

IPC. I would need to override the publish/subscribe routines of the

Assuming you actually need to run “other stuff” in separate OS processes
on the same machine (as running the WAMP server), the simplest IPC
mechanism to use is probably pipes (standard

input/output).

base pubsub engine (of the main WAMP server) and do my own thing.

Specifically in my case, for example, my publish routine would use IPC

and send the data across to the appropriate server-process. Am I correct?

Yes. Technically, you don’t override the WampServerProtocol base class

methods, but use registerHandlerForPub to install your custom logic

into the message broker.

How do you suggest handling RPCs in this case? Note that the routine

would actually be defined in one of the server-application-processes,

not in the main WAMP server.

Now you lost me: do you want a) to have logic in your background server

processes that you can call from WAMP clients via RPC, or b) have

logic in your background processes that get called when events are
published by WAMP clients?

a). similar to what you have defined in WAMP.

···

On Tuesday, August 7, 2012 2:21:50 PM UTC-7, Tobias Oberstein wrote:

Am 07.08.2012 20:29, schrieb PC:

Thanks,

PC

On Tuesday, August 7, 2012 1:00:52 AM UTC-7, Tobias Oberstein wrote:

Hi Paddy,
Am 07.08.2012 00:19, schrieb PC:
 > Hi -
 >      I am planning to design a system (over websockets) that allows
 > browser/non-browser "worker-clients" to publish data to a "server
 > application process" and also the "server application process" to
push
 > data to the worker-clients. Note that the requirement is that I
should
 > be able to add new server-application-processes dynamically that
 > register unique topics that they listen on.
 >
 > I like the way WAMP has been defined. I was also looking into other
 > options such as STOMP over websockets.
 >
 > It is clear that STOMP requires a messsage queue broker server to
which
 > many clients can connect to - specifically in my case, the
 > worker-clients and server application process would be a client
to the
 > broker.
 >
 > Questions:
 > It appears to me that the Autobahn WAMP server does not have a
broker
 > server even though it implements a pubsub pattern. Am I correct?
No, already AutobahnPython includes a message broker. To get started,
have a look at the 2nd example here:
[http://autobahn.ws/python](http://autobahn.ws/python)
and the PubSub examples here
[https://github.com/tavendo/AutobahnPython/tree/master/examples/wamp/pubsub](https://github.com/tavendo/AutobahnPython/tree/master/examples/wamp/pubsub)
<[https://github.com/tavendo/AutobahnPython/tree/master/examples/wamp/pubsub](https://github.com/tavendo/AutobahnPython/tree/master/examples/wamp/pubsub)>
 > If so, how can I dynamically add new WAMP applications that can
register
 > topics dynamically? Should each server process run its own WAMP
server
 > in a different port?
Running on different ports is of course possible, but there is no need
to .. and I'd guess it's probably more complex. You can dynamically
spawn processes from your single AutobahnPython WAMP server running
on a single port using
[http://twistedmatrix.com/documents/current/core/howto/process.html](http://twistedmatrix.com/documents/current/core/howto/process.html)
<[http://twistedmatrix.com/documents/current/core/howto/process.html](http://twistedmatrix.com/documents/current/core/howto/process.html)>
This allows to dynamically spawn new processes and talk to them,
forwarding between PubSub and the processes' stdin/stdout in both
directions.
You can register new topics for PubSub now only at the beginning of
a WAMP session, but also during it's lifetime.
Since you want to do something "more" than merely have the builtin
PubSub message broker dispatch events between clients (probably
talk to your processes), you might want to use custom handlers
which you can do:
registerHandlerForPubSub
registerHandlerForSub
registerHandlerForPub
[http://autobahn.ws/python/reference](http://autobahn.ws/python/reference)
<[http://autobahn.ws/python/reference](http://autobahn.ws/python/reference)>
[https://github.com/tavendo/AutobahnPython/tree/master/examples/wamp/pubsub/custom](https://github.com/tavendo/AutobahnPython/tree/master/examples/wamp/pubsub/custom)
<[https://github.com/tavendo/AutobahnPython/tree/master/examples/wamp/pubsub/custom](https://github.com/tavendo/AutobahnPython/tree/master/examples/wamp/pubsub/custom)>
Using this feature allows you to do your "special stuff", but reuse
the builtin broker for all the boilerplate around PubSub.
Hope that helps,
Tobias
 >
 > Thanks,
 > PC
0 Likes

#8

    Now you lost me: do you want a) to have logic in your background server
    processes that you can call from WAMP clients via RPC, or b) have
    logic in your background processes that get called when events are
    published by WAMP clients?

a). similar to what you have defined in WAMP.

In this case you can use

registerHandlerMethodForRpc
registerHandlerProcedureForRpc

where you then can implement the communication with your background processes.

When your background processes boot up, they also need to announce the
RPC endpoints they provide via IPC to the WAMP server which then needs
to setup the fowarding like above.

What language(s) are the background programs written in?

0 Likes

#9

PHP. But they can be written in any language since they are different processes. That’s one of the features that I’m shooting for.

···

What language(s) are the background programs written in?

0 Likes

#10

Ok, I really know too little to give a thorough advice on architecture
on such a thing. Just a couple of further hints:

- Who will be responsible for starting and monitoring the background processes (if one fails, who restarts etc)?
- Will those programs always run locally together with the frontend WAMP server?
- Essentially, you will need to have something like WAMP (at least the RPC part) over IPC
- In a way thats not sufficient, since the frontend WAMP server needs to know what RPC endpoint are available .. on which program

For PHP (standalone), there is a WAMP server implementation: Ratchet.

For transparent forwarding of WAMP RPCs to anything that exposes
a REST/HTTP or Ext.Direct API, our commercial offering Tavendo WebMQ
is capable of doing that. You can call from any WAMP client via WebSocket into anything with REST or Ext.Direct API today, and
database stored procedures in the coming future. You can also push
PubSub events from within your PHP i.e. This is a feature of WebMQ.
WebMQ is an integration server based on Autobahn, with a whole range
of flexible integration options.

Should a commercial offering be of interest to you, we should talk
and determine if it already could do what you want, and if not if
we can make that feature available.

Cheers,
Tobias

···

Am 08.08.2012 19:06, schrieb PC:

On Wednesday, August 8, 2012 9:26:56 AM UTC-7, Tobias Oberstein wrote:

     > Now you lost me: do you want a) to have logic in your
    background server
     > processes that you can call from WAMP clients via RPC, or b)
    have
     > logic in your background processes that get called when
    events are
     > published by WAMP clients?
     >
     > a). similar to what you have defined in WAMP.

    In this case you can use

    registerHandlerMethodForRpc
    registerHandlerProcedureForRpc

    where you then can implement the communication with your background
    processes.

    When your background processes boot up, they also need to announce the
    RPC endpoints they provide via IPC to the WAMP server which then needs
    to setup the fowarding like above.

    What language(s) are the background programs written in?

    PHP currently. But since they are different processes, they can be
    written in any language. This is one of the features that I'm
    shooting for.

0 Likes

#11

Hi Tobias,
    You probably have done a good amount of research on scalability of
WAMP Servers. Can you please let me know how it (built-in broker)
compares with a standalone broker (such as Apache ActiveMQ) in terms
of scalability?
-PC

···

On Aug 8, 10:18 am, Tobias Oberstein <tobias.o...@gmail.com> wrote:

Am 08.08.2012 19:06, schrieb PC:

> On Wednesday, August 8, 2012 9:26:56 AM UTC-7, Tobias Oberstein wrote:

> > Now you lost me: do you want a) to have logic in your
> background server
> > processes that you can call from WAMP clients via RPC, or b)
> have
> > logic in your background processes that get called when
> events are
> > published by WAMP clients?

> > a). similar to what you have defined in WAMP.

> In this case you can use

> registerHandlerMethodForRpc
> registerHandlerProcedureForRpc

> where you then can implement the communication with your background
> processes.

> When your background processes boot up, they also need to announce the
> RPC endpoints they provide via IPC to the WAMP server which then needs
> to setup the fowarding like above.

> What language(s) are the background programs written in?

> PHP currently. But since they are different processes, they can be
> written in any language. This is one of the features that I'm
> shooting for.

Ok, I really know too little to give a thorough advice on architecture
on such a thing. Just a couple of further hints:

- Who will be responsible for starting and monitoring the background
processes (if one fails, who restarts etc)?
- Will those programs always run locally together with the frontend WAMP
server?
- Essentially, you will need to have something like WAMP (at least the
RPC part) over IPC
- In a way thats not sufficient, since the frontend WAMP server needs to
know what RPC endpoint are available .. on which program

For PHP (standalone), there is a WAMP server implementation: Ratchet.

For transparent forwarding of WAMP RPCs to anything that exposes
a REST/HTTP or Ext.Direct API, our commercial offering Tavendo WebMQ
is capable of doing that. You can call from any WAMP client via
WebSocket into anything with REST or Ext.Direct API today, and
database stored procedures in the coming future. You can also push
PubSub events from within your PHP i.e. This is a feature of WebMQ.
WebMQ is an integration server based on Autobahn, with a whole range
of flexible integration options.

Should a commercial offering be of interest to you, we should talk
and determine if it already could do what you want, and if not if
we can make that feature available.

Cheers,
Tobias

0 Likes