efficiency questions about how to publish

#1

Hi,

I was playing around with the dirwatch code, it is a very good example of how to mix async input from a different source and push it to the current subscribers. I took dirwatch, changed it to ‘listen’ for postgres changes, then publish the change event to all subscribers that need to know. Just amazing!

I am considering using Autobahn at my company for a generic web socket service. In the design I am using, I configure postgres to tell me that a change in the database has occurred. I then look through the subscriber (I keep this in the database too) and find the subscribers that may be interested in this database change. That is a single postgres query that returns an array of ‘eligible’ recipients. I then use this array in the publish call to make sure that the information only goes to recipients that are allowed to see it.

So, is this efficient? I mean from an Autobahn point of view. Does broadcasting to an ‘eligible’ list happen in an efficient manner? As the number of subscribers grow would it be more efficient to have different subscription ‘channels’ for each endpoint (subscriber), or a single ‘channel’ for all of them, and only direct to eligible subscribers?

Will I run in to any sort of concurrency issues with my database query? Since each incoming event needs to have a query in the database run to see who to send it to, should I expect congestion here? I am still new to this Autobahn. Do I get a different thread spun up for each event, or are the events serialized through a single thread? It feels like they are serialized through the same thread, I’m just not sure.

Kind of a related question, I guess the number of subscribers is limited to the number of ports, right? so, around 65K subscribers. Will Autobahn scale to that (I know, there are variables :slight_smile: )?

Thanks, great piece of code Autobahn is !

-g

0 Likes

#2

Hi,

I was playing around with the dirwatch code, it is a very good example
of how to mix async input from a different source and push it to the
current subscribers. I took dirwatch, changed it to 'listen' for
postgres changes, then publish the change event to all subscribers that
need to know. Just amazing!

Great! Good to hear the "lots of examples approach" is helpful, since its also a lot of work;)

I am considering using Autobahn at my company for a generic web socket
service. In the design I am using, I configure postgres to tell me that

Awesome! If you are doing a technical due diligence / evaluation and have specific needs, I'd be happy to help, either here, or direct mail me ..

a change in the database has occurred. I then look through the
subscriber (I keep this in the database too) and find the subscribers
that may be interested in this database change. That is a single
postgres query that returns an array of 'eligible' recipients. I then
use this array in the publish call to make sure that the information
only goes to recipients that are allowed to see it.

Now this is funny;) If you like Autobahn and WAMP, but are into database integration, check out the following. I guess you could get
smiling even more;)

http://crossbar.io/

In particular:

http://crossbar.io/howitworks/

Crossbar.io builds on Autobahn. It is still a little rough, since this is brand new. Crossbar.io was under internal development for nearly 2 years internally at Tavendo - we just brought that code base into the light of open-source very recently (and that broke some things we need to catch again). It's open-source with commercial support. Currently the Oracle integration is working, the PG lacks behind, but we are working on that also. It's just so many things to do.

Anyway, Crossbar.io is already used by customers for real-world products. Here is one based on Oracle:

http://youtu.be/9A9oDj42fqM
http://youtu.be/isjrpC2GH74

It's _completely_ Crossbar.io based. Single-page HTML5/JS (actually ExtJS) frontend, all business logic inside Oracle in PL/SQL, no "app server" whatsoever (!), but only Crossbar.io for integration with Oracle.

So, is this efficient? I mean from an Autobahn point of view. Does
broadcasting to an 'eligible' list happen in an efficient manner? As
the number of subscribers grow would it be more efficient to have
different subscription 'channels' for each endpoint (subscriber), or a
single 'channel' for all of them, and only direct to eligible subscribers?

Will I run in to any sort of concurrency issues with my database query?
  Since each incoming event needs to have a query in the database run to
see who to send it to, should I expect congestion here? I am still new
to this Autobahn. Do I get a different thread spun up for each event,
or are the events serialized through a single thread? It feels like
they are serialized through the same thread, I'm just not sure.

You want a different design for efficiency. Getting notified of database changes via PostgreSQL NOTIFY is one element definitely. It's just the most efficient channel (at least of the natively builtin). However it's much more powerful to be able to just "publish()" _from within_ PL/pgSQL or any other stored procedure language supported by PG - and you can do that in DB triggers of course to get table change events also. It's very powerful. If you are interested in technical details, the code for Crossbar.io is all on GitHub, however, what I am trying to suggest: no need to reinvent the wheel;)

In fact, doing away with app servers and complex multi-tier architectures with app code sprinkled over many layers was our original vision for WAMP! And we continue that road of creating the next-gen app integration. There is even more to come, we just started.

Kind of a related question, I guess the number of subscribers is limited
to the number of ports, right? so, around 65K subscribers. Will
Autobahn scale to that (I know, there are variables :slight_smile: )?

Firstly, subscribers != clients. Every client (a TCP/WebSocket/WAMP connection held by the server) can do many "subscribe()" to topics. The term "subscriber" is any client that is subscribed to a specific topic.

Then (I guess that is what you are actually after): no, the number of clients a single instance of Crossbar.io (or Autobahn) can handle is not limited to 64k. The 64k limit applies to _outgoing_ connections from some host (a client) on _one_ IP. If you alias your NIC with many IPs, that number will be #(NIC IP aliases) x 64k. The 64k is about _ephemeral ports_. To reach that you need to tune your OS.

Now the number of TCP connections incoming that Crossbar.io/Autobahn can handle is formally limited by the number of (source IP, source port, target IP, target port) combinations. So with fixed target IP/port it's still 64k _for each_ different client IP! Virtually no limit.

Now, there are practical limits of course. Memory. Others. I have tested Autobahn with 180k concurrently connected clients on _my notebook_ inside a _VM_! If you have any kind of more beefy machine, it'l be millions.

We are working on benchmarks and hard numbers for that, but I would roughly guess a sane number is some 100k-1Mio per node. We are also working to scale-up (multi-core) and scale-out (multi-node) Crossbar.io - our goal is to demonstrate scalability to 100 Mio. connections.

Thanks, great piece of code Autobahn is !

You're welcome! As said, contact me any time probably DM me, and we can work with your company - would be a pleasure,

/Tobias

···

Am 10.11.2013 00:53, schrieb Greg Fausak:

-g

--
You received this message because you are subscribed to the Google
Groups "Autobahn" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to autobahnws+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

0 Likes