Using multiple cores for routing

Hello,

We only use Crossbar’s routing features. In our typical running system, we have roughly 10,000 long-lived WAMP connections to Crossbar, and some smaller number of short, one-off connections. Those connections each register as their own entity. After that, all of the traffic is routing messages amongst those registered connections.

We currently only take advantage of a single core per instance. I’d like to be able to utilize additional cores on our instances. What is the best way for us to utilize more cores in our use case (i.e. mostly routing messages and occasionally establishing lots of new connections after restarting another service)?

I’ve read about the Proxy Workers, but it sounds like those won’t be helpful in routing all of the messages. Is that correct?

I’ve also read about r-links, but it sounds like those are not production-ready yet. Is that correct?

Thank you,
Ryan

Hi Ryan,

I’d like to be able to utilize additional cores on our instances. What is the best way for us to utilize more cores in our use case (i.e. mostly routing messages and occasionally establishing lots of new connections after restarting another service)?

Proxy workers are able to off-load (scale to multiple cores / boxes) all of these portions of the workload of WAMP routing:

  • accepting and authenticating incoming WAMP connections from clients
  • supporting multiple WAMP transports, including WebSocket and RawSocket
  • supporting all WAMP serializers, including JSON, CBOR, MessagePack and Flatbuffers
  • terminating TLS and off-loading encryption
  • serving Web services, such as static files, redirects, HTML templates and so on

We’ve recently added some docs for these new features

https://crossbario.com/docs/crossbarfx/scaling/introduction.html#web-and-router-clusters

So eg should your clients connect over WAMP-WebSocket-JSON-TLS, you will be able to scale substantially already using proxy workers alone.

Here is an example where we measured a test setup for a customer:

This test ran load clients on one machine, where the clients connect over RawSocket-TCP (no TLS, CBOR serialization) and call a WAMP procedure with 256 random bytes as the (single positional) argument.

The backend procedure called is running on the testee machine, and simply returns the 256 random bytes provided as call argument.

  • CrossbarFX configuration: 8 router worker, 16 proxy worker (ratio proxy-to-router workers is 2:1)
  • 128 client connections were used (#router X #proxies => 8 X 16 = 128)

Test results:

  • more than 150,000 WAMP calls/sec (@ 256 bytes/call) are performed by the load clients
  • traffic runs over a real network (AWS internal) with almost 1Gb/s WAMP client (up+down) traffic
  • CrossbarFX consumes 12 CPU cores and 6GB RAM
  • the test was run constantly at full load for more than an hour with zero errors
  • memory consumption remained constant, the testee machine stable

Besides proxy workers, the other element of clustering are indeed “router-to-router” links (aka “rlinks”).

These are used to scale the actual core routing for a single realm beyond a single worker process. This is not yet officially announced, but we’ve recently made RPC fully work (in addition to PubSub) over rlinks -/

Both proxy workers and rlinks are part of Crossbar.io OSS and fully usable “as is”, that is with manually written node configuration files. Crossbar.io FX is adding a master node that can automatically orchestrate, manage and monitor setups with many nodes / workers.

Actually, we will now very soonish publicly announce this stuff and also publish the complete source code for Crossbar.io FX as well (on GitHub).

Cheers,
/Tobias

1 Like

Thanks, @oberstet! It sounds like Proxy workers make the most sense to try first. Just to confirm, even if the majority of our traffic through Crossbar is only routing messages for long-lived connections, you think Proxy Workers still should be able to handle a decent amount of the load?

For a two core machine, would you recommend one proxy worker and one router? Or two proxy workers in addition to the router?

Thanks,
Ryan

Just to confirm, even if the majority of our traffic through Crossbar is only routing messages for long-lived connections, you think Proxy Workers still should be able to handle a decent amount of the load?

yep, exactly! compared to the WAMP routing decision logic and message handling, other parts like WebSocket processing, JSON parsing and TLS encryption consume lots more CPU cycles

For a two core machine, would you recommend one proxy worker and one router? Or two proxy workers in addition to the router?

I would try 2 proxy + 1 router and 4 proxy + 1 router workers - it will depend on the specific mix of connections

Thanks, @oberstet! I’ll give it a shot and follow up with how it goes.

Testing proxy workers now, seems to work! Some thoughts:

  1. It’d be nice to specify a number of replicas for a proxy worker. I want my workers to have the exact same config, just scaled to N instances. Currently, it seems like I have to just copy+paste the same proxy config N times.

Example config:

{
  "type": "proxy",
  "replicas": 2
}
  1. I’m getting intermittent wamp.error.authentication_failed errors after switching to proxy workers. Not sure why, I didn’t change the authentication config.

Checking out rlinks now, hopefully it’s easy to setup, the docs are a bit technical! I’d appreciate some guide on how to setup rlinks on kubernetes.

Really looking forward to the announcement.

Thanks for testing feedback! We need to hash out the remaining issues before the “big release”. And that includes authentication/authorization (support all methods with proxy and router worker clusters).

Rgd the “replicas” idea: in fact, we have something even better that also works with multiple nodes.

This command

crossbarfx shell --realm default add webcluster-node cluster1 all \
    --config '{"parallel": 8}'

will create, start and keep running a total of “number of nodes X 8” proxy workers on cluster1. Eg with 4 nodes paired to your management realm, above command will fire up and manage 32 proxy worker processes.

https://crossbario.com/docs/crossbarfx/scaling/examples.html#parallel-web-clusters

Rgd simpler, more practical docs: absolutely. This is another area we’re working on (before big release).

What we already have is a super convenient/powerful Terraform module

You can use it to create a complete AWS based Crossbar.io cloud setup.

Rgd K8: what exactly would you like to see? Ideas/suggestions?

Personally, I can see these building blocks (when assuming K8):

A. For each K8 (application) pod, add a Crossbar.io node as a side-car container that only runs router workers that listen locally (within the pod) but maintain rlinks to all other side-car nodes (that is, the side-cars are fully meshed)
B. Have a K8 pod with Crossbar.io proxy workers that run public listening endpoints, and keep backend connections to the router workers
C. Have the master node separated in another K8 pod

Incoming client connections would need to be directed (at TCP/IP level) by K8 to the proxy workers in Pod B. The proxy worker will direct (at WAMP level) by proxy workers to router workers running as sidecars (A). The correct setup of proxy/router workers and routes/connections/rlinks is managed by the master node.