Need some input on my Crossbar design

#1

Hi all

I’m working on using Crossbar to provide PubSub notifications for changes on a Django model in a project I’m working on. I want some input because I realise

that my initial version, while it works, is probably doing things The Wrong Way ™.

So, I have my Django Tastypie application with a signal hooked up on a specific model. When the signal fires it enqueues a message in RabbitMQ using the Kombu SimpleQueue interface.

My Crossbar system currently consists of three Python components running inside the Router:

  • A ticket authentication component
  • An authorization component
  • A publisher component
    The Publisher component runs using the standard publisher loop illustrated in the Autobahn docs. Inside then loop it uses Kombu to consume messages enqueued by the Django signal.

These messages consist of the PK of a Django model to be loaded. Here comes the part I suspect is pretty Wrong. When it starts the Publisher component uses django.conf.settings.configure()

and django.setup() to enable usage of the Django ORM. I use the Django ORM to load the model instance specified the message, do a little bit of process and then publish it via Autobahn. As I

understand it, this pretty bad because Django ORM is not asynchronous and even worse because I’m running it inside the Router. Is my understanding on this correct?

The Authentication and Authorization components, much like the Publisher, also use django.conf.settings.configure() and django.setup() to enable usage of the Django ORM.

Calls to the Authentication RPC end-point perform a Django ORM query in order to determine if the given authid/ticket combination is a valid Django auth_user/tastypie_apikey combination and,

assuming they are, assign them a specific role.

The Authorizarion RPC end-point is used to determine if a connection with the specific role assigned by the Authentication components is permitted to subscribe to a topic published by the Publisher component.

The topics published by the Publisher vary based on the model instance data so, as with the Authentication, this code also makes use of the Django ORM to determine if a connection is authorized for

a given topic.

So, as you can see, that’s a lot of nasty synchronous DB access happening in my async code! I was feeling quite proud of my first Crossbar effort until I realised I was probably stabbing Crossbar

in the eye with those usages of Django ORM.

I’d like some advise on how I can fix this to not be so horrible.

The Publisher is actually pretty easy to fix, as I can move a bunch of code from that component into the Django signal and just submit the final data to be published to the Publisher component, thus

removing any usage of the Django ORM in the Publisher component.

The Authentication and Authorization are a little trickier. I was wondering if moving each of them into their own Container would reduce the impact of the Django ORM usage my moving it out of the

same process space as the Router? I’m still worried that even if I do this, though, the Django ORM will continue to be a performance killer due to its synchronous nature. Am I correct in thinking this?

I have some ideas about removing the need to use Django ORM in these two components as well, I just want to know if I should put time into investigating said ideas.

Any advice would be greatly appreciated. This my first Crossbar project and I’m really enjoying it but at the same time I do need to produce something that won’t fall over when we actually put it into

production :slight_smile:

Thanks

Adam

0 Likes

#2

For anyone wondering, I moved to a design where each of my Components runs inside a separate Container connecting to the Crossbar router over a local socket connection. To reduce the impact of ORM usage I’m caching the results for a period using Redis.

I am slowly working to alter the usage of blocking I/O in each Component to non-blocking alternatives at which point I may move them back into the Router.

0 Likes