Celery + Autobahn

#1

Hi,

I’ve set up a python websocket client with Autobahn and would like to control it via Celery dispatched tasks. For example, I’d like to be able to dispatch a task to Celery and have it send a message on the open websocket.

Since Celery and Twisted each have their own event loops I’m having trouble getting the two to talk to each other. Would my best approach be to create a Twisted protocol that opens a (unix) socket and then use that socket to talk between the Celery process and the Autobahn process? (i.e. when my Celery worker handles a task it sends a message along the unix socket to the Twisted protocol, the Twisted protocol then talks to Autobahn and tells it to send a message on the websocket).

Alternatively, if anyone has a suggestion for a better way to control the websocket client from a remote process or server using message queues or some other technology, I’m all ears.

Thanks,

James

0 Likes

#2

It looks like the accepted approach is indeed to control Twisted via a separate control connection (e.g. Twisted listens on TCP). There are a couple of examples on SO:
http://stackoverflow.com/questions/23582706/twisted-and-autobahn-pub-sub-listen-on-secondary-port-for-messages

http://stackoverflow.com/questions/23584554/understanding-autobahn-and-twisted-integration

···

On Sunday, May 11, 2014 10:30:47 PM UTC-7, James Piechota wrote:

Hi,

I’ve set up a python websocket client with Autobahn and would like to control it via Celery dispatched tasks. For example, I’d like to be able to dispatch a task to Celery and have it send a message on the open websocket.

Since Celery and Twisted each have their own event loops I’m having trouble getting the two to talk to each other. Would my best approach be to create a Twisted protocol that opens a (unix) socket and then use that socket to talk between the Celery process and the Autobahn process? (i.e. when my Celery worker handles a task it sends a message along the unix socket to the Twisted protocol, the Twisted protocol then talks to Autobahn and tells it to send a message on the websocket).

Alternatively, if anyone has a suggestion for a better way to control the websocket client from a remote process or server using message queues or some other technology, I’m all ears.

Thanks,

James

0 Likes

#3

James,

I haven't worked with Celery, but the the approach you mention below seems sane (since as you mention, Twisted or asyncio come with their own event loop).

And yes, you could talk over Unix domain sockets or TCP loopback.

Probably, another approach would be using

http://docs.celeryproject.org/en/latest/userguide/remote-tasks.html

Don't know if you can make Celery issue a HTTP/GET or HTTP/POST to a preconfigured URL upon tasks being enqueued or in general tasks changing their state.

If so, this would make integration even easier: have Twisted Web running side-by-side with Autobahn under same event loop. Forward stuff received as HTTP/GET/POST in Twisted Web to Autobahn.

Cheers,
/Tobias

···

Am 12.05.2014 18:05, schrieb James Piechota:

It looks like the accepted approach is indeed to control Twisted via a
separate control connection (e.g. Twisted listens on TCP). There are a
couple of examples on SO:
http://stackoverflow.com/questions/23582706/twisted-and-autobahn-pub-sub-listen-on-secondary-port-for-messages
http://stackoverflow.com/questions/23584554/understanding-autobahn-and-twisted-integration

On Sunday, May 11, 2014 10:30:47 PM UTC-7, James Piechota wrote:

    Hi,

    I've set up a python websocket client with Autobahn and would like
    to control it via Celery dispatched tasks. For example, I'd like to
    be able to dispatch a task to Celery and have it send a message on
    the open websocket.

    Since Celery and Twisted each have their own event loops I'm having
    trouble getting the two to talk to each other. Would my best
    approach be to create a Twisted protocol that opens a (unix) socket
    and then use that socket to talk between the Celery process and the
    Autobahn process? (i.e. when my Celery worker handles a task it
    sends a message along the unix socket to the Twisted protocol, the
    Twisted protocol then talks to Autobahn and tells it to send a
    message on the websocket).

    Alternatively, if anyone has a suggestion for a better way to
    control the websocket client from a remote process or server using
    message queues or some other technology, I'm all ears.

    Thanks,

    James

--
You received this message because you are subscribed to the Google
Groups "Autobahn" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to autobahnws+...@googlegroups.com
<mailto:autobahnws+...@googlegroups.com>.
For more options, visit https://groups.google.com/d/optout.

0 Likes

#4

Thanks, Tobias. That’s a neat idea - particularly cause it looks like it would retain all the celery task management functionality (like aynchronously querying results) which I’d probably have to re-implement were I to use the raw socket approach.

Great, I have few things to look into now. Appreciate it!

James

···

On Mon, May 12, 2014 at 10:48 AM, Tobias Oberstein tobias.o...@gmail.com wrote:

James,

I haven’t worked with Celery, but the the approach you mention below seems sane (since as you mention, Twisted or asyncio come with their own event loop).

And yes, you could talk over Unix domain sockets or TCP loopback.

Probably, another approach would be using

http://docs.celeryproject.org/en/latest/userguide/remote-tasks.html

Don’t know if you can make Celery issue a HTTP/GET or HTTP/POST to a preconfigured URL upon tasks being enqueued or in general tasks changing their state.

If so, this would make integration even easier: have Twisted Web running side-by-side with Autobahn under same event loop. Forward stuff received as HTTP/GET/POST in Twisted Web to Autobahn.

Cheers,

/Tobias

Am 12.05.2014 18:05, schrieb James Piechota:

It looks like the accepted approach is indeed to control Twisted via a

separate control connection (e.g. Twisted listens on TCP). There are a

couple of examples on SO:

http://stackoverflow.com/questions/23582706/twisted-and-autobahn-pub-sub-listen-on-secondary-port-for-messages

http://stackoverflow.com/questions/23584554/understanding-autobahn-and-twisted-integration

On Sunday, May 11, 2014 10:30:47 PM UTC-7, James Piechota wrote:

Hi,



I've set up a python websocket client with Autobahn and would like

to control it via Celery dispatched tasks. For example, I'd like to

be able to dispatch a task to Celery and have it send a message on

the open websocket.



Since Celery and Twisted each have their own event loops I'm having

trouble getting the two to talk to each other. Would my best

approach be to create a Twisted protocol that opens a (unix) socket

and then use that socket to talk between the Celery process and the

Autobahn process? (i.e. when my Celery worker handles a task it

sends a message along the unix socket to the Twisted protocol, the

Twisted protocol then talks to Autobahn and tells it to send a

message on the websocket).



Alternatively, if anyone has a suggestion for a better way to

control the websocket client from a remote process or server using

message queues or some other technology, I'm all ears.



Thanks,



James

You received this message because you are subscribed to the Google

Groups “Autobahn” group.

To unsubscribe from this group and stop receiving emails from it, send

an email to autobahnws+unsubscribe@googlegroups.com

mailto:autobahnws+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

You received this message because you are subscribed to a topic in the Google Groups “Autobahn” group.

To unsubscribe from this topic, visit https://groups.google.com/d/topic/autobahnws/ilC6kBYO_p4/unsubscribe.

To unsubscribe from this group and all its topics, send an email to autobahnws+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

0 Likes