Nested asynchronous (wamp) calls

#1

Hi everybody, I got the following situation:

Inside some wamp procedure i have other asynchronous calls, which are called in a chain using callbacks. Now, at the end, i need to return the result from my last inner asynchronous call to the original caller of my method (called through wamp rpc from another client). But as twisted is non blocking, the execution of my outer method ends returning nothing before my inner calls have finished. Chaining the inner calls together (because they depend on each others results) works fine. The asynchronous calls may be calls to other registered wamp procedures or any other asynchronous methods (e.g. in my case it is database interaction using a non blocking driver)

How can i wait/make my outer method return the result of the inner method as soon as it is ready?

Before you say “don’t do this, this breaks all the advantages of asynchronous programming”, i got my reason why i need this.

Code example to illustrate what i want:

class MyComponent(ApplicationSession):
@inlineCallbacks
def onJoin(self, details):

    def some_method(x, y):
        # ...
        async_result_1 = do_some_async_call(x)
        async_result_1.addCallback(fire_on_1)

        def fire_on_1(result_from_1):
            async_result_2 = do_some_other_async_call(y, result_from_1)
            async_result_2.addCallback(fire_on_2)

        def fire_on_2(result_from_2):
            return result_from_2 # <--result needs to get to caller of some_method()

``

Bringing nested calls even a step further, how would the following work:

Again i have a remote procedure with inner asynchronous calls. But now those internal calls do not depend on each other in a linear way. Lets say i have 4 inner calls (inner_1,…,inner_4). The calls inner_1, inner_2 and inner_3 do not depend on each other, but to make call inner_4 i have to wait for their results. Thinking in the “old” thread based parallel execution i would start calls inner_1…3 and wait for them to finish and then run inner_4. This does not work with the asynchronous model and simple chaining would be bad because i would potentially waste performance (e.g. inner_1 needs to wait for some network input, then inner_2 and inner_3 would be blocked as well for no good reason even when they could already perform their work)

Is there any way to do this? How would i access the results from inner_1…3 to use them in inner_4?

By the way i’m using python 3.5 and the current version of autobahn from pip.

hope you can enlighten my way a little :slight_smile:

0 Likes

#2

Hi,

yes, the implementation of a procedure in a callee can itself issue asynchronous calls.

When you want to issue multiple calls that should run parallel and do something when _all_ calls have finished, without chaining (which would lead to non-parallel, though still asynch. execution), have a look at:

https://twistedmatrix.com/documents/current/core/howto/defer.html#gatherresults
http://txaio.readthedocs.org/en/latest/api.html#txaio.gather

Cheers,
/Tobias

···

Am 27.01.2016 um 02:44 schrieb sieben tupel:

Hi everybody, I got the following situation:

Inside some wamp procedure i have other asynchronous calls, which are
called in a chain using callbacks. Now, at the end, i need to return the
result from my last inner asynchronous call to the original caller of my
method (called through wamp rpc from another client). But as twisted is
non blocking, the execution of my outer method ends returning nothing
before my inner calls have finished. Chaining the inner calls together
(because they depend on each others results) works fine. The
asynchronous calls may be calls to other registered wamp procedures or
any other asynchronous methods (e.g. in my case it is database
interaction using a non blocking driver)

How can i wait/make my outer method return the result of the inner
method as soon as it is ready?

Before you say "don't do this, this breaks all the advantages of
asynchronous programming", i got my reason why i need this.

Code example to illustrate what i want:
>
classMyComponent(ApplicationSession):
@inlineCallbacks
defonJoin(self,details):

defsome_method(x,y):
# ...
             async_result_1 =do_some_async_call(x)
             async_result_1.addCallback(fire_on_1)

deffire_on_1(result_from_1):
                 async_result_2 =do_some_other_async_call(y,result_from_1)
                 async_result_2.addCallback(fire_on_2)

deffire_on_2(result_from_2):
returnresult_from_2 # <--result needs to get to caller of some_method()
>

Bringing nested calls even a step further, how would the following work:

Again i have a remote procedure with inner asynchronous calls. But now
those internal calls do not depend on each other in a linear way. Lets
say i have 4 inner calls (/inner_1,...,inner_4/). The calls /inner_1,
inner_2 /and/inner_3/ do not depend on each other, but to make call
/inner_4/ i have to wait for their results. Thinking in the "old" thread
based parallel execution i would start calls /inner_1..3/ and wait for
them to finish and then run /inner_4/. This does not work with the
asynchronous model and simple chaining would be bad because i would
potentially waste performance (e.g. /inner_1/ needs to wait for some
network input, then /inner_2/ and /inner_3/ would be blocked as well for
no good reason even when they could already perform their work)

Is there any way to do this? How would i access the results from
/inner_1..3 /to use them in/inner_4/?

By the way i'm using python 3.5 and the current version of autobahn from
pip.

hope you can enlighten my way a little :slight_smile:

--
You received this message because you are subscribed to the Google
Groups "Autobahn" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to autobahnws+...@googlegroups.com
<mailto:autobahnws+...@googlegroups.com>.
To post to this group, send email to autob...@googlegroups.com
<mailto:autob...@googlegroups.com>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/autobahnws/4260a7d7-8dc4-48de-a564-b7044753a172%40googlegroups.com
<https://groups.google.com/d/msgid/autobahnws/4260a7d7-8dc4-48de-a564-b7044753a172%40googlegroups.com?utm_medium=email&utm_source=footer>.
For more options, visit https://groups.google.com/d/optout.

0 Likes

#3

Hi everybody, I got the following situation:

Inside some wamp procedure i have other asynchronous calls, which are called in a chain using callbacks. Now, at the end, i need to return the result from my last inner asynchronous call to the original caller of my method (called through wamp rpc from another client). But as twisted is non blocking, the execution of my outer method ends returning nothing before my inner calls have finished. Chaining the inner calls together (because they depend on each others results) works fine. The asynchronous calls may be calls to other registered wamp procedures or any other asynchronous methods (e.g. in my case it is database interaction using a non blocking driver)

How can i wait/make my outer method return the result of the inner method as soon as it is ready?

Before you say “don’t do this, this breaks all the advantages of asynchronous programming”, i got my reason why i need this.

Code example to illustrate what i want:

class MyComponent(ApplicationSession):

@inlineCallbacks
def onJoin(self, details):
    def some_method(x, y):
        # ...
        async_result_1 = do_some_async_call(x)
        async_result_1.addCallback(fire_on_1)
        def fire_on_1(result_from_1):
            async_result_2 = do_some_other_async_call(y, result_from_1)
            async_result_2.addCallback(fire_on_2)
        def fire_on_2(result_from_2):
            return result_from_2 # <--result needs to get to caller of some_method()

Can’t you just yield on the calls to your inner methods? You’re even using the inlineCallbacks decorator in your example above, but you’re not yielding anywhere.

The other approach would be to manually construct a Deferred which you return immediately in your endpoint (outermost) method, and which you .callback(…) on when you have the final result in your last inner method.

But sounds like you should just use inline callbacks and yield.

For your other question, just like Tobias said, there are mechanism to “gather” results. (I think the plain twisted method is gatherResults).

Cheers,

Elvis

···

Den 27 jan 2016 2:44 fm skrev “sieben tupel” 7tu...@googlemail.com:

Bringing nested calls even a step further, how would the following work:

Again i have a remote procedure with inner asynchronous calls. But now those internal calls do not depend on each other in a linear way. Lets say i have 4 inner calls (inner_1,…,inner_4). The calls inner_1, inner_2 and inner_3 do not depend on each other, but to make call inner_4 i have to wait for their results. Thinking in the “old” thread based parallel execution i would start calls inner_1…3 and wait for them to finish and then run inner_4. This does not work with the asynchronous model and simple chaining would be bad because i would potentially waste performance (e.g. inner_1 needs to wait for some network input, then inner_2 and inner_3 would be blocked as well for no good reason even when they could already perform their work)

Is there any way to do this? How would i access the results from inner_1…3 to use them in inner_4?

By the way i’m using python 3.5 and the current version of autobahn from pip.

hope you can enlighten my way a little :slight_smile:

You received this message because you are subscribed to the Google Groups “Autobahn” group.

To unsubscribe from this group and stop receiving emails from it, send an email to autobahnws+...@googlegroups.com.

To post to this group, send email to autob...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/autobahnws/4260a7d7-8dc4-48de-a564-b7044753a172%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

0 Likes

#4

hi, thank you very much, booth of you. The gatherresults is exactly what i need for the later case and in the first one i have tried using the @inlinecallbacks decorator before, but i made i mistake some place else so it didn’t work. Fortunately i figured out this other mistake and now it works as it is supposed to.

thank you very much, cheers mo

0 Likes

#5

hi, thank you very much, booth of you. The gatherresults is exactly what i
need for the later case and in the first one i have tried using the
@inlinecallbacks decorator before, but i made i mistake some place else so
it didn't work. Fortunately i figured out this other mistake and now it
works as it is supposed to.

thank you very much, cheers mo

Glad it worked out.

Elvis

···

2016-01-28 0:41 GMT+01:00 sieben tupel <7tu...@googlemail.com>:

--
You received this message because you are subscribed to the Google Groups
"Autobahn" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to autobahnws+...@googlegroups.com.
To post to this group, send email to autob...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/autobahnws/d991131b-b0e3-441d-befe-067341c7a362%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

0 Likes