how to do fully asynchronous publishing?

#1

@Tobias? :slight_smile:
how can i, from clientA, trigger a function on clientB, whereby clientB runs several functions and publishes to several endpoints, and each publish() is forwarded immediately through crossbar instead of waiting until clientB has finished performing all functions?
e.g.
browser js clientA does either an rpc call or publish to com.example.trigger
python backend clientB, having either a matching rpc function or subscription callback for com.example.trigger (doesn’t matter which i use) then runs callback(). callback will look like the following:

class Butterfly(ApplicationSession):

@inlineCallbacks
def __yield(self, uri, data):
logger.info(log some stuff)
yield self.publish(uri, data)

@wamp.register(com.example.trigger)

def callback(self, opts):
@asyncio.coroutine
def foo1(opts):
results=compute() # takes <1 second
__yield(com.example.something1, results)
@asyncio.coroutine
def foo2(opts):
results=compute() # takes <1 second
__yield(com.example.something2, results)
@asyncio.coroutine
def foo3(opts):
results=compute() # takes <1 second
__yield(com.example.something3, results)
@asyncio.coroutine
def foo4(opts):
results=compute() # takes 60 seconds
__yield(com.example.something4, results)

loop = asyncio.get_event.loop()

tasks = [asyncio.Task(foo1(opts)),
asyncio.Task(foo2(opts)),
asyncio.Task(foo3(opts)),
asyncio.Task(foo4(opts))]

loop.run_until_complete(asyncio.wait(tasks))

at present, crossbar router does see published data immediately for the first three functions foo1(), foo2(), and foo3(); but it waits until after foo4() has completed before any of the published data is forwarded to the browser client. --loglevel debug and tcpdump confirms it.

how do i make crossbar forward the messages immediately to subscribed clients using a single trigger?

0 Likes

#2

Hi,

This appears to be because the item that takes 60 seconds is doing so synchronously, and is blocking the entire reactor.

Twisted and asyncio can only do things when you let them - if you spend 60 seconds doing CPU time and not yielding to the reactor, it cannot run. They are both cooperative multitasking frameworks.

What is the nature of your processing here? Is it network I/O (like database access – which should be done using Twisted or asyncio specific libraries), disk I/O (reading/writing a lot of files – in which case you should use a threadpool/thread executor) or CPU-heavy (in which case you may want to use something like Twisted’s process support to spin up a new Python to do the work specifically, and communicate with it asynchronously)?

  • Amber
···

On Thursday, 3 December 2015 06:31:32 UTC+8, David Ford wrote:

@Tobias? :slight_smile:
how can i, from clientA, trigger a function on clientB, whereby clientB runs several functions and publishes to several endpoints, and each publish() is forwarded immediately through crossbar instead of waiting until clientB has finished performing all functions?
e.g.
browser js clientA does either an rpc call or publish to com.example.trigger
python backend clientB, having either a matching rpc function or subscription callback for com.example.trigger (doesn’t matter which i use) then runs callback(). callback will look like the following:

class Butterfly(ApplicationSession):

@inlineCallbacks
def __yield(self, uri, data):
logger.info(log some stuff)
yield self.publish(uri, data)

@wamp.register(com.example.trigger)

def callback(self, opts):
@asyncio.coroutine
def foo1(opts):
results=compute() # takes <1 second
__yield(com.example.something1, results)
@asyncio.coroutine
def foo2(opts):
results=compute() # takes <1 second
__yield(com.example.something2, results)
@asyncio.coroutine
def foo3(opts):
results=compute() # takes <1 second
__yield(com.example.something3, results)
@asyncio.coroutine
def foo4(opts):
results=compute() # takes 60 seconds
__yield(com.example.something4, results)

loop = asyncio.get_event.loop()

tasks = [asyncio.Task(foo1(opts)),
asyncio.Task(foo2(opts)),
asyncio.Task(foo3(opts)),
asyncio.Task(foo4(opts))]

loop.run_until_complete(asyncio.wait(tasks))

at present, crossbar router does see published data immediately for the first three functions foo1(), foo2(), and foo3(); but it waits until after foo4() has completed before any of the published data is forwarded to the browser client. --loglevel debug and tcpdump confirms it.

how do i make crossbar forward the messages immediately to subscribed clients using a single trigger?

0 Likes

#3

i considered this, but discarded this line of thought for two reasons.

  1. this 60 second task yields in several places

  2. running crossbar with loglevel debug (and confirmed by tcpdump), crossbar gets the published messages from the other tasks within a few dozen milliseconds

the long running task is doing a chain of DNS requests using async coroutines with yielded calls and internal yields. the actual run time is about 2-4 seconds but i stuffed a yielded asyncio.sleep(60) at the end of it

0 Likes

#4

    i considered this, but discarded this line of thought for two reasons.

1) this 60 second task yields in several places
2) running crossbar with loglevel debug (and confirmed by tcpdump),
crossbar gets the published messages from the other tasks within a few
dozen milliseconds

the long running task is doing a chain of DNS requests using async
coroutines with yielded calls and internal yields. the actual run time
is about 2-4 seconds but i stuffed a yielded asyncio.sleep(60) at the
end of it

yielding each step individually essentially serializes the performed operations - they won't run concurrently.

···

Am 03.12.2015 um 19:30 schrieb David Ford:

--
You received this message because you are subscribed to the Google
Groups "Crossbar" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to crossbario+...@googlegroups.com
<mailto:crossbario+...@googlegroups.com>.
To post to this group, send email to cross...@googlegroups.com
<mailto:cross...@googlegroups.com>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/crossbario/1e40eecc-c0be-4910-a337-5b40687cafec%40googlegroups.com
<https://groups.google.com/d/msgid/crossbario/1e40eecc-c0be-4910-a337-5b40687cafec%40googlegroups.com?utm_medium=email&utm_source=footer>.
For more options, visit https://groups.google.com/d/optout.

0 Likes

#5

ok, given the following structure of these 4 functions, let’s presume that when using psycopg2 asynchronously, aio has yielded my best (it works) async wrapper for database lookups. twisted’s adbapi seems to have vanished, it doesn’t get installed when the package is built and installed and references for it missing are scarce at best. further, i must mix autobahn.twisted.wamp with asyncio. using ApplicationSession from autobahn.asyncio.wamp fails to finish startup. see https://github.com/crossbario/crossbar/issues/534.

so therefore:

  • i must use autobahn.twisted.wamp.ApplicationSession and i use asyncio for aio and psycopg2
  • therefore, i mix @asyncio.coroutine with @inlineCallbacks
  • my sql queries are fast
  • my pythonwhois takes between 1 and several seconds
  • my dnspython queries take between 2 and several seconds

foo1 and foo2 run separate fast sql queries using aio and psycopg2

foo3 does a pythonwhois lookup

foo4 does several dozen DNS queries that can range from milliseconds to several seconds.

all of these functions are needed when a particular web page is opened to populate numerous DIVs with data.

how should i restructure this so each DIV is populated as fast as possible with as little impact to other functions as possible?

thank you very much in advance for helping,

-d

···

On Wednesday, December 2, 2015 at 10:31:32 PM UTC, David Ford wrote:

@Tobias? :slight_smile:
how can i, from clientA, trigger a function on clientB, whereby clientB runs several functions and publishes to several endpoints, and each publish() is forwarded immediately through crossbar instead of waiting until clientB has finished performing all functions?
e.g.
browser js clientA does either an rpc call or publish to com.example.trigger
python backend clientB, having either a matching rpc function or subscription callback for com.example.trigger (doesn’t matter which i use) then runs callback(). callback will look like the following:

class Butterfly(ApplicationSession):

@inlineCallbacks
def __yield(self, uri, data):
logger.info(log some stuff)
yield self.publish(uri, data)

@wamp.register(com.example.trigger)

def callback(self, opts):
@asyncio.coroutine
def foo1(opts):
results=compute() # takes <1 second
__yield(com.example.something1, results)
@asyncio.coroutine
def foo2(opts):
results=compute() # takes <1 second
__yield(com.example.something2, results)
@asyncio.coroutine
def foo3(opts):
results=compute() # takes <1 second
__yield(com.example.something3, results)
@asyncio.coroutine
def foo4(opts):
results=compute() # takes 60 seconds
__yield(com.example.something4, results)

loop = asyncio.get_event.loop()

tasks = [asyncio.Task(foo1(opts)),
asyncio.Task(foo2(opts)),
asyncio.Task(foo3(opts)),
asyncio.Task(foo4(opts))]

loop.run_until_complete(asyncio.wait(tasks))

at present, crossbar router does see published data immediately for the first three functions foo1(), foo2(), and foo3(); but it waits until after foo4() has completed before any of the published data is forwarded to the browser client. --loglevel debug and tcpdump confirms it.

how do i make crossbar forward the messages immediately to subscribed clients using a single trigger?

0 Likes

#6

There is no need (and it doesnt make sense) to mix tx anf asyncio.

Simply dont yield things you want to run concurrently, but usr Deferreds (tx) or Futures (aio).

···

Sent from Mobile (Google Nexus 5)

Am 05.12.2015 6:19 vorm. schrieb “David Ford” firefig...@gmail.com:

ok, given the following structure of these 4 functions, let’s presume that when using psycopg2 asynchronously, aio has yielded my best (it works) async wrapper for database lookups. twisted’s adbapi seems to have vanished, it doesn’t get installed when the package is built and installed and references for it missing are scarce at best. further, i must mix autobahn.twisted.wamp with asyncio. using ApplicationSession from autobahn.asyncio.wamp fails to finish startup. see https://github.com/crossbario/crossbar/issues/534.

so therefore:

  • i must use autobahn.twisted.wamp.ApplicationSession and i use asyncio for aio and psycopg2
  • therefore, i mix @asyncio.coroutine with @inlineCallbacks
  • my sql queries are fast
  • my pythonwhois takes between 1 and several seconds
  • my dnspython queries take between 2 and several seconds

foo1 and foo2 run separate fast sql queries using aio and psycopg2

foo3 does a pythonwhois lookup

foo4 does several dozen DNS queries that can range from milliseconds to several seconds.

all of these functions are needed when a particular web page is opened to populate numerous DIVs with data.

how should i restructure this so each DIV is populated as fast as possible with as little impact to other functions as possible?

thank you very much in advance for helping,

-d

On Wednesday, December 2, 2015 at 10:31:32 PM UTC, David Ford wrote:

@Tobias? :slight_smile:
how can i, from clientA, trigger a function on clientB, whereby clientB runs several functions and publishes to several endpoints, and each publish() is forwarded immediately through crossbar instead of waiting until clientB has finished performing all functions?
e.g.
browser js clientA does either an rpc call or publish to com.example.trigger
python backend clientB, having either a matching rpc function or subscription callback for com.example.trigger (doesn’t matter which i use) then runs callback(). callback will look like the following:

class Butterfly(ApplicationSession):

@inlineCallbacks
def __yield(self, uri, data):
logger.info(log some stuff)
yield self.publish(uri, data)

@wamp.register(com.example.trigger)

def callback(self, opts):
@asyncio.coroutine
def foo1(opts):
results=compute() # takes <1 second
__yield(com.example.something1, results)
@asyncio.coroutine
def foo2(opts):
results=compute() # takes <1 second
__yield(com.example.something2, results)
@asyncio.coroutine
def foo3(opts):
results=compute() # takes <1 second
__yield(com.example.something3, results)
@asyncio.coroutine
def foo4(opts):
results=compute() # takes 60 seconds
__yield(com.example.something4, results)

loop = asyncio.get_event.loop()

tasks = [asyncio.Task(foo1(opts)),
asyncio.Task(foo2(opts)),
asyncio.Task(foo3(opts)),
asyncio.Task(foo4(opts))]

loop.run_until_complete(asyncio.wait(tasks))

at present, crossbar router does see published data immediately for the first three functions foo1(), foo2(), and foo3(); but it waits until after foo4() has completed before any of the published data is forwarded to the browser client. --loglevel debug and tcpdump confirms it.

how do i make crossbar forward the messages immediately to subscribed clients using a single trigger?

–

You received this message because you are subscribed to the Google Groups “Crossbar” group.

To unsubscribe from this group and stop receiving emails from it, send an email to crossbario+...@googlegroups.com.

To post to this group, send email to cross...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/crossbario/85acf967-5410-45e2-88b3-094fe1f721a3%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

0 Likes

#7

can’t go pure asyncio as per the bug i reported. crossbar won’t finish starting up. i require asyncio to use aio with psycopg2.

thank you

-d

···

On Sat, Dec 5, 2015 at 9:53 AM, Tobias Oberstein tobias.o...@gmail.com wrote:

There is no need (and it doesnt make sense) to mix tx anf asyncio.

Simply dont yield things you want to run concurrently, but usr Deferreds (tx) or Futures (aio).

Sent from Mobile (Google Nexus 5)

Am 05.12.2015 6:19 vorm. schrieb “David Ford” firefig...@gmail.com:

ok, given the following structure of these 4 functions, let’s presume that when using psycopg2 asynchronously, aio has yielded my best (it works) async wrapper for database lookups. twisted’s adbapi seems to have vanished, it doesn’t get installed when the package is built and installed and references for it missing are scarce at best. further, i must mix autobahn.twisted.wamp with asyncio. using ApplicationSession from autobahn.asyncio.wamp fails to finish startup. see https://github.com/crossbario/crossbar/issues/534.

so therefore:

  • i must use autobahn.twisted.wamp.ApplicationSession and i use asyncio for aio and psycopg2
  • therefore, i mix @asyncio.coroutine with @inlineCallbacks
  • my sql queries are fast
  • my pythonwhois takes between 1 and several seconds
  • my dnspython queries take between 2 and several seconds

foo1 and foo2 run separate fast sql queries using aio and psycopg2

foo3 does a pythonwhois lookup

foo4 does several dozen DNS queries that can range from milliseconds to several seconds.

all of these functions are needed when a particular web page is opened to populate numerous DIVs with data.

how should i restructure this so each DIV is populated as fast as possible with as little impact to other functions as possible?

thank you very much in advance for helping,

-d

On Wednesday, December 2, 2015 at 10:31:32 PM UTC, David Ford wrote:

@Tobias? :slight_smile:
how can i, from clientA, trigger a function on clientB, whereby clientB runs several functions and publishes to several endpoints, and each publish() is forwarded immediately through crossbar instead of waiting until clientB has finished performing all functions?
e.g.
browser js clientA does either an rpc call or publish to com.example.trigger
python backend clientB, having either a matching rpc function or subscription callback for com.example.trigger (doesn’t matter which i use) then runs callback(). callback will look like the following:

class Butterfly(ApplicationSession):

@inlineCallbacks
def __yield(self, uri, data):
logger.info(log some stuff)
yield self.publish(uri, data)

@wamp.register(com.example.trigger)

def callback(self, opts):
@asyncio.coroutine
def foo1(opts):
results=compute() # takes <1 second
__yield(com.example.something1, results)
@asyncio.coroutine
def foo2(opts):
results=compute() # takes <1 second
__yield(com.example.something2, results)
@asyncio.coroutine
def foo3(opts):
results=compute() # takes <1 second
__yield(com.example.something3, results)
@asyncio.coroutine
def foo4(opts):
results=compute() # takes 60 seconds
__yield(com.example.something4, results)

loop = asyncio.get_event.loop()

tasks = [asyncio.Task(foo1(opts)),
asyncio.Task(foo2(opts)),
asyncio.Task(foo3(opts)),
asyncio.Task(foo4(opts))]

loop.run_until_complete(asyncio.wait(tasks))

at present, crossbar router does see published data immediately for the first three functions foo1(), foo2(), and foo3(); but it waits until after foo4() has completed before any of the published data is forwarded to the browser client. --loglevel debug and tcpdump confirms it.

how do i make crossbar forward the messages immediately to subscribed clients using a single trigger?

–

You received this message because you are subscribed to the Google Groups “Crossbar” group.

To unsubscribe from this group and stop receiving emails from it, send an email to crossbario+...@googlegroups.com.

To post to this group, send email to cross...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/crossbario/85acf967-5410-45e2-88b3-094fe1f721a3%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

–

You received this message because you are subscribed to the Google Groups “Crossbar” group.

To unsubscribe from this group and stop receiving emails from it, send an email to crossbario+...@googlegroups.com.

To post to this group, send email to cross...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/crossbario/CABuE%2BY7ZNxCn2yiQbQ1u%3DvrbrATA8Rfw_kDC_Y_KLboR70sp6w%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

–

Gay/FireRescue/Geek in 33484, USA
It’s the ideals of Linux and Open Source that are amazing, it embodies what WE want, not what is marketed

0 Likes

#8

Have you considered using txpostgres (https://pypi.python.org/pypi/txpostgres) instead of aio? It is the twisted equivalent of aio.

···

On Sunday, December 6, 2015 at 2:37:58 PM UTC-6, David Ford wrote:

can’t go pure asyncio as per the bug i reported. crossbar won’t finish starting up. i require asyncio to use aio with psycopg2.

thank you

-d

On Sat, Dec 5, 2015 at 9:53 AM, Tobias Oberstein tobia...@gmail.com wrote:

There is no need (and it doesnt make sense) to mix tx anf asyncio.

Simply dont yield things you want to run concurrently, but usr Deferreds (tx) or Futures (aio).

Sent from Mobile (Google Nexus 5)

Am 05.12.2015 6:19 vorm. schrieb “David Ford” firef...@gmail.com:

so therefore:

  • i must use autobahn.twisted.wamp.ApplicationSession and i use asyncio for aio and psycopg2
  • therefore, i mix @asyncio.coroutine with @inlineCallbacks
  • my sql queries are fast
  • my pythonwhois takes between 1 and several seconds
  • my dnspython queries take between 2 and several seconds

foo1 and foo2 run separate fast sql queries using aio and psycopg2

foo3 does a pythonwhois lookup

foo4 does several dozen DNS queries that can range from milliseconds to several seconds.

all of these functions are needed when a particular web page is opened to populate numerous DIVs with data.

how should i restructure this so each DIV is populated as fast as possible with as little impact to other functions as possible?

thank you very much in advance for helping,

-d

On Wednesday, December 2, 2015 at 10:31:32 PM UTC, David Ford wrote:

@Tobias? :slight_smile:
how can i, from clientA, trigger a function on clientB, whereby clientB runs several functions and publishes to several endpoints, and each publish() is forwarded immediately through crossbar instead of waiting until clientB has finished performing all functions?
e.g.
browser js clientA does either an rpc call or publish to com.example.trigger
python backend clientB, having either a matching rpc function or subscription callback for com.example.trigger (doesn’t matter which i use) then runs callback(). callback will look like the following:

class Butterfly(ApplicationSession):

@inlineCallbacks
def __yield(self, uri, data):
logger.info(log some stuff)
yield self.publish(uri, data)

@wamp.register(com.example.trigger)

def callback(self, opts):
@asyncio.coroutine
def foo1(opts):
results=compute() # takes <1 second
__yield(com.example.something1, results)
@asyncio.coroutine
def foo2(opts):
results=compute() # takes <1 second
__yield(com.example.something2, results)
@asyncio.coroutine
def foo3(opts):
results=compute() # takes <1 second
__yield(com.example.something3, results)
@asyncio.coroutine
def foo4(opts):
results=compute() # takes 60 seconds
__yield(com.example.something4, results)

loop = asyncio.get_event.loop()

tasks = [asyncio.Task(foo1(opts)),
asyncio.Task(foo2(opts)),
asyncio.Task(foo3(opts)),
asyncio.Task(foo4(opts))]

loop.run_until_complete(asyncio.wait(tasks))

at present, crossbar router does see published data immediately for the first three functions foo1(), foo2(), and foo3(); but it waits until after foo4() has completed before any of the published data is forwarded to the browser client. --loglevel debug and tcpdump confirms it.

how do i make crossbar forward the messages immediately to subscribed clients using a single trigger?

ok, given the following structure of these 4 functions, let’s presume that when using psycopg2 asynchronously, aio has yielded my best (it works) async wrapper for database lookups. twisted’s adbapi seems to have vanished, it doesn’t get installed when the package is built and installed and references for it missing are scarce at best. further, i must mix autobahn.twisted.wamp with asyncio. using ApplicationSession from autobahn.asyncio.wamp fails to finish startup. see https://github.com/crossbario/crossbar/issues/534.

–

You received this message because you are subscribed to the Google Groups “Crossbar” group.

To unsubscribe from this group and stop receiving emails from it, send an email to crossbario+...@googlegroups.com.

To post to this group, send email to cros...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/crossbario/85acf967-5410-45e2-88b3-094fe1f721a3%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

–

You received this message because you are subscribed to the Google Groups “Crossbar” group.

To unsubscribe from this group and stop receiving emails from it, send an email to crossbario+...@googlegroups.com.

To post to this group, send email to cros...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/crossbario/CABuE%2BY7ZNxCn2yiQbQ1u%3DvrbrATA8Rfw_kDC_Y_KLboR70sp6w%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

–
Gay/FireRescue/Geek in 33484, USA
It’s the ideals of Linux and Open Source that are amazing, it embodies what WE want, not what is marketed

0 Likes