How to use with sync Python libraries (SQLAlchemy ORM, etc.)

#1

I am developing an real-time, location-based iOS mobile app and a Python based backend. I am looking to choose between using an MQTT broker + Eclipse Paho Python client, or Crossbar.io with the Autobahn libraries.

What puzzles me is that it seems that I’d need to use either asyncio or twisted on the server, and none of them allows me to use blocking libraries, like SQLAlchemy ORM for example.

The Paho client on the other end is a sync client with a simple loop, allowing me to start a few workers and just use all existing Python libs. It is not async, but with a number of workers it would hopefully handle the load well.

Can you help me to understand:

  1. Is it possible to use the Autobahn Python client in any async workflow with SQLAlchemy ORM for example (and all the other usual Python libs)?

  2. If async is not possible, is it possible to use it in a sync way, just like the Paho client? I mean some kind of infinite loop in a normal app, with a couple of workers started on a server.

0 Likes

#2

1. Is it possible to use the Autobahn Python client in any async workflow

    > with SQLAlchemy ORM for example (and all the other usual
    > Python libs)?

You can use something like 'Gino' package
( https://pypi.python.org/pypi/gino/0.4.1 ) to use the SQLAlchemy sql
layer + a simplyfied ORM layer with an async database driver like
asyncpg. Unfortunately the standard SA's ORM isn't async
compatible.

What are "all the other usual python libs"?

    > 2. If async is not possible, is it possible to use it in a sync way, just
    > like the Paho client? I mean some kind of infinite loop in a normal app,
    > with a couple of workers started on a server.

I don't know what you mean... maybe you can try to better explain
yourself? Usually in an asyncio application all the stuff to be executed
is in some way codede as a task for the event loop. It's possible to run
synchronous tasks using an executor
https://docs.python.org/3/library/asyncio-eventloop.html#executor

0 Likes

#3

Adding to Alberto response, there is also the peewee ORM with an asyncio extension.
Peewee is really great to use, very close to native SQL.

You can use executors (Thread or Processes) to run CPU bound work.

I usually define 2 functions: defer_to_threads and defer_to_processes, that wraps the run_in_executor to ease future coding (taking kwargs parameters as well)

async def defer_to_thread(func, *args, **kwargs):

loop = asyncio.get_event_loop()

result = await loop.run_in_executor(THREAD_EXECUTOR, functools.partial(func, *args, **kwargs))

return result

``

You could even define a “defer_to_celery” to delegate the CPU intensive tasks to a celery-driven cluster.

Advantages of WAMP: RPC comes in (in addition to pub/sub) while MQTT seems to only provides pub/sub natively + very good JS integration for browser & hybride apps (with modules for famous frameworks).

Advantages of MQTT: battle tested.

···

Le jeudi 31 août 2017 12:19:11 UTC+2, Zsolt Ero a écrit :

I am developing an real-time, location-based iOS mobile app and a Python based backend. I am looking to choose between using an MQTT broker + Eclipse Paho Python client, or Crossbar.io with the Autobahn libraries.

What puzzles me is that it seems that I’d need to use either asyncio or twisted on the server, and none of them allows me to use blocking libraries, like SQLAlchemy ORM for example.

The Paho client on the other end is a sync client with a simple loop, allowing me to start a few workers and just use all existing Python libs. It is not async, but with a number of workers it would hopefully handle the load well.

Can you help me to understand:

  1. Is it possible to use the Autobahn Python client in any async workflow with SQLAlchemy ORM for example (and all the other usual Python libs)?
  1. If async is not possible, is it possible to use it in a sync way, just like the Paho client? I mean some kind of infinite loop in a normal app, with a couple of workers started on a server.
0 Likes

#4

What are "all the other usual python libs"?

I mean just generally any Python lib out there is not going to be
async compatible, unless specially written for async, isn't it? If
your project has about 20 dependencies, and each of them has a chance
of blocking async then it quite probably won't work, will it?

If you are interested, here is a list of 93 modules I'm using in a
current project: alembic appnope awesome-slugify awscli
backports.shutil-get-terminal-size bcrypt bleach boto3 botocore cffi
colorama contextlib2 decorator dnspython docutils email-validator
enum34 fastkml futures geoip2 geojson gpxpy gunicorn html2text
html5lib huey hupper idna infinity intervals ipaddress
ipython-genutils ipython jinja2 jmespath lxml mako markupsafe
marrow.mailer marrow.util maxminddb passlib paste pastedeploy pathlib2
pexpect pickleshare prompt-toolkit psycopg2 ptyprocess pyasn1
pycparser pygeoif pygments pyramid-ipython pyramid-jinja2
pyramid-redis-sessions pyramid-tm pyramid python-dateutil
python-editor pyyaml raven redis regex repoze.lru requests rsa
s3transfer scandir shapely simplegeneric simplejson six
sqlalchemy-utils sqlalchemy traitlets transaction translationstring
ua-parser unidecode validators venusian waitress wcwidth webencodings
webob wtforms-alchemy wtforms-components wtforms zope.deprecation
zope.interface zope.sqlalchemy

Even without sqlalchemy, it simply not possible that this would work in asyncio.

When I was referring to sync mode, I was talking about the simple,
blocking concept of the Paho MQTT client, like in this example:
https://github.com/eclipse/paho.mqtt.python/blob/master/examples/client_sub.py

Afaik, this code will just block all other messages and queue them at
the broker, until it can receive the new message. The good thing about
it is that this way it can just work with all the existing Python
infrastructure.

0 Likes

#5

What are "all the other usual python libs"?
    
    > I mean just generally any Python lib out there is not going to be
    > async compatible, unless specially written for async, isn't it? If
    > your project has about 20 dependencies, and each of them has a chance
    > of blocking async then it quite probably won't work, will it?

If these libraries do something computationally intensive, it will block
the interpreter in with both an async or sync architecture ....

The main difference between the two architectures is how tasks that
involve IO are being managed. Usually libraries using different
architecture models aren't directly inter-operable

    > If you are interested, here is a list of 93 modules I'm using in a
    > current project: alembic appnope awesome-slugify awscli
    > backports.shutil-get-terminal-size bcrypt bleach boto3 botocore cffi
    > colorama contextlib2 decorator dnspython docutils email-validator
    > enum34 fastkml futures geoip2 geojson gpxpy gunicorn html2text
    > html5lib huey hupper idna infinity intervals ipaddress
    > ipython-genutils ipython jinja2 jmespath lxml mako markupsafe
    > marrow.mailer marrow.util maxminddb passlib paste pastedeploy pathlib2
    > pexpect pickleshare prompt-toolkit psycopg2 ptyprocess pyasn1
    > pycparser pygeoif pygments pyramid-ipython pyramid-jinja2
    > pyramid-redis-sessions pyramid-tm pyramid python-dateutil
    > python-editor pyyaml raven redis regex repoze.lru requests rsa
    > s3transfer scandir shapely simplegeneric simplejson six
    > sqlalchemy-utils sqlalchemy traitlets transaction translationstring
    > ua-parser unidecode validators venusian waitress wcwidth webencodings
    > webob wtforms-alchemy wtforms-components wtforms zope.deprecation
    > zope.interface zope.sqlalchemy

    > Even without sqlalchemy, it simply not possible that this would work in asyncio.

Here the main remaining issue is with pyramid, but there are some
libraries that try to let you use pyramid and asyncio... have you tried
them or if you don't specifically need to use pyramid jinja can be used
with aiohttp

    > When I was referring to sync mode, I was talking about the simple,
    > blocking concept of the Paho MQTT client, like in this example:
    > https://github.com/eclipse/paho.mqtt.python/blob/master/examples/client_sub.py

    > Afaik, this code will just block all other messages and queue them at
    > the broker, until it can receive the new message. The good thing about
    > it is that this way it can just work with all the existing Python
    > infrastructure.

you think? looking at that "mqttc.loop_forever()" last line it seems
that in order for it to work together with other server frameworks like
pyramid it will need at least some integration code.

0 Likes

#6

I would split the architecture into a Pyramid based WSGI server and
Autobahn or MQTT client based workers, so as the workers wouldn't load
Pyramid or related code. They would run in different ports as totally
different processes, so no problem there. "Integration" would be the
shared PostgreSQL and Redis DB.

I found an article which explains mqttc.loop_forever():
http://www.steves-internet-guide.com/loop-python-mqtt-client/
From that it seems to me that all my sync code can be used, but of
course it'll be blocking.

Do you mean that with asyncio I could also use all my code, just also
with blocking?

···

On 31 August 2017 at 18:41, Alberto Berti <alb...@metapensiero.it> wrote:

    >> What are "all the other usual python libs"?

    > I mean just generally any Python lib out there is not going to be
    > async compatible, unless specially written for async, isn't it? If
    > your project has about 20 dependencies, and each of them has a chance
    > of blocking async then it quite probably won't work, will it?

If these libraries do something computationally intensive, it will block
the interpreter in with both an async or sync architecture ....

The main difference between the two architectures is how tasks that
involve IO are being managed. Usually libraries using different
architecture models aren't directly inter-operable

    > If you are interested, here is a list of 93 modules I'm using in a
    > current project: alembic appnope awesome-slugify awscli
    > backports.shutil-get-terminal-size bcrypt bleach boto3 botocore cffi
    > colorama contextlib2 decorator dnspython docutils email-validator
    > enum34 fastkml futures geoip2 geojson gpxpy gunicorn html2text
    > html5lib huey hupper idna infinity intervals ipaddress
    > ipython-genutils ipython jinja2 jmespath lxml mako markupsafe
    > marrow.mailer marrow.util maxminddb passlib paste pastedeploy pathlib2
    > pexpect pickleshare prompt-toolkit psycopg2 ptyprocess pyasn1
    > pycparser pygeoif pygments pyramid-ipython pyramid-jinja2
    > pyramid-redis-sessions pyramid-tm pyramid python-dateutil
    > python-editor pyyaml raven redis regex repoze.lru requests rsa
    > s3transfer scandir shapely simplegeneric simplejson six
    > sqlalchemy-utils sqlalchemy traitlets transaction translationstring
    > ua-parser unidecode validators venusian waitress wcwidth webencodings
    > webob wtforms-alchemy wtforms-components wtforms zope.deprecation
    > zope.interface zope.sqlalchemy

    > Even without sqlalchemy, it simply not possible that this would work in asyncio.

Here the main remaining issue is with pyramid, but there are some
libraries that try to let you use pyramid and asyncio... have you tried
them or if you don't specifically need to use pyramid jinja can be used
with aiohttp

    > When I was referring to sync mode, I was talking about the simple,
    > blocking concept of the Paho MQTT client, like in this example:
    > https://github.com/eclipse/paho.mqtt.python/blob/master/examples/client_sub.py

    > Afaik, this code will just block all other messages and queue them at
    > the broker, until it can receive the new message. The good thing about
    > it is that this way it can just work with all the existing Python
    > infrastructure.

you think? looking at that "mqttc.loop_forever()" last line it seems
that in order for it to work together with other server frameworks like
pyramid it will need at least some integration code.

--
You received this message because you are subscribed to a topic in the Google Groups "Crossbar" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/crossbario/aw6DpC7B7nM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to crossbario+...@googlegroups.com.
To post to this group, send email to cross...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/crossbario/87pobbbsfm.fsf%40ender.lizardnet.
For more options, visit https://groups.google.com/d/optout.

0 Likes

#7

Contributing to this thread a little late but you can probably leverage the component-based style of developing WAMP applications in conjunction with the thread/process executor system provided by asyncio.

A general scheme for doing this might be something as follows:

SQLAlchemyDataProviderComponent: This component exposes RPC end-points for executing raw SQL and returning the resulting data. It makes use of a thread executor in conjunction with SQLAlchemy’s scoped session system to execute queries “asynchronously” via the mechanism of threading in conjunction with connection pooling. As this component executes raw SQL it should not be exposes to any components that you do not trust, it is essentially for internal usage only.

SQLAlchemDataConsumerComponent: This component provides a base with some async utility functions that other components can extend to make use of said functions. The idea behind these async utility functions is that they accept a SQLAlchemy query and process it into raw SQL. They then dispatch this to the provider component via a standard WAMP RPC call and then process the returned results back into SQLAlchemy objects.

Obviously this is a somewhat complex system to implement but in theory it should work.

0 Likes