Django ORM migration

#1

Dear all,

I am currently considering in moving a website form Django to Autobahn, due to the necessity of realtime communications. Currently I am using:

  1. JSON_RPC over HTTPS through RPC4Django,

  2. Twisted-AMP (RPC-like) for providing lower delay message exchange services,

  3. and pusher.com (PUB/SUB) for realtime notifications from the server.

The system is working but I would like to integrate the whole 3 services within a single communications architecture; a task for which I therefore require a communications framework that supports RPC and PUB/SUB. Here is where autobahn+crossbar comes into play through its Wamplets.

I have figured out most of the problems but I have all the models already developed for Django ORM. I know that the Django ORM is blocking and, therefore, it may cause troubles with Autobahn.

  1. Would I have to migrate all the models or can I use Django’s ORM from within a Wamplet?

  2. What ORM do you recommend to be used with Autobahn Wamplets?

Thanks, Ricardo.

0 Likes

#2

There are currently non blocking ORM on the Python market. It will change with async/await being introduced in Python 3.5, because descriptors will be able to implement await_ and we can expect SQLAchemy to be usable transparently in the next year or so.

But in the mean time, we have to use the current toolbox.

You can keep using your Django models. The Django ORM works outside of django just fine if you use settings.configure():

http://stackoverflow.com/questions/2180415/using-django-database-layer-outside-of-django

But you must ensure you will not block your event loop. You have 2 solutions :

  • use deferToThread()
  • isolate your ORM calls in separate services

I would go for the second solution. You setup a wamp client with registered remote procedures that call the ORM, and use that in other service. This way blocking code is not affecting the rest of your application.

···

On Wednesday, April 22, 2015 at 1:26:16 AM UTC+2, Ricardo Tubío-Pardavila wrote:

Dear all,

I am currently considering in moving a website form Django to Autobahn, due to the necessity of realtime communications. Currently I am using:

  1. JSON_RPC over HTTPS through RPC4Django,
  1. Twisted-AMP (RPC-like) for providing lower delay message exchange services,
  1. and pusher.com (PUB/SUB) for realtime notifications from the server.

The system is working but I would like to integrate the whole 3 services within a single communications architecture; a task for which I therefore require a communications framework that supports RPC and PUB/SUB. Here is where autobahn+crossbar comes into play through its Wamplets.

I have figured out most of the problems but I have all the models already developed for Django ORM. I know that the Django ORM is blocking and, therefore, it may cause troubles with Autobahn.

  1. Would I have to migrate all the models or can I use Django’s ORM from within a Wamplet?
  1. What ORM do you recommend to be used with Autobahn Wamplets?

Thanks, Ricardo.

0 Likes

#3

Hi, I am having some issues like that also.

And I wonder how it would prevent the blocking code to affect the rest of the application.

With a wamp client dedicated for ORM calls, the client won’t support at least two concurrent calls, right?

I mean, the first call is dispatched, the handler is called, it blocks. At the same time, the second call is dispatched but the handler won’t be called as another handler hasn’t finished.

I’m not sure if it really likes that it works, I’m not familiar with how Twisted internals and Crossbar internals works.

So I may be wrong. But I would like to understand!

So, basically, if you’re right, we could easily wrap SQLAlchemy in a WAMP client, and use it as we would use it in our past web application?

Thanks !

···

Le dimanche 10 mai 2015 11:40:08 UTC+2, Michel Desmoulin a écrit :

  • isolate your ORM calls in separate services

I would go for the second solution. You setup a wamp client with registered remote procedures that call the ORM, and use that in other service. This way blocking code is not affecting the rest of your application.

On Wednesday, April 22, 2015 at 1:26:16 AM UTC+2, Ricardo Tubío-Pardavila wrote:

Dear all,

I am currently considering in moving a website form Django to Autobahn, due to the necessity of realtime communications. Currently I am using:

  1. JSON_RPC over HTTPS through RPC4Django,
  1. Twisted-AMP (RPC-like) for providing lower delay message exchange services,
  1. and pusher.com (PUB/SUB) for realtime notifications from the server.

The system is working but I would like to integrate the whole 3 services within a single communications architecture; a task for which I therefore require a communications framework that supports RPC and PUB/SUB. Here is where autobahn+crossbar comes into play through its Wamplets.

I have figured out most of the problems but I have all the models already developed for Django ORM. I know that the Django ORM is blocking and, therefore, it may cause troubles with Autobahn.

  1. Would I have to migrate all the models or can I use Django’s ORM from within a Wamplet?
  1. What ORM do you recommend to be used with Autobahn Wamplets?

Thanks, Ricardo.

0 Likes

#4

Hi, I am having some issues like that also.

And I wonder how it would prevent the blocking code to affect the rest of the application.

With a wamp client dedicated for ORM calls, the client won’t support at least two concurrent calls, right?

Yes.

I mean, the first call is dispatched, the handler is called, it blocks. At the same time, the second call is dispatched but the handler won’t be called as another handler hasn’t finished.

No. It will block, process the previous one, then handle the next one. Your requests are buffered by your OS, they are not lost if the client don’t process it. They are just processed one by one.

This is already the case with blocking technology. It’s why django setup use multiple threads : because you can only process one call at a time per thread.

I’m not sure if it really likes that it works, I’m not familiar with how Twisted internals and Crossbar internals works.

So I may be wrong. But I would like to understand!

So, basically, if you’re right, we could easily wrap SQLAlchemy in a WAMP client, and use it as we would use it in our past web application?

Thanks !

I would actually make several clients : one service for each “logicial” group of features : authentication, user account, content management, etc. This forces you to think about about separation of concern, balance the load of your app, allow you to move a set of features on another machine if you need to, and hide the implementation behind an API so you can change anything later.

···

On Sunday, May 10, 2015 at 7:59:24 PM UTC+2, Raito Bezarius wrote:

Le dimanche 10 mai 2015 11:40:08 UTC+2, Michel Desmoulin a écrit :

  • isolate your ORM calls in separate services

I would go for the second solution. You setup a wamp client with registered remote procedures that call the ORM, and use that in other service. This way blocking code is not affecting the rest of your application.

On Wednesday, April 22, 2015 at 1:26:16 AM UTC+2, Ricardo Tubío-Pardavila wrote:

Dear all,

I am currently considering in moving a website form Django to Autobahn, due to the necessity of realtime communications. Currently I am using:

  1. JSON_RPC over HTTPS through RPC4Django,
  1. Twisted-AMP (RPC-like) for providing lower delay message exchange services,
  1. and pusher.com (PUB/SUB) for realtime notifications from the server.

The system is working but I would like to integrate the whole 3 services within a single communications architecture; a task for which I therefore require a communications framework that supports RPC and PUB/SUB. Here is where autobahn+crossbar comes into play through its Wamplets.

I have figured out most of the problems but I have all the models already developed for Django ORM. I know that the Django ORM is blocking and, therefore, it may cause troubles with Autobahn.

  1. Would I have to migrate all the models or can I use Django’s ORM from within a Wamplet?
  1. What ORM do you recommend to be used with Autobahn Wamplets?

Thanks, Ricardo.

0 Likes