Application design pattern - sync list on frontend with list on backend

#1

All,

I’m finding that as we develop this new client application using Autobahn JS and Crossbar/Wamp Router, we are using the same design pattern over and over. Namely, we need to have a list on the client (front end) that comes from a list being maintained on the back end. When the client first starts, we need to pull the current list (RPC) but if the backend changes the list data, we need to sync those changes (PUB/SUB). So, the pattern we have been using is one like this:

subscribe (com.example.data.list.on_change) ----> on success ----> call (com.example.data.list.get)

We’ve been calling this the SUB + LIST (subscribe plus list) process. For list, we expect data as an array of objects in the ‘args’ parameter. [ {…}, {…}, {…} ], etc.

The ‘on_change’ topic will send “deltas” to the list which are essentially one of the CRUD actions ([C]reate/add, [U]pdate/modify, [D]elete/remove) followed by a replacement object if C or U: [action, {…} ]. So we end up with this:

list = [ Object, Object, Object ]

on_change = [ action, object_id, Object ]

When you map this onto existing patterns, this is just like using tables in a database. The ‘list’ is like SELECT that returns an array of ‘row’ objects. Each Object is expected to have an ‘id’ that is the unique id for the table/row. The on_change topic is like the INSERT, UPDATE, and DELETE statements of a table where ‘action’ defines which of the 3 actions to perform, object_id is the unique id of the row, and Object is a full row of data to insert or replace in our table.

I foresee some small problems with this design dealing with race conditions, etc. It’s possible that the subscribe succeeds and the client immediately receives an on_change publication. Meanwhile, the backend will generate the list and send it. Thereby, some of the list delta changes might arrive before the list arrives, etc.

I’m using AngularJS on this project and have a ListCollection object that will store these lists on the client front end, subscribe to topics, and manage the list of data for me (works as an AngularJS Service) when the WAMP connection opens. Because of the similarity to essentially having a copy of the backend database on the front end client, I can’t help but think that maybe this problem has been solved before (some kind of DB table sync layer) so I don’t have to reinvent something that already exists. I don’t just want an active-record type object, however, because I don’t always have a database for these objects. Sometimes the data might come from MySQL, sometimes, SQLite, sometimes MongoDB, and sometimes just flat files, etc. So, I’m looking more for a “pattern” to this keep-a-list-in-sync problem rather than just a DB-clone pattern.

Thoughts?

– Dante

0 Likes

#2

Hi Dante!

There is indeed the possibility of a race condition. (Not sure whether and how often this happens in practice, but there’s no guarantee that it can’t happen.)

What we ended up with was a slight addition to your SUB + LIST pattern: a flag “initial_data_received” in the subsciption handler which is initially set to ‘false’ when the subscription is made, and changed to ‘true’ once the initial data from the call has been received. As long as it is ‘false’, and subscription events are discarded by the handler. This at least makes sure that no updates are processed before there is data to update.

Hope this helps a bit.

Any other patterns you’ve come across that are repeated? We’re thinking of adding a code recipes section which would cover things like this.

Regards,

Alex

···

Am Freitag, 20. Februar 2015 18:24:50 UTC+1 schrieb da...@lorenso.com:

All,

I’m finding that as we develop this new client application using Autobahn JS and Crossbar/Wamp Router, we are using the same design pattern over and over. Namely, we need to have a list on the client (front end) that comes from a list being maintained on the back end. When the client first starts, we need to pull the current list (RPC) but if the backend changes the list data, we need to sync those changes (PUB/SUB). So, the pattern we have been using is one like this:

subscribe (com.example.data.list.on_change) ----> on success ----> call (com.example.data.list.get)

We’ve been calling this the SUB + LIST (subscribe plus list) process. For list, we expect data as an array of objects in the ‘args’ parameter. [ {…}, {…}, {…} ], etc.

The ‘on_change’ topic will send “deltas” to the list which are essentially one of the CRUD actions ([C]reate/add, [U]pdate/modify, [D]elete/remove) followed by a replacement object if C or U: [action, {…} ]. So we end up with this:

list = [ Object, Object, Object ]

on_change = [ action, object_id, Object ]

When you map this onto existing patterns, this is just like using tables in a database. The ‘list’ is like SELECT that returns an array of ‘row’ objects. Each Object is expected to have an ‘id’ that is the unique id for the table/row. The on_change topic is like the INSERT, UPDATE, and DELETE statements of a table where ‘action’ defines which of the 3 actions to perform, object_id is the unique id of the row, and Object is a full row of data to insert or replace in our table.

I foresee some small problems with this design dealing with race conditions, etc. It’s possible that the subscribe succeeds and the client immediately receives an on_change publication. Meanwhile, the backend will generate the list and send it. Thereby, some of the list delta changes might arrive before the list arrives, etc.

I’m using AngularJS on this project and have a ListCollection object that will store these lists on the client front end, subscribe to topics, and manage the list of data for me (works as an AngularJS Service) when the WAMP connection opens. Because of the similarity to essentially having a copy of the backend database on the front end client, I can’t help but think that maybe this problem has been solved before (some kind of DB table sync layer) so I don’t have to reinvent something that already exists. I don’t just want an active-record type object, however, because I don’t always have a database for these objects. Sometimes the data might come from MySQL, sometimes, SQLite, sometimes MongoDB, and sometimes just flat files, etc. So, I’m looking more for a “pattern” to this keep-a-list-in-sync problem rather than just a DB-clone pattern.

Thoughts?

– Dante

0 Likes

#3

Even with this method, there is a case where you lost data

Say the case

  1. FE Get the List, and subscribe at the same time

  2. backend generate list [ A, B, C ]

  3. element D is created, and sent to FE, but FE discard because list not yet received

  4. backend finish transmitting [A,B,C]

For me, to keep synchronysation, you need

  1. FE Subscribe, and wait for subscribe ack

  2. FE get the list

  3. if get update before list is received, you need to keep them for post processing.

  4. FE finish receive the full list init state. Needs to merge the update received in between with the init state.

We have same problem with buildbot nine. You can see it working here: http://nine.buildbot.net/#/builders/6

For now, buildbot is implemented with REST + SSE, but the problem is pretty similar as it will be for wamp.

The problem I have with this design is that there is a lot of serialization. you need a REST call for subscribe then a REST call for getting the list, this slows down a lot when there are lot of latency between server and client

This is why I think with wamp I can accelerate the problem. I wanted to use progressive call in order to make this work better with the race conditions. The first YIELD will send the whole list, and the next YIELDs updates to the list.

Pierre

Le Sat Feb 21 2015 at 15:05:11, Alexander Gödde alexande...@googlemail.com a écrit :

···

Hi Dante!

There is indeed the possibility of a race condition. (Not sure whether and how often this happens in practice, but there’s no guarantee that it can’t happen.)

What we ended up with was a slight addition to your SUB + LIST pattern: a flag “initial_data_received” in the subsciption handler which is initially set to ‘false’ when the subscription is made, and changed to ‘true’ once the initial data from the call has been received. As long as it is ‘false’, and subscription events are discarded by the handler. This at least makes sure that no updates are processed before there is data to update.

Hope this helps a bit.

Any other patterns you’ve come across that are repeated? We’re thinking of adding a code recipes section which would cover things like this.

Regards,

Alex

Am Freitag, 20. Februar 2015 18:24:50 UTC+1 schrieb da...@lorenso.com:

All,

I’m finding that as we develop this new client application using Autobahn JS and Crossbar/Wamp Router, we are using the same design pattern over and over. Namely, we need to have a list on the client (front end) that comes from a list being maintained on the back end. When the client first starts, we need to pull the current list (RPC) but if the backend changes the list data, we need to sync those changes (PUB/SUB). So, the pattern we have been using is one like this:

subscribe (com.example.data.list.on_change) ----> on success ----> call (com.example.data.list.get)

We’ve been calling this the SUB + LIST (subscribe plus list) process. For list, we expect data as an array of objects in the ‘args’ parameter. [ {…}, {…}, {…} ], etc.

The ‘on_change’ topic will send “deltas” to the list which are essentially one of the CRUD actions ([C]reate/add, [U]pdate/modify, [D]elete/remove) followed by a replacement object if C or U: [action, {…} ]. So we end up with this:

list = [ Object, Object, Object ]

on_change = [ action, object_id, Object ]

When you map this onto existing patterns, this is just like using tables in a database. The ‘list’ is like SELECT that returns an array of ‘row’ objects. Each Object is expected to have an ‘id’ that is the unique id for the table/row. The on_change topic is like the INSERT, UPDATE, and DELETE statements of a table where ‘action’ defines which of the 3 actions to perform, object_id is the unique id of the row, and Object is a full row of data to insert or replace in our table.

I foresee some small problems with this design dealing with race conditions, etc. It’s possible that the subscribe succeeds and the client immediately receives an on_change publication. Meanwhile, the backend will generate the list and send it. Thereby, some of the list delta changes might arrive before the list arrives, etc.

I’m using AngularJS on this project and have a ListCollection object that will store these lists on the client front end, subscribe to topics, and manage the list of data for me (works as an AngularJS Service) when the WAMP connection opens. Because of the similarity to essentially having a copy of the backend database on the front end client, I can’t help but think that maybe this problem has been solved before (some kind of DB table sync layer) so I don’t have to reinvent something that already exists. I don’t just want an active-record type object, however, because I don’t always have a database for these objects. Sometimes the data might come from MySQL, sometimes, SQLite, sometimes MongoDB, and sometimes just flat files, etc. So, I’m looking more for a “pattern” to this keep-a-list-in-sync problem rather than just a DB-clone pattern.

Thoughts?

– Dante

You received this message because you are subscribed to the Google Groups “Autobahn” group.

To unsubscribe from this group and stop receiving emails from it, send an email to autobahnws+...@googlegroups.com.

To post to this group, send email to autob...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/autobahnws/c94cad2c-bf9a-4670-9ca3-21ff145c8062%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

0 Likes

#4

Thanks for raising this discussion, I’m sure the community (well, at least I) will learn from your findings.

Can you please explain why you do not use a LIST + SUB pattern instead of your SUB + LIST?. Or, if this should be very obvious, some pointer to documentation or Google terms I should’ve used…

···

Op vrijdag 20 februari 2015 18:24:50 UTC+1 schreef da...@lorenso.com:

All,

I’m finding that as we develop this new client application using Autobahn JS and Crossbar/Wamp Router, we are using the same design pattern over and over. Namely, we need to have a list on the client (front end) that comes from a list being maintained on the back end. When the client first starts, we need to pull the current list (RPC) but if the backend changes the list data, we need to sync those changes (PUB/SUB). So, the pattern we have been using is one like this:

subscribe (com.example.data.list.on_change) ----> on success ----> call (com.example.data.list.get)

We’ve been calling this the SUB + LIST (subscribe plus list) process. For list, we expect data as an array of objects in the ‘args’ parameter. [ {…}, {…}, {…} ], etc.

The ‘on_change’ topic will send “deltas” to the list which are essentially one of the CRUD actions ([C]reate/add, [U]pdate/modify, [D]elete/remove) followed by a replacement object if C or U: [action, {…} ]. So we end up with this:

list = [ Object, Object, Object ]

on_change = [ action, object_id, Object ]

When you map this onto existing patterns, this is just like using tables in a database. The ‘list’ is like SELECT that returns an array of ‘row’ objects. Each Object is expected to have an ‘id’ that is the unique id for the table/row. The on_change topic is like the INSERT, UPDATE, and DELETE statements of a table where ‘action’ defines which of the 3 actions to perform, object_id is the unique id of the row, and Object is a full row of data to insert or replace in our table.

I foresee some small problems with this design dealing with race conditions, etc. It’s possible that the subscribe succeeds and the client immediately receives an on_change publication. Meanwhile, the backend will generate the list and send it. Thereby, some of the list delta changes might arrive before the list arrives, etc.

I’m using AngularJS on this project and have a ListCollection object that will store these lists on the client front end, subscribe to topics, and manage the list of data for me (works as an AngularJS Service) when the WAMP connection opens. Because of the similarity to essentially having a copy of the backend database on the front end client, I can’t help but think that maybe this problem has been solved before (some kind of DB table sync layer) so I don’t have to reinvent something that already exists. I don’t just want an active-record type object, however, because I don’t always have a database for these objects. Sometimes the data might come from MySQL, sometimes, SQLite, sometimes MongoDB, and sometimes just flat files, etc. So, I’m looking more for a “pattern” to this keep-a-list-in-sync problem rather than just a DB-clone pattern.

Thoughts?

– Dante

0 Likes

#5

Hi Pierre,

Yes, there is a race even here … probably infrequent, but there is.

Here is an alternative: have the backend database table contain a “tid” column with a numeric transaction ID per row. Whenever a record is updated, the “tid” is incremented, and the event published will contain the tid. When a new record is inserted, the tid is set to 1. The init call to get the current rows from the table also contains the tid column information. Whenever a record comes into the client (via event handler or via init call result), the record is only inserted if the tid is larger than any existing record for the object. Deleting a record in the database again creates a new tid, but the record is marked as “deleted” - a flag that also needs to be transmitted in events and call results.

Cheers,
/Tobias

···

Am Samstag, 21. Februar 2015 20:10:42 UTC+1 schrieb Pierre Tardy:

Even with this method, there is a case where you lost data

Say the case

  1. FE Get the List, and subscribe at the same time
  1. backend generate list [ A, B, C ]
  1. element D is created, and sent to FE, but FE discard because list not yet received
  1. backend finish transmitting [A,B,C]

For me, to keep synchronysation, you need

  1. FE Subscribe, and wait for subscribe ack
  1. FE get the list
  1. if get update before list is received, you need to keep them for post processing.
  1. FE finish receive the full list init state. Needs to merge the update received in between with the init state.

We have same problem with buildbot nine. You can see it working here: http://nine.buildbot.net/#/builders/6

For now, buildbot is implemented with REST + SSE, but the problem is pretty similar as it will be for wamp.

The problem I have with this design is that there is a lot of serialization. you need a REST call for subscribe then a REST call for getting the list, this slows down a lot when there are lot of latency between server and client

This is why I think with wamp I can accelerate the problem. I wanted to use progressive call in order to make this work better with the race conditions. The first YIELD will send the whole list, and the next YIELDs updates to the list.

Pierre

Le Sat Feb 21 2015 at 15:05:11, Alexander Gödde alexander.goedde@googlemail.com a écrit :

Hi Dante!

There is indeed the possibility of a race condition. (Not sure whether and how often this happens in practice, but there’s no guarantee that it can’t happen.)

What we ended up with was a slight addition to your SUB + LIST pattern: a flag “initial_data_received” in the subsciption handler which is initially set to ‘false’ when the subscription is made, and changed to ‘true’ once the initial data from the call has been received. As long as it is ‘false’, and subscription events are discarded by the handler. This at least makes sure that no updates are processed before there is data to update.

Hope this helps a bit.

Any other patterns you’ve come across that are repeated? We’re thinking of adding a code recipes section which would cover things like this.

Regards,

Alex

Am Freitag, 20. Februar 2015 18:24:50 UTC+1 schrieb da...@lorenso.com:

All,

I’m finding that as we develop this new client application using Autobahn JS and Crossbar/Wamp Router, we are using the same design pattern over and over. Namely, we need to have a list on the client (front end) that comes from a list being maintained on the back end. When the client first starts, we need to pull the current list (RPC) but if the backend changes the list data, we need to sync those changes (PUB/SUB). So, the pattern we have been using is one like this:

subscribe (com.example.data.list.on_change) ----> on success ----> call (com.example.data.list.get)

We’ve been calling this the SUB + LIST (subscribe plus list) process. For list, we expect data as an array of objects in the ‘args’ parameter. [ {…}, {…}, {…} ], etc.

The ‘on_change’ topic will send “deltas” to the list which are essentially one of the CRUD actions ([C]reate/add, [U]pdate/modify, [D]elete/remove) followed by a replacement object if C or U: [action, {…} ]. So we end up with this:

list = [ Object, Object, Object ]

on_change = [ action, object_id, Object ]

When you map this onto existing patterns, this is just like using tables in a database. The ‘list’ is like SELECT that returns an array of ‘row’ objects. Each Object is expected to have an ‘id’ that is the unique id for the table/row. The on_change topic is like the INSERT, UPDATE, and DELETE statements of a table where ‘action’ defines which of the 3 actions to perform, object_id is the unique id of the row, and Object is a full row of data to insert or replace in our table.

I foresee some small problems with this design dealing with race conditions, etc. It’s possible that the subscribe succeeds and the client immediately receives an on_change publication. Meanwhile, the backend will generate the list and send it. Thereby, some of the list delta changes might arrive before the list arrives, etc.

I’m using AngularJS on this project and have a ListCollection object that will store these lists on the client front end, subscribe to topics, and manage the list of data for me (works as an AngularJS Service) when the WAMP connection opens. Because of the similarity to essentially having a copy of the backend database on the front end client, I can’t help but think that maybe this problem has been solved before (some kind of DB table sync layer) so I don’t have to reinvent something that already exists. I don’t just want an active-record type object, however, because I don’t always have a database for these objects. Sometimes the data might come from MySQL, sometimes, SQLite, sometimes MongoDB, and sometimes just flat files, etc. So, I’m looking more for a “pattern” to this keep-a-list-in-sync problem rather than just a DB-clone pattern.

Thoughts?

– Dante

You received this message because you are subscribed to the Google Groups “Autobahn” group.

To unsubscribe from this group and stop receiving emails from it, send an email to autobahnws+unsubscribe@googlegroups.com.

To post to this group, send email to autob...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/autobahnws/c94cad2c-bf9a-4670-9ca3-21ff145c8062%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

0 Likes