Subscribing to large numbers of topics?

#1

Hi folks, I’m a (fairly) new user of Autobahn, and I was wondering if I could get some advice on dealing with large numbers of subscriptions.

Imagine an application that has a ‘search’ feature - such as an email app, bulletin board or a social network. Now suppose you want live updates on the search results - that is, if someone adds a comment to a post, or makes some mutation to one of the objects in the search result set, you’d want to publish that so that other users get notified.

So how do we set up a subscription topic that represents these search results?

The brute force approach is to have a separate topic for each item, something like ‘posts.update.’. However, that means that if you have 100 items in your search results, you have to subscribe to 100 topics. So my first question is, is that likely to be a problem, performance-wise? (There’s also the issue that subscription is async, so you’d have to wait for 100 promises, and manage 100 subscription handles. And when you start talking multiple async operations, there’s race conditions to consider - what happens if someone mutates a bunch of objects at once, and some subscriptions have completed the handshake while others are still pending?)

Note that you can’t use topic wildcards in this case because the set of search results is an arbitrary set - listening for ‘posts.update.*’ is going to give you every post, not just the ones in the search results, which is not what we’d want.

A somewhat more graceful approach would be to somehow encode the search terms in the subscription topic. If it’s a simple keyword search, you could have a topic name such as ‘posts.search.my.search.keywords’, but this won’t handle cases like ‘from:ellen to:me’. And it requires the publisher to listen for topic creation / delete events so as to know what search terms are actively being subscribed to.

So I’m curious to know how folks have approached these kinds of issues.

– Talin

0 Likes

#2

Perhaps there’s a different, more appropriate forum for questions like these? I didn’t want to ask this on SO because they like clear-cut answers whereas this requires more of a judgment call.

···

On Monday, June 6, 2016 at 3:14:19 PM UTC-7, Talin wrote:

Hi folks, I’m a (fairly) new user of Autobahn, and I was wondering if I could get some advice on dealing with large numbers of subscriptions.

Imagine an application that has a ‘search’ feature - such as an email app, bulletin board or a social network. Now suppose you want live updates on the search results - that is, if someone adds a comment to a post, or makes some mutation to one of the objects in the search result set, you’d want to publish that so that other users get notified.

So how do we set up a subscription topic that represents these search results?

The brute force approach is to have a separate topic for each item, something like ‘posts.update.’. However, that means that if you have 100 items in your search results, you have to subscribe to 100 topics. So my first question is, is that likely to be a problem, performance-wise? (There’s also the issue that subscription is async, so you’d have to wait for 100 promises, and manage 100 subscription handles. And when you start talking multiple async operations, there’s race conditions to consider - what happens if someone mutates a bunch of objects at once, and some subscriptions have completed the handshake while others are still pending?)

Note that you can’t use topic wildcards in this case because the set of search results is an arbitrary set - listening for ‘posts.update.*’ is going to give you every post, not just the ones in the search results, which is not what we’d want.

A somewhat more graceful approach would be to somehow encode the search terms in the subscription topic. If it’s a simple keyword search, you could have a topic name such as ‘posts.search.my.search.keywords’, but this won’t handle cases like ‘from:ellen to:me’. And it requires the publisher to listen for topic creation / delete events so as to know what search terms are actively being subscribed to.

So I’m curious to know how folks have approached these kinds of issues.

– Talin

0 Likes

#3

Hi Talin,

maybe my googling-foo is to blame, but normally I’m not interested in more than the top 5 to 10 results from a search. For this amount of topics, I’d say your brute force approach should suffice.

As J.W. von Göthe wrote:

in der Beschränkung zeigt sich erst der Meister
but I’m curious for your use case in which really large number of subscriptions are needed.

···

Op dinsdag 7 juni 2016 00:14:19 UTC+2 schreef Talin:

Hi folks, I’m a (fairly) new user of Autobahn, and I was wondering if I could get some advice on dealing with large numbers of subscriptions.

Imagine an application that has a ‘search’ feature - such as an email app, bulletin board or a social network. Now suppose you want live updates on the search results - that is, if someone adds a comment to a post, or makes some mutation to one of the objects in the search result set, you’d want to publish that so that other users get notified.

So how do we set up a subscription topic that represents these search results?

The brute force approach is to have a separate topic for each item, something like ‘posts.update.’. However, that means that if you have 100 items in your search results, you have to subscribe to 100 topics. So my first question is, is that likely to be a problem, performance-wise? (There’s also the issue that subscription is async, so you’d have to wait for 100 promises, and manage 100 subscription handles. And when you start talking multiple async operations, there’s race conditions to consider - what happens if someone mutates a bunch of objects at once, and some subscriptions have completed the handshake while others are still pending?)

Note that you can’t use topic wildcards in this case because the set of search results is an arbitrary set - listening for ‘posts.update.*’ is going to give you every post, not just the ones in the search results, which is not what we’d want.

A somewhat more graceful approach would be to somehow encode the search terms in the subscription topic. If it’s a simple keyword search, you could have a topic name such as ‘posts.search.my.search.keywords’, but this won’t handle cases like ‘from:ellen to:me’. And it requires the publisher to listen for topic creation / delete events so as to know what search terms are actively being subscribed to.

So I’m curious to know how folks have approached these kinds of issues.

– Talin

0 Likes

#4

Hi Roger, thanks for the interest :slight_smile:

The objects that I’m subscribing to aren’t actually topics - the application I’m working on is a cloud-based 3d animation tool - however, I used topics and posts as an example because I thought folks would be more familiar with that.

Probably a better example of what I am talking about is Google Drive, where you have folders full of documents of various types (docs, spreadsheets, drawings, etc). Typically you have a view of a folder which contains potentially large numbers of documents. Creating a WAMP topic for this case is fairly simple, it would be something like ‘folder.’. The server would publish any mutations to objects contained in that folder.

However, Google Docs also supports searching for docs, and the result is a folder-like view which is defined by the current search keywords. In this case, you can’t really represent the search criteria as topic, so the only recourse is to subscribe to all of the individual results of the search.

···

On Wednesday, June 8, 2016 at 11:25:39 AM UTC-7, Roger Erens wrote:

Hi Talin,

maybe my googling-foo is to blame, but normally I’m not interested in more than the top 5 to 10 results from a search. For this amount of topics, I’d say your brute force approach should suffice.

As J.W. von Göthe wrote:

in der Beschränkung zeigt sich erst der Meister
but I’m curious for your use case in which really large number of subscriptions are needed.

Op dinsdag 7 juni 2016 00:14:19 UTC+2 schreef Talin:

Hi folks, I’m a (fairly) new user of Autobahn, and I was wondering if I could get some advice on dealing with large numbers of subscriptions.

Imagine an application that has a ‘search’ feature - such as an email app, bulletin board or a social network. Now suppose you want live updates on the search results - that is, if someone adds a comment to a post, or makes some mutation to one of the objects in the search result set, you’d want to publish that so that other users get notified.

So how do we set up a subscription topic that represents these search results?

The brute force approach is to have a separate topic for each item, something like ‘posts.update.’. However, that means that if you have 100 items in your search results, you have to subscribe to 100 topics. So my first question is, is that likely to be a problem, performance-wise? (There’s also the issue that subscription is async, so you’d have to wait for 100 promises, and manage 100 subscription handles. And when you start talking multiple async operations, there’s race conditions to consider - what happens if someone mutates a bunch of objects at once, and some subscriptions have completed the handshake while others are still pending?)

Note that you can’t use topic wildcards in this case because the set of search results is an arbitrary set - listening for ‘posts.update.*’ is going to give you every post, not just the ones in the search results, which is not what we’d want.

A somewhat more graceful approach would be to somehow encode the search terms in the subscription topic. If it’s a simple keyword search, you could have a topic name such as ‘posts.search.my.search.keywords’, but this won’t handle cases like ‘from:ellen to:me’. And it requires the publisher to listen for topic creation / delete events so as to know what search terms are actively being subscribed to.

So I’m curious to know how folks have approached these kinds of issues.

– Talin

0 Likes