Force close a connection from the JS client

#1

Hi Tobias et al,

It’s been a while since I was active on here. I’m happy to share that my engineering team has migrated from our previous custom pubsub transport layer (Tornado + redis) to Autobahn (AutobahnJS and AutobahnPython) and in the following months we’ve seen a noticeable improvement in performance. Our throughput is at least 3500% greater with Autobahn, measured in messages delivered per second without performance degradation.

I’ve taken notes on a handful of issues, questions, and feedback that I’d like to share in the near future. For now, I wanted to focus on a particular issue I recently discovered.

In the JS client, autobahn.Connection.close doesn’t immediately close the socket when there is an active session but instead sends a GOODBYE request in Session.leave. It only closes the socket once it gets a GOODBYE acknowledgement from the server (relevant code: https://github.com/tavendo/AutobahnJS/blob/v0.9.4/package/lib/connection.js#L351). This leads to several unexpected behaviors:

A) RPC callbacks from calls that were made before the close request could be invoked after connection.close() is called.

B) PubSub subscription callbacks can be invoked after connection.close() is called.

Because there’s no guarantee when the Autobahn server will process the client’s GOODBYE request relative to other messages intended for the client, the client may or may not observe either of these behaviors. Both of these are race conditions and have caused errors in our application. In our case, here’s what is happening:

  1. We call connection.close() if the Autobahn client stops getting responses to its keepalive messages after 30 seconds. We implement these keepalives as RPCs though it would be nice if Autobahn implemented client/server keep-alive itself, as I’m sure other developers would benefit from it.

  2. We then immediately create a new Connection object, call open on it, resubscribe to all relevant topics, and kick off a new round of RPC keepalives.

  3. In some cases the original Autobahn client (the one we called close on in #1) regains connectivity and suddenly receives a whole bunch of RPC callbacks and subscription callbacks. That’s because the server hasn’t yet processed its GOODBYE request. This causes all sorts of havoc for our application since we now have two Autobahn connections (the “zombie” connection in #1 that temporarily came back from the dead then gets terminated by the server, and the new connection we created in #2).

A few questions/comments:

  • Would it be possible to prevent these behaviors in the JS library? It would be nice for the Session to invalidate any incoming messages if self._goodbye_sent is True.

  • At the very least, it would be helpful to mention these race conditions in the documentation so other developers don’t encounter errors from false assumptions like I did.

  • In the meantime, do you have suggestions on the best way to force a hard disconnect to eliminate the possibility of these behaviors? In this case, the client would close its socket immediately without waiting for a GOODBYE acknowledgement. My initial implementation involves calling connection.close() followed by connection._transport.close() but I’m wondering if there’s a better way.

Thanks,

Nick

0 Likes

#2

Hi Nick,

> It's been a while since I was active on here. I'm happy to share that my

engineering team has migrated from our previous custom pubsub transport
layer (Tornado + redis) to Autobahn (AutobahnJS and AutobahnPython) and
in the following months we've seen a noticeable improvement in
performance. Our throughput is at least 3500% greater with Autobahn,
measured in messages delivered per second without performance degradation.

This is awesome! Also: real world user experience like this might be of interest to developers in general - means: probably you have a blog post about this I could share/RT?

I've taken notes on a handful of issues, questions, and feedback that
I'd like to share in the near future. For now, I wanted to focus on a
particular issue I recently discovered.

In the JS client, autobahn.Connection.close doesn't immediately close
the socket when there is an active session but instead sends a GOODBYE
request in Session.leave. It only closes the socket once it gets a
GOODBYE acknowledgement from the server (relevant code:
https://github.com/tavendo/AutobahnJS/blob/v0.9.4/package/lib/connection.js#L351).
This leads to several unexpected behaviors:
A) RPC callbacks from calls that were made before the close request
could be invoked after connection.close() is called.
B) PubSub subscription callbacks can be invoked after connection.close()
is called.

_before_ calling close is still processed.

Because there's no guarantee when the Autobahn server will process the
client's GOODBYE request relative to other messages intended for the

SThe is guarantee: it will not process any messages _after_ having received GOODBYE.

client, the client may or may not observe either of these behaviors.
Both of these are race conditions and have caused errors in our
application. In our case, here's what is happening:
1) We call connection.close() if the Autobahn client stops getting
responses to its keepalive messages after 30 seconds. We implement these
keepalives as RPCs though it would be nice if Autobahn implemented
client/server keep-alive itself, as I'm sure other developers would
benefit from it.

Yeah, IMO this shouldn't be handled at app level, but transport level.

Crossbar.io has configurable WebSocket ping/pong/keepalive/timeout. Checkout the "auto_ping_XXX" options here

http://crossbar.io/docs/WebSocket-Options/

The revised WAMP-over-RawSocket transport also has transport level ping/pong and Crossbar.io will have that too.

2) We then immediately create a new Connection object, call open on it,
resubscribe to all relevant topics, and kick off a new round of RPC
keepalives.
3) In some cases the original Autobahn client (the one we called close
on in #1) regains connectivity and suddenly receives a whole bunch of
RPC callbacks and subscription callbacks. That's because the server
hasn't yet processed its GOODBYE request. This causes all sorts of havoc
for our application since we now have two Autobahn connections (the
"zombie" connection in #1 that temporarily came back from the dead then
gets terminated by the server, and the new connection we created in #2).

A few questions/comments:
- Would it be possible to prevent these behaviors in the JS library? It
would be nice for the Session to invalidate any incoming messages if
self._goodbye_sent is True.

We could add an option to make it behave like above. If you are interested, please file an issue to AutobahnJS.

One open question then is: what about outstanding promises for calls not fired yet? Reject them so the app can react? Or just do nothing with those?

There is a related issue already: https://github.com/tavendo/AutobahnJS/issues/27

- At the very least, it would be helpful to mention these race
conditions in the documentation so other developers don't encounter
errors from false assumptions like I did.

I wouldn't call it a race condition, since it is deterministic behavior: everything _before_ the client starts to close is still processed. Everything after not. But I nevertheless can see your point/issue. So let's address that.

- In the meantime, do you have suggestions on the best way to force a
hard disconnect to eliminate the possibility of these behaviors? In this
case, the client would close its socket immediately without waiting for
a GOODBYE acknowledgement. My initial implementation involves calling
connection.close() followed by connection._transport.close() but I'm
wondering if there's a better way.

Yes, you can just hard close the underyling transport (WebSocket).

Cheers,
/Tobias

···

From my point of view, this is exactly what I'd expect: everything

Thanks,

Nick

--
You received this message because you are subscribed to the Google
Groups "Autobahn" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to autobahnws+...@googlegroups.com
<mailto:autobahnws+...@googlegroups.com>.
To post to this group, send email to autob...@googlegroups.com
<mailto:autob...@googlegroups.com>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/autobahnws/940e4358-ed2d-4a68-a1a4-c509fe295d34%40googlegroups.com
<https://groups.google.com/d/msgid/autobahnws/940e4358-ed2d-4a68-a1a4-c509fe295d34%40googlegroups.com?utm_medium=email&utm_source=footer>.
For more options, visit https://groups.google.com/d/optout.

0 Likes