AutobahnPython native iOS and Safari support

#1

Hi all,

The latest AutobahnPython now supports Hixie-76:

  - server, and also client
  - server supports mixed Hybi and Hixie clients
  - message and streaming API
  - correct, and fully incremental UTF-8 payload validation
  - proper closing handshake (HS)
  - proper timeouts on opening HS, closing HS, server conn. drop
  - works with WAMP (tested with mixed AutobahnJS clients: Hybi+Hixie)

Practically, this means support for: Safari iOS and OSX.

Tested with a years old iPod Touch (iOS 4.2.1) and (old) OSX Safari (5.0.6).

For security reasons, the support is disabled by default!

To enable for servers:

    factory = WebSocketServerFactory("ws://localhost:9000")
    factory.protocol = YourAwesomeProtocol
    factory.setProtocolOptions(allowHixie76 = True)
    listenWS(factory)

To speak Hixie-76 from clients:

    factory = WebSocketClientFactory("ws://localhost:9000")
    factory.protocol = YourAwesomeProtocol
    factory.setProtocolOptions(allowHixie76 = True, version = 0)
    connectWS(factory)

Note: Really, please don't use the client. It's only there because I hate asymmetry, to test/develop the server and probably to write up
some test cases for the test suite.

Cheers,
\Tobias

0 Likes

#2

That’s excellent news! Not that I don’t hate those outdated browsers (opera+safari), but sometimes you just don’t have the freedom to choose against them. My custom solution, running Autobahn and WebSocketServer (https://gist.github.com/512987) in parallel has it’s own annoyances which I’m hoping to elimitate with this new feature. Will be checking it out tonight!

···

On Tuesday, March 27, 2012 6:18:27 PM UTC+2, Tobias Oberstein wrote:

Hi all,
The latest AutobahnPython now supports Hixie-76:

  • server, and also client
    • server supports mixed Hybi and Hixie clients
    • message and streaming API
    • correct, and fully incremental UTF-8 payload validation
    • proper closing handshake (HS)
    • proper timeouts on opening HS, closing HS, server conn. drop
    • works with WAMP (tested with mixed AutobahnJS clients: Hybi+Hixie)

Practically, this means support for: Safari iOS and OSX.

Tested with a years old iPod Touch (iOS 4.2.1) and (old) OSX Safari (5.0.6).

For security reasons, the support is disabled by default!

To enable for servers:

factory = WebSocketServerFactory(“ws://localhost:9000”)
factory.protocol = YourAwesomeProtocol
factory.setProtocolOptions(allowHixie76 = True)
listenWS(factory)

To speak Hixie-76 from clients:

factory = WebSocketClientFactory(“ws://localhost:9000”)
factory.protocol = YourAwesomeProtocol
factory.setProtocolOptions(allowHixie76 = True, version = 0)
connectWS(factory)

Note: Really, please don’t use the client. It’s only there because I
hate asymmetry, to test/develop the server and probably to write up
some test cases for the test suite.

Cheers,
\Tobias

0 Likes

#3

I think there is a regression bug somewhere in the WebSocketServerFactory or the WebSocketServerProtocol.

I haven’t upgraded periodically, and I can’t remember with which Autobahn version I’ve been coding on, but now that I upgraded, most of my clients are not working anymore.

What’s even weirder is that even the examples are not running. They work just fine when using them locally on the same machine, but when I put the server code on a remote machine, then the following works reliable:

All examples work fine with Google Chrome

Firefox 11 works with rpc and pubsub (I think all the WAMP stuff works) but has problems with the echo and broadcast servers. In the broadcast example, when I “publish” a message with Chrome, Firefox does receive it immediately. But if I send a broadcast message from firefox, it’s more of a random issue. Either they arrive (seldomly), or they get buffered and then arrive all at once with a large delay, or never arrive.

Opera and Safari, which are supposed to work with Hixie79, don’t work at all. ("[MyServerProtocol,6,10.10.20.20] unsubscribed peer x.x.x.x:55338 from all topics" is all I get for each pageload)

I think the Android client also does not work correcly. I can’t remember when I upgraded the Android library, I believe it was about two weeks ago, it used to work perfectly, now it doesn’t at all. There are some messages about the Android clients trying to connect, I think there was a message about host being null during the upgrade.

Firewalls are turned off.

All works lovely on localhost. For example, If I have Chrome + Firefox on my local machine, the server on a remote machine, and on that remote machine also has an instance of Opera, then I can publish messages with Opera and Chrome and Firefox will receive them. I can also publish messages on Firefox which Chrome and Opera receive immediately and without loss. All three browsers use the same URL (hostname:8080 to get the pubsub/simple html page).

So, with WAMP Chrome and Firefox are OK, Opera and Safari won’t work on a machine which doesn’t run the server.
With the simpler echo and broadcast examples Chrome always works fine, Firefox fails almost always, and Opera and Safari never work.

Hope this helps.

···

On Tuesday, March 27, 2012 8:14:12 PM UTC+2, Daniel F. wrote:

That’s excellent news! Not that I don’t hate those outdated browsers (opera+safari), but sometimes you just don’t have the freedom to choose against them. My custom solution, running Autobahn and WebSocketServer (https://gist.github.com/512987) in parallel has it’s own annoyances which I’m hoping to elimitate with this new feature. Will be checking it out tonight!

On Tuesday, March 27, 2012 6:18:27 PM UTC+2, Tobias Oberstein wrote:

Hi all,
The latest AutobahnPython now supports Hixie-76:

  • server, and also client
    • server supports mixed Hybi and Hixie clients
    • message and streaming API
    • correct, and fully incremental UTF-8 payload validation
    • proper closing handshake (HS)
    • proper timeouts on opening HS, closing HS, server conn. drop
    • works with WAMP (tested with mixed AutobahnJS clients: Hybi+Hixie)

Practically, this means support for: Safari iOS and OSX.

Tested with a years old iPod Touch (iOS 4.2.1) and (old) OSX Safari (5.0.6).

For security reasons, the support is disabled by default!

To enable for servers:

factory = WebSocketServerFactory(“ws://localhost:9000”)
factory.protocol = YourAwesomeProtocol
factory.setProtocolOptions(allowHixie76 = True)
listenWS(factory)

To speak Hixie-76 from clients:

factory = WebSocketClientFactory(“ws://localhost:9000”)
factory.protocol = YourAwesomeProtocol
factory.setProtocolOptions(allowHixie76 = True, version = 0)
connectWS(factory)

Note: Really, please don’t use the client. It’s only there because I
hate asymmetry, to test/develop the server and probably to write up
some test cases for the test suite.

Cheers,
\Tobias

0 Likes

#4

I’ve been doing more testing on this issue.

This is the setup:

Latest Chrome browser, latest Firefox Portable without any extensions.

One remote machine running broadcast_server.py and one instance of Firefox Portable
One local machine running Chome and Firefox (both Portable and the standard one)

I’ve modified the broadcast_client.html to be more verbose, and added the debug options to the broadcast_server.

The local Chrome instance connects just fine to the server, is able to send a message to the server in the onopen, and receives that message again. The entire connect->send->receive process takes about 30ms.

The remote Firefox, local to the server, connecting to the same instance of the server, is able to do the same in about 200ms

The local Firefox, remote to the server has issues connecting to the server. The server registers this client immediately, onopen gets called about 30-40ms later. onopen then sends the message to the server, but it doesn’t appear in the server log. Firefox says it has sent the message – webSocket.send(message) does return immediately (~1 ms later). Ticks which the server is sending ARE arriving at this Firefox instance. I hit “Send” a couple of more times, and after some seconds I get all of them sent back to me (and any other connected client) from the server, including the first message sent in the onopen callback, all in chronological order. I embedded the javascript timestamp in the message, so that each “Send” press sends a message with the current time. The times do correspond to the moment when I pressed the button, which is no surprise. After I get this entire bundle of initial messages back to me, then the responses to a “Send” press are immediately, then all the problems appear to be fixed. The problem is receiving that initial bunch of responses to the browser’s own sent messages (remember that the ticks – and messages sent from Chrome to the server – do arrive just fine). It is as if the sent messages from Firefox do get buffered somewhere along the pipe, and once this clears up, the problem is solved for that connection. This buffering varies in time. Sometimes it a bit less than one second, sometimes way over 10 seconds, and apparently it is related to how fast you click the “Send” button after websocket connected. But one thing is certain: the message sent in the onopen callback never gets processed immediately by the server, I think that this would be a good starting point for doing the debugging, as this used to work perfectly on all hybi browsers as wel as in the android client. And I do rely so much on that initial message in the onopen callback. Also keep in mind that the stuff built on WAMP does work on a local and a remote Firefox instance.

I’ll now install an older version of Autobahn.

0 Likes

#5

Hi Daniel,

I've done extensive testing the day over. I have none of the issues you see.

You may want to pull the latest commits from the repo which make the
echo/broadcast demos more verbose/convenient. Those commit didnt touch
any library code.

Browser/platforms tested with echo/broadcast demos:

Opera Mobile: Nokia N8, Android Galaxy ACE, Android Galaxy Tab 1
Opera: Win7 32 bit, Win7 64 Bit
iOS: 4.2.1 (iPod Touch)
Firefox 11, 13
Firefox Mobile: Android
Chrome 15, 17

Desktop testing was both localhost and remote. Mobiles obv. all remote.

You could try removing everything AB related from site-packages, and
do a clean easy_install from Pypi. This is what I did for 1 of the
remote machines to verify that nothing stale was left and a clean
install did happen.

So it works for me here,

Tobias

···

Am 28.03.2012 14:14, schrieb Daniel F.:

I've been doing more testing on this issue.

This is the setup:

Latest Chrome browser, latest Firefox Portable without any extensions.

One remote machine running broadcast_server.py and one instance of
Firefox Portable
One local machine running Chome and Firefox (both Portable and the
standard one)

I've modified the broadcast_client.html to be more verbose, and added
the debug options to the broadcast_server.

The local Chrome instance connects just fine to the server, is able to
send a message to the server in the onopen, and receives that message
again. The entire connect->send->receive process takes about 30ms.

The remote Firefox, local to the server, connecting to the same instance
of the server, is able to do the same in about 200ms

The local Firefox, remote to the server has issues connecting to the
server. The server registers this client immediately, onopen gets called
about 30-40ms later. onopen then sends the message to the server, but it
doesn't appear in the server log. Firefox says it has sent the message
-- webSocket.send(message) does return immediately (~1 ms later). Ticks
which the server is sending ARE arriving at this Firefox instance. I hit
"Send" a couple of more times, and after some seconds I get all of them
sent back to me (and any other connected client) from the server,
including the first message sent in the onopen callback, all in
chronological order. I embedded the javascript timestamp in the message,
so that each "Send" press sends a message with the current time. The
times do correspond to the moment when I pressed the button, which is no
surprise. After I get this entire bundle of initial messages back to me,
then the responses to a "Send" press are immediately, then all the
problems appear to be fixed. The problem is receiving that initial bunch
of responses to the browser's own sent messages (remember that the ticks
-- and messages sent from Chrome to the server -- do arrive just fine).
It is as if the sent messages from Firefox do get buffered somewhere
along the pipe, and once this clears up, the problem is solved for that
connection. This buffering varies in time. Sometimes it a bit less than
one second, sometimes way over 10 seconds, and apparently it is related
to how fast you click the "Send" button after websocket connected. But
one thing is certain: the message sent in the onopen callback never gets
processed immediately by the server, I think that this would be a good
starting point for doing the debugging, as this used to work perfectly
on all hybi browsers as wel as in the android client. And I do rely so
much on that initial message in the onopen callback. Also keep in mind
that the stuff built on WAMP does work on a local and a remote Firefox
instance.

I'll now install an older version of Autobahn.

0 Likes

#6

Hi Daniel,

ok, we have the new Autobahn 0.5.1 in production use on autobahn.ws Web site. You can "verify": it replies "Server: AutobahnPython/0.5.1" in HTTP headers, and it speaks Hybi and Hixie.

You can checkout the live demos there using your browsers.

It is working for us from various different Hybi, Hixie, desktop and mobile browsers - at least

http://autobahn.ws/demo/pubsub

works on all. The others have issues still on certain mobiles due to
the use of HTML5 audio and/or jQuery (non-mobile).

There was 1 glitch related to Hixie opening handshake, that only now
appeared when running on _standard_ ports 80 or 443:

https://github.com/tavendo/AutobahnPython/blob/master/autobahn/autobahn/websocket.py#L2687

The fix commit is

https://github.com/tavendo/AutobahnPython/commit/049d3ea979c71714b1504924696586a1e467acee

The reason is that Safari/Opera are _very_ picky about the Sec-WebSocket-Location header returned. When running WS on port 80 (as we do for production), it wants to have _no_ port (no 80) in that header at all. If a port is present, they close the WS immediately
without any notice. Awesome.

Hope that helps,
Tobias

0 Likes

#7

Hi Tobias,

so I tried reversing the situation, installing the server on the workstation and connecting to the workstation from a browser on the server, and that worked flawlessly. I followed your suggestion of isolating the python installation with virtualenv on the server, but the problem persisted. So I used Wireshark on the workstation and noticed that the packets indeed first got collected for a random amount of time on the workstation before getting out on the wire. My belief over the entire day was that I messed up something in twisted, serverside. I double checked the Firewall and there it was. There’s a “feature” called “content filter” which buffers a certain ammount of bytes (even if it’s configured to not filter the content)… It’s even stranger to notice that the same content filter is active on Chrome, but there it doesn’t appear to have any effect. Go figure. Next time I’ll double check the Firewall. The oddest thing is that the content filter buffering is still active even when the Firewall is disabled (but not shut down, what a coding…)!

At least this wasn’t for nothing. I checked out a bit more about the internals of autobahn, it’s examples and got reminded about virtualenv…

So, thanks Tobias for your help.

Daniel

···

On Wednesday, March 28, 2012 7:44:57 PM UTC+2, Tobias Oberstein wrote:

Hi Daniel,
ok, we have the new Autobahn 0.5.1 in production use on autobahn.ws Web
site. You can “verify”: it replies “Server: AutobahnPython/0.5.1” in
HTTP headers, and it speaks Hybi and Hixie.

You can checkout the live demos there using your browsers.

It is working for us from various different Hybi, Hixie, desktop and
mobile browsers - at least

http://autobahn.ws/demo/pubsub

works on all. The others have issues still on certain mobiles due to
the use of HTML5 audio and/or jQuery (non-mobile).

There was 1 glitch related to Hixie opening handshake, that only now
appeared when running on standard ports 80 or 443:

https://github.com/tavendo/AutobahnPython/blob/master/autobahn/autobahn/websocket.py#L2687

The fix commit is

https://github.com/tavendo/AutobahnPython/commit/049d3ea979c71714b1504924696586a1e467acee

The reason is that Safari/Opera are very picky about the
Sec-WebSocket-Location header returned. When running WS on port 80 (as
we do for production), it wants to have no port (no 80) in that header
at all. If a port is present, they close the WS immediately
without any notice. Awesome.

Hope that helps,
Tobias

0 Likes

#8

Hi Daniel,

I'm glad you resolved your issue. And also that Autobahn wasn't the
guilty one;)

WebSocket is new stuff, there will be issues. Point is: community
as a whole needs to learn, and it would probably be helpful for
others if you described your specific setup, the guilty software
and what you did to identify / resolve in some detail, i.e.
on this list where others are reporting too:

http://www.ietf.org/mail-archive/web/hybi/current/msg09490.html

Cheers,
Tobias

···

Am 29.03.2012 03:21, schrieb Daniel F.:

Hi Tobias,

so I tried reversing the situation, installing the server on the
workstation and connecting to the workstation from a browser on the
server, and that worked flawlessly. I followed your suggestion of
isolating the python installation with virtualenv on the server, but the
problem persisted. So I used Wireshark on the workstation and noticed
that the packets indeed first got collected for a random amount of time
on the workstation before getting out on the wire. My belief over the
entire day was that I messed up something in twisted, serverside. I
double checked the Firewall and there it was. There's a "feature" called
"content filter" which buffers a certain ammount of bytes (even if it's
configured to not filter the content)... It's even stranger to notice
that the same content filter is active on Chrome, but there it doesn't
appear to have any effect. Go figure. Next time I'll double check the
Firewall. The oddest thing is that the content filter buffering is still
active even when the Firewall is disabled (but not shut down, what a
coding...)!

At least this wasn't for nothing. I checked out a bit more about the
internals of autobahn, it's examples and got reminded about virtualenv...

So, thanks Tobias for your help.

Daniel

On Wednesday, March 28, 2012 7:44:57 PM UTC+2, Tobias Oberstein wrote:

    Hi Daniel,

    ok, we have the new Autobahn 0.5.1 in production use on autobahn.ws
    <http://autobahn.ws> Web
    site. You can "verify": it replies "Server: AutobahnPython/0.5.1" in
    HTTP headers, and it speaks Hybi and Hixie.

    You can checkout the live demos there using your browsers.

    It is working for us from various different Hybi, Hixie, desktop and
    mobile browsers - at least

    http://autobahn.ws/demo/pubsub

    works on all. The others have issues still on certain mobiles due to
    the use of HTML5 audio and/or jQuery (non-mobile).

    There was 1 glitch related to Hixie opening handshake, that only now
    appeared when running on _standard_ ports 80 or 443:

    https://github.com/tavendo/AutobahnPython/blob/master/autobahn/autobahn/websocket.py#L2687
    <https://github.com/tavendo/AutobahnPython/blob/master/autobahn/autobahn/websocket.py#L2687>

    The fix commit is

    https://github.com/tavendo/AutobahnPython/commit/049d3ea979c71714b1504924696586a1e467acee
    <https://github.com/tavendo/AutobahnPython/commit/049d3ea979c71714b1504924696586a1e467acee>

    The reason is that Safari/Opera are _very_ picky about the
    Sec-WebSocket-Location header returned. When running WS on port 80 (as
    we do for production), it wants to have _no_ port (no 80) in that
    header
    at all. If a port is present, they close the WS immediately
    without any notice. Awesome.

    Hope that helps,
    Tobias

0 Likes