Update: WebSocket test suite reports

#1

Hello,

we have updated reports for WebSocket protocol tests for browsers (Chrome, Firefox, IE10):

http://autobahn.ws/testsuite/reports/clients/index.html

The results are quite encouraging: all green;)

Mobile and server reports will follow up.

Cheers,
Tobias

Couple of notes:

1. The "fail" of IE10 for Case 2.10 is actually testsuite fault (the spec does not require respnding to each and every ping).

2. As can be seen from the 6.4.* Cases, IE10 does fail fast on invalid UTF-8 _per frame_. Firefox and Chrome only fail per-message (no fail fast). Autobahn does fail-fast even within frames, on the very first offending octet.

3. As can be seen from Cases 9.1.*/9.2.*, Chrome Canary has improved it's large payload processing efficiency a lot compared to current released Chrome.

4. The "non-strict" case outcomes for Firefox are due to it's asynchronous processing of WebSocket. That's perfectly fine. The spec does not require to process strictly everything before a break of the protocol. All other clients however seem to do that.

5. The handling of close codes is somewhat inconsistent/different between clients.

6. As can be seen from comparing the columns AutobahnClient and AutobahnClient/PyPy, running PyPy vastly improves Autobahn's performance for large payloads. The main bottlenecks improved are masking, UTF-8 validation and general bit banging/shuffling.

Caveat: the AutobahnClient/PyPy was running locally next to Autobahn Testsuite, different from all the other clients.

7. Safari is not included, since Apple seems to have stopped shipping Safari for Windows (and stopped delivering updates for my old Macbook).

Test setup:

Machine 1:
  - Core i7, 3.4 Ghz, 12GB Ram
  - Win8 Rel. Prev. 64 Bit
  - Browsers and Autobahn on CPython

Machine 2:
  - Core i7, 3.4 Ghz, 12GB Ram
  - VirtualBox VM running Ubuntu 10.04 LTS on Win7 64-Bit host
  - Autobahn Testsuite running on PyPy (current trunk compiled from source)
  - Autobahn Client PyPy

Network: 1GbE fully switched

0 Likes

#2

Wow… results between CPython and PyPy are impressive. CPython seems dead slow in some cases.

···

On Thu, Oct 18, 2012 at 11:12 AM, Tobias Oberstein tobias.o...@gmail.com wrote:

Hello,

we have updated reports for WebSocket protocol tests for browsers (Chrome, Firefox, IE10):

http://autobahn.ws/testsuite/reports/clients/index.html

The results are quite encouraging: all green;)

Mobile and server reports will follow up.

  • Sylvain

http://www.defuze.org
http://twitter.com/lawouach

0 Likes

#3

Wow... results between CPython and PyPy are impressive. CPython seems
dead slow in some cases.

Yeah, PyPy is fast. It's primarily the masking and byte banging stuff.

However, we have done other tests (not included in the testsuite) that suggest there is 1 area where CPython is better than PyPy: GC.

I.e. on tests doing massive numbers of messages and measure roundtrip time from large number of client connections, the median RTT for PyPy is better than CPy, but the worst-case RTT is much worse. When GC kicks in, PyPy stalls for up to hundreds of ms ..

We plan to add C implementations of inner loop (masking/utf8) for CPy Autobahn .. hope that brings performance on-par with PyPy for large payloads and still has decent median and worst-case RTTs.

Cheers,
Tobias

0 Likes

#4

Wow… results between CPython and PyPy are impressive. CPython seems

dead slow in some cases.

Yeah, PyPy is fast. It’s primarily the masking and byte banging stuff.

However, we have done other tests (not included in the testsuite) that suggest there is 1 area where CPython is better than PyPy: GC.

I.e. on tests doing massive numbers of messages and measure roundtrip time from large number of client connections, the median RTT for PyPy is better than CPy, but the worst-case RTT is much worse. When GC kicks in, PyPy stalls for up to hundreds of ms …

Very interesting. Thanks for sharing that bit of info.

We plan to add C implementations of inner loop (masking/utf8) for CPy Autobahn … hope that brings performance on-par with PyPy for large payloads and still has decent median and worst-case RTTs.

I can see why this is needed to match other implementations (from different language). Python is a bit slow on certain CPU intensive tasks like that.

···

On Thu, Oct 18, 2012 at 11:44 AM, Tobias Oberstein tobias.o...@gmail.com wrote:

http://twitter.com/lawouach

0 Likes