Test Suite questions

#1

Hi, thanks for the excellent test suites. I got some questions from running it against my websocket server implementation.

  1. Origin Header in websocket request

My server by default requires that Origin matches Host. This seems to be a sensible default for most server applications. Maybe it would be a good idea for autobahn to set Origin header in websocket request that matches Host?

  1. Case 2.10 - consecutive Ping frames

My server receives 10 consecutive pings in one round of parsing; responds with only one Pong (for the last Ping). This conforms to the spec (5.5.3), and arguably is better than responding with 10 consecutive Pongs.

Maybe autobahn could add some delays between pings, so that servers will not consolidate them?

  1. Consecutive frames

In case 3.2, my server sees an invalid frame before it can respond to the prev text message. The behavior is considered as ‘NON-STRICT’ by autobahn. But I think it’s perfectly valid to peek ahead to see the error and close connection immediately. Maybe autobahn could add some delays between frames?

As far as I see, this applies to case 3.2 / 4.1.3 / 4.1.4 / 4.2.3 / 4.2.4 / 5.15

  1. Case 7.1.5 - close frame in the middle of a message

My server considers this as an error and drops the connection uncleanly. I cannot find in the spec how it should be treated; is it required that the server sends a close frame?

  1. Invalid UTF-8

My server does not yet parse UTF-8 bytes, it just blindly echoes back the raw bytes. To my surprise, all 6.. case are passed:) For example in case 6.3.1

002 TX FRAME : OPCODE=1, FIN=True, RSV=0, PAYLOAD-LEN=20, MASK=1ef3056a, PAYLOAD-REPEAT-LEN=None, CHOPSIZE=None, SYNC=False
0xcebae1bdb9cf83cebcceb5eda080656469746564

005 RX OCTETS: 8114cebae1bdb9cf83cebcceb5eda080656469746564

008 RX FRAME : OPCODE=1, FIN=True, RSV=0, PAYLOAD-LEN=0, MASKED=False, MASK=None
009 RX OCTETS: 8800
010 RX FRAME : OPCODE=8, FIN=True, RSV=0, PAYLOAD-LEN=0, MASKED=False, MASK=None
011 TCP DROPPED BY PEER

In these cases, if the server responds with invalid UTF-8 bytes, autobahn would consider the test passed? That seems to be incorrect.

Best regards,
Zhong Yu

0 Likes

#2

Zhong,

Hi, thanks for the excellent test suites. I got some questions from

Thanks! Good to hear it's of use to you ..

running it against my websocket server implementation.

1. Origin Header in websocket request

My server by default requires that Origin matches Host. This seems to be
a sensible default for most server applications. Maybe it would be a
good idea for autobahn to set Origin header in websocket request that
matches Host?

No. A server that _requires_ the presence of "origin" is non-compliant.
Browser WebSocket clients are required to set "origin" and disallow JavaScript to modify this. Non-browser clients can send anything in "origin", and hence the server cannot rely on this header anyway.
AutobahnPython still allows to set (for clients) the origin header and process (for server) the header (if present) - optionally.

=> INVALID

2. Case 2.10 - consecutive Ping frames

My server receives 10 consecutive pings in one round of parsing;
responds with only one Pong (for the last Ping). This conforms to the
spec (5.5.3), and arguably is better than responding with 10 consecutive
Pongs.

Maybe autobahn could add some delays between pings, so that servers will
not consolidate them?

I agree. The testsuite needs to be able to handle such implementations as well. I does not currently. Btw: IE10 behaves similarily, but Chrome/FF will respond to each and every ping with a separate pong (as does AutobahnPython).

=> BUG

3. Consecutive frames

In case 3.2, my server sees an invalid frame before it can respond to
the prev text message. The behavior is considered as 'NON-STRICT' by
autobahn. But I think it's perfectly valid to peek ahead to see the
error and close connection immediately. Maybe autobahn could add some
delays between frames?

As far as I see, this applies to case 3.2 / 4.1.3 / 4.1.4 / 4.2.3 /
4.2.4 / 5.15

The WS spec does not require valid WS traffic preceding invalid traffic to be consumed/processed. However, an implementation never knows if some invalid WS comes in subsequently, and hence, how long is your "look ahead" before actual app level processing occurs?

Hence, the testsuite definition on "strict" is: process any WS exactly up to the first octet which renders the received WS invalid. This is "canonical" and "definite".

Note that "non-strict" is still conformant to the spec!

=> WONTFIX

4. Case 7.1.5 - close frame in the middle of a message

My server considers this as an error and drops the connection uncleanly.
I cannot find in the spec how it should be treated; is it required that
the server sends a close frame?

Yes. The spec a) allows control frames to occur in the middle of a (fragmented) message and b) defines how to react upon receiving a close frame and c) nowhere alters b) for the special case of occuring in the middle of a fragmented message.

=> INVALID

5. Invalid UTF-8

My server does not yet parse UTF-8 bytes, it just blindly echoes back
the raw bytes. To my surprise, all 6.*.* case are passed:) For example
in case 6.3.1

002 TX FRAME : OPCODE=1, FIN=True, RSV=0, PAYLOAD-LEN=20, MASK=1ef3056a,
PAYLOAD-REPEAT-LEN=None, CHOPSIZE=None, SYNC=False
0xcebae1bdb9cf83cebcceb5eda080656469746564
...
005 RX OCTETS: 8114cebae1bdb9cf83cebcceb5eda080656469746564
...
008 RX FRAME : OPCODE=1, FIN=True, RSV=0, PAYLOAD-LEN=0, MASKED=False,
MASK=None
009 RX OCTETS: 8800
010 RX FRAME : OPCODE=8, FIN=True, RSV=0, PAYLOAD-LEN=0, MASKED=False,
MASK=None
011 TCP DROPPED BY PEER

In these cases, if the server responds with invalid UTF-8 bytes,
autobahn would consider the test passed? That seems to be incorrect.

This is incorrect. I need to look into this.

=> BUG

···

Am 30.08.2013 20:12, schrieb zhong...@gmail.com:

===

Best would be to file issues on GitHub (https://github.com/tavendo/AutobahnTestSuite) or even better: provide code / pull requests;)

Cheers,
Tobias

Best regards,
Zhong Yu

--
You received this message because you are subscribed to the Google
Groups "Autobahn" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to autobahnws+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

0 Likes

#3

Zhong,

Hi, thanks for the excellent test suites. I got some questions from

Thanks! Good to hear it's of use to you ..

running it against my websocket server implementation.

1. Origin Header in websocket request

My server by default requires that Origin matches Host. This seems to be
a sensible default for most server applications. Maybe it would be a
good idea for autobahn to set Origin header in websocket request that
matches Host?

No. A server that _requires_ the presence of "origin" is non-compliant.

My server _by default_ enforces same origin policy, I needed to
disable that feature for autobahn testsuite. Of course, same origin
policy is not mandatory, but it's probably a sensible default
configuration for most websocket server apps.

I am not suggesting that autobahn tests a server for how it handles
"Origin" header; I'm just saying that the requests could add the
"Origin" header anyway, to accommodate servers that do require the
header by default, so that they don't need to change the default
configuration to be tested by autobahn.

This won't break any server that does not check "Origin" header by default.

Browser WebSocket clients are required to set "origin" and disallow
JavaScript to modify this. Non-browser clients can send anything in
"origin", and hence the server cannot rely on this header anyway.
AutobahnPython still allows to set (for clients) the origin header and
process (for server) the header (if present) - optionally.

=> INVALID

2. Case 2.10 - consecutive Ping frames

My server receives 10 consecutive pings in one round of parsing;
responds with only one Pong (for the last Ping). This conforms to the
spec (5.5.3), and arguably is better than responding with 10 consecutive
Pongs.

Maybe autobahn could add some delays between pings, so that servers will
not consolidate them?

I agree. The testsuite needs to be able to handle such implementations as
well. I does not currently. Btw: IE10 behaves similarily, but Chrome/FF will
respond to each and every ping with a separate pong (as does
AutobahnPython).

=> BUG

3. Consecutive frames

In case 3.2, my server sees an invalid frame before it can respond to
the prev text message. The behavior is considered as 'NON-STRICT' by
autobahn. But I think it's perfectly valid to peek ahead to see the
error and close connection immediately. Maybe autobahn could add some
delays between frames?

As far as I see, this applies to case 3.2 / 4.1.3 / 4.1.4 / 4.2.3 /
4.2.4 / 5.15

The WS spec does not require valid WS traffic preceding invalid traffic to
be consumed/processed. However, an implementation never knows if some
invalid WS comes in subsequently, and hence, how long is your "look ahead"
before actual app level processing occurs?

If inbound and outbound are processed concurrently, it's possible that
the inbound sees a protocol error and drops the connection, before the
outbound can flush the echo to the previous message.

I'm guessing that a 10ms delay should be enough to avoid such concurrency issue.

Hence, the testsuite definition on "strict" is: process any WS exactly up to
the first octet which renders the received WS invalid. This is "canonical"
and "definite".

Note that "non-strict" is still conformant to the spec!

=> WONTFIX

4. Case 7.1.5 - close frame in the middle of a message

My server considers this as an error and drops the connection uncleanly.
I cannot find in the spec how it should be treated; is it required that
the server sends a close frame?

Yes. The spec a) allows control frames to occur in the middle of a
(fragmented) message and b) defines how to react upon receiving a close
frame and c) nowhere alters b) for the special case of occuring in the
middle of a fragmented message.

You are right, the spec is being very absolute here (which I think is
a mistake). So the testsuite is correct.

I have another concern though - the testsuite probably shouldn't send
more data after the close frame. It may trigger the TCP RST problem
which prevents the testsuite to receive all server responses.

=> INVALID

5. Invalid UTF-8

My server does not yet parse UTF-8 bytes, it just blindly echoes back
the raw bytes. To my surprise, all 6.*.* case are passed:) For example
in case 6.3.1

002 TX FRAME : OPCODE=1, FIN=True, RSV=0, PAYLOAD-LEN=20, MASK=1ef3056a,
PAYLOAD-REPEAT-LEN=None, CHOPSIZE=None, SYNC=False
0xcebae1bdb9cf83cebcceb5eda080656469746564
...
005 RX OCTETS: 8114cebae1bdb9cf83cebcceb5eda080656469746564
...
008 RX FRAME : OPCODE=1, FIN=True, RSV=0, PAYLOAD-LEN=0, MASKED=False,
MASK=None
009 RX OCTETS: 8800
010 RX FRAME : OPCODE=8, FIN=True, RSV=0, PAYLOAD-LEN=0, MASKED=False,
MASK=None
011 TCP DROPPED BY PEER

In these cases, if the server responds with invalid UTF-8 bytes,
autobahn would consider the test passed? That seems to be incorrect.

This is incorrect. I need to look into this.

=> BUG

===

Best would be to file issues on GitHub
(https://github.com/tavendo/AutobahnTestSuite) or even better: provide code
/ pull requests;)

Will do in future.

Thanks,
Zhong Yu

···

On Sat, Aug 31, 2013 at 7:04 AM, Tobias Oberstein <tobias.o...@gmail.com> wrote:

Am 30.08.2013 20:12, schrieb zhong...@gmail.com:

Cheers,
Tobias

Best regards,
Zhong Yu

--
You received this message because you are subscribed to the Google
Groups "Autobahn" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to autobahnws+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"Autobahn" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to autobahnws+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

0 Likes