Random connection drops/timeout; did some debugging; dont know how to proceed

#1

Hi,

i am writing here because i am running out of ideas and things to try in order to accomplish a stable wamp connection.

Let me start by eplaining our use case

We have a elearning scenario where there are 20 participents and one teacher. A controller web app and crossbar are running on the teachers pc.

Every participent has a tablet with a client app using ionic,angular and autobahnjs. All nodes are on a single purpose (quite expensive) wifi network.

Using the controller app and the wamp protocol (with calls and publishs) the teacher is able to send questions and content to the devices.

However almost every sent question has some tablets not recieving the wamp messages. Please note that there are also wamp connection drops on

a good internet connection to a nearby server with only one web app active.

Configuration

crossbar (newest commit)

“ws”: {

“type”: “websocket”,

“debug”: true,

“options”: {

“auto_ping_interval”: 5000,

“auto_ping_timeout”: 10000,

“auto_ping_size”: 4

}

}

reverse nginx proxy

location /ws {

    # WebSocket support (nginx 1.4)

proxy_http_version 1.1;

proxy_set_header Upgrade $http_upgrade;

proxy_set_header Connection “upgrade”;

proxy_read_timeout 57600s;

proxy_pass http://localhost:8080/ws;

proxy_set_header X-Real-IP $remote_addr;

proxy_set_header Host $host;

proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

}

autobahn 9.9. with a thin angular js wrapper

max_retries:-1,

initial_retry_delay:1.5,

max_retry_delay:5

  • There is a local 5 second intervall for another ping

Logs

The WebApp log just shows the connection drop and the reconnection (which fails for a while (random retry times) and then works).

Therefore i included only the crossbar log:

The log suggest that the physical connection is bad so the pings are not going through and we get a onAutoPingTimeout with the subsequent connection close.

What irritates me is this:

  1. It happens pretty often (30 minutes, 5, 90,6,12 etc.)

  2. Websockets are using TCP so complete package losses should handled good enough.

  3. My connection is not that bad that there are >10 second outtages so no ping gets through.

I ran a ping command on that server for a few hours to demonstrate this:

5964 packets transmitted, 5964 packets received, 0.0% packet loss

round-trip min/avg/max/stddev = 27.846/29.700/295.139/7.953 ms

i am lost here, do you have any suggestions i could try?

best regards

crossbar log from random connection drop

2016-10-13T09:11:26+0000 [Controller 436 crossbar.controller.processtypes.RouterWorkerProcess] Worker worker-001 -> Controller traffic: {2: {‘count’: 1931, ‘bytes’: 458274}, 3: {‘count’: 17, ‘bytes’: 4726}}

2016-10-13T09:11:27+0000 [Router 856 crossbar.router.protocol.WampWebSocketServerProtocol] Auto ping/pong: sending ping auto-ping/pong

2016-10-13T09:11:27+0000 [Router 856 crossbar.router.protocol.WampWebSocketServerProtocol] Expecting ping in 10.0 seconds for auto-ping/pong

2016-10-13T09:11:31+0000 [Router 856 crossbar.router.protocol.WampWebSocketServerProtocol] Auto ping/pong: onAutoPingTimeout fired

2016-10-13T09:11:31+0000 [Router 856 crossbar.router.protocol.WampWebSocketServerProtocol] dropping connection: None

2016-10-13T09:11:31+0000 [Router 856 crossbar.router.protocol.WampWebSocketServerProtocol] Connection to/from tcp4:127.0.0.1:45334 was aborted locally

2016-10-13T09:11:31+0000 [Router 856 crossbar.router.protocol.WampWebSocketServerProtocol] _connectionLost: [Failure instance: Traceback (failure with no frames): <class ‘twisted.internet.error.ConnectionAborted’>: Connection was aborted locally, using.

]

2016-10-13T09:11:31+0000 [Router 856 crossbar.router.protocol.WampWebSocketServerProtocol] WAMP-over-WebSocket transport lost: wasClean=False, code=1006, reason=“connection was closed uncleanly (None)”

2016-10-13T09:11:31+0000 [Controller 436 crossbar.controller.processtypes.RouterWorkerProcess] Worker worker-001 -> Controller traffic: {2: {‘count’: 1936, ‘bytes’: 459956}, 3: {‘count’: 17, ‘bytes’: 4726}}

2016-10-13T09:11:32+0000 [Router 856 crossbar.router.protocol.WampWebSocketServerProtocol] Auto ping/pong: onAutoPingTimeout fired

2016-10-13T09:11:32+0000 [Router 856 crossbar.router.protocol.WampWebSocketServerProtocol] dropping connection: None

2016-10-13T09:11:32+0000 [Router 856 crossbar.router.protocol.WampWebSocketServerProtocol] Connection to/from tcp4:127.0.0.1:45340 was aborted locally

2016-10-13T09:11:32+0000 [Router 856 crossbar.router.protocol.WampWebSocketServerProtocol] _connectionLost: [Failure instance: Traceback (failure with no frames): <class ‘twisted.internet.error.ConnectionAborted’>: Connection was aborted locally, using.

]

2016-10-13T09:11:32+0000 [Router 856 crossbar.router.protocol.WampWebSocketServerProtocol] WAMP-over-WebSocket transport lost: wasClean=False, code=1006, reason=“connection was closed uncleanly (None)”

2016-10-13T09:11:33+0000 [Router 856 crossbar.router.protocol.WampWebSocketServerProtocol]

[(‘logOctets’, False, ‘WampWebSocketServerFactory’),

(‘logFrames’, False, ‘WampWebSocketServerFactory’),

(‘trackTimings’, False, ‘WampWebSocketServerFactory’),

(‘utf8validateIncoming’, True, ‘WampWebSocketServerFactory’),

(‘applyMask’, True, ‘WampWebSocketServerFactory’),

(‘maxFramePayloadSize’, 0, ‘WampWebSocketServerFactory’),

(‘maxMessagePayloadSize’, 0, ‘WampWebSocketServerFactory’),

(‘autoFragmentSize’, 0, ‘WampWebSocketServerFactory’),

(‘failByDrop’, False, ‘WampWebSocketServerFactory’),

(‘echoCloseCodeReason’, False, ‘WampWebSocketServerFactory’),

(‘openHandshakeTimeout’, 0.0, ‘WampWebSocketServerFactory’),

(‘closeHandshakeTimeout’, 0.0, ‘WampWebSocketServerFactory’),

(‘tcpNoDelay’, True, ‘WampWebSocketServerFactory’),

(‘autoPingInterval’, 5.0, ‘WampWebSocketServerFactory’),

(‘autoPingTimeout’, 10.0, ‘WampWebSocketServerFactory’),

(‘autoPingSize’, 4, ‘WampWebSocketServerFactory’),

(‘versions’, [8, 13], ‘WampWebSocketServerFactory’),

(‘webStatus’, True, ‘WampWebSocketServerFactory’),

(‘requireMaskedClientFrames’, True, ‘WampWebSocketServerFactory’),

(‘maskServerFrames’, False, ‘WampWebSocketServerFactory’),

(‘perMessageCompressionAccept’,

<function at 0x7f720a7abc08>,

‘WampWebSocketServerFactory’),

(‘serveFlashSocketPolicy’, False, ‘WampWebSocketServerFactory’),

(‘flashSocketPolicy’,

u’\n \n\x00’,

‘WampWebSocketServerFactory’),

(‘allowedOrigins’,

[u’ws://*’, u’http://*’, u’file://*’, u’https://*’],

‘WampWebSocketServerFactory’),

(‘allowedOriginsPatterns’,

[<_sre.SRE_Pattern object at 0x7f720a822870>,

<_sre.SRE_Pattern object at 0x7f720a7bb1d0>,

<_sre.SRE_Pattern object at 0x7f720a7bb100>,

<_sre.SRE_Pattern object at 0x7f720a84e618>],

‘WampWebSocketServerFactory’),

(‘allowNullOrigin’, True, ‘WampWebSocketServerFactory’),

(‘maxConnections’, 0, ‘WampWebSocketServerFactory’)]

0 Likes

#2

Hi,

this might be due to intermediaries (Web proxies, firewalls, etc) inteferring.

In general, the best results can be achieved by using TLS (secure WebSocket) with no reverse proxies in between, but Crossbar.io terminating the TLS connection.

IOW, I’d recommend removing Nginx (for now) and using TLS.

Let me know what you get with that. Depending on the outcome, we can then track down your issues further … I’ll help.

Cheers,
/Tobias

···

Am Donnerstag, 13. Oktober 2016 16:20:24 UTC+2 schrieb Bebopp:

Hi,

i am writing here because i am running out of ideas and things to try in order to accomplish a stable wamp connection.

Let me start by eplaining our use case

We have a elearning scenario where there are 20 participents and one teacher. A controller web app and crossbar are running on the teachers pc.

Every participent has a tablet with a client app using ionic,angular and autobahnjs. All nodes are on a single purpose (quite expensive) wifi network.

Using the controller app and the wamp protocol (with calls and publishs) the teacher is able to send questions and content to the devices.

However almost every sent question has some tablets not recieving the wamp messages. Please note that there are also wamp connection drops on

a good internet connection to a nearby server with only one web app active.

Configuration

crossbar (newest commit)

“ws”: {

“type”: “websocket”,

“debug”: true,

“options”: {

“auto_ping_interval”: 5000,

“auto_ping_timeout”: 10000,

“auto_ping_size”: 4

}

}

reverse nginx proxy

location /ws {

    # WebSocket support (nginx 1.4)

proxy_http_version 1.1;

proxy_set_header Upgrade $http_upgrade;

proxy_set_header Connection “upgrade”;

proxy_read_timeout 57600s;

proxy_pass http://localhost:8080/ws;

proxy_set_header X-Real-IP $remote_addr;

proxy_set_header Host $host;

proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

}

autobahn 9.9. with a thin angular js wrapper

max_retries:-1,

initial_retry_delay:1.5,

max_retry_delay:5

  • There is a local 5 second intervall for another ping

Logs

The WebApp log just shows the connection drop and the reconnection (which fails for a while (random retry times) and then works).

Therefore i included only the crossbar log:

The log suggest that the physical connection is bad so the pings are not going through and we get a onAutoPingTimeout with the subsequent connection close.

What irritates me is this:

  1. It happens pretty often (30 minutes, 5, 90,6,12 etc.)
  1. Websockets are using TCP so complete package losses should handled good enough.
  1. My connection is not that bad that there are >10 second outtages so no ping gets through.

I ran a ping command on that server for a few hours to demonstrate this:

5964 packets transmitted, 5964 packets received, 0.0% packet loss

round-trip min/avg/max/stddev = 27.846/29.700/295.139/7.953 ms

i am lost here, do you have any suggestions i could try?

best regards

crossbar log from random connection drop

2016-10-13T09:11:26+0000 [Controller 436 crossbar.controller.processtypes.RouterWorkerProcess] Worker worker-001 -> Controller traffic: {2: {‘count’: 1931, ‘bytes’: 458274}, 3: {‘count’: 17, ‘bytes’: 4726}}

2016-10-13T09:11:27+0000 [Router 856 crossbar.router.protocol.WampWebSocketServerProtocol] Auto ping/pong: sending ping auto-ping/pong

2016-10-13T09:11:27+0000 [Router 856 crossbar.router.protocol.WampWebSocketServerProtocol] Expecting ping in 10.0 seconds for auto-ping/pong

2016-10-13T09:11:31+0000 [Router 856 crossbar.router.protocol.WampWebSocketServerProtocol] Auto ping/pong: onAutoPingTimeout fired

2016-10-13T09:11:31+0000 [Router 856 crossbar.router.protocol.WampWebSocketServerProtocol] dropping connection: None

2016-10-13T09:11:31+0000 [Router 856 crossbar.router.protocol.WampWebSocketServerProtocol] Connection to/from tcp4:127.0.0.1:45334 was aborted locally

2016-10-13T09:11:31+0000 [Router 856 crossbar.router.protocol.WampWebSocketServerProtocol] _connectionLost: [Failure instance: Traceback (failure with no frames): <class ‘twisted.internet.error.ConnectionAborted’>: Connection was aborted locally, using.

]

2016-10-13T09:11:31+0000 [Router 856 crossbar.router.protocol.WampWebSocketServerProtocol] WAMP-over-WebSocket transport lost: wasClean=False, code=1006, reason=“connection was closed uncleanly (None)”

2016-10-13T09:11:31+0000 [Controller 436 crossbar.controller.processtypes.RouterWorkerProcess] Worker worker-001 -> Controller traffic: {2: {‘count’: 1936, ‘bytes’: 459956}, 3: {‘count’: 17, ‘bytes’: 4726}}

2016-10-13T09:11:32+0000 [Router 856 crossbar.router.protocol.WampWebSocketServerProtocol] Auto ping/pong: onAutoPingTimeout fired

2016-10-13T09:11:32+0000 [Router 856 crossbar.router.protocol.WampWebSocketServerProtocol] dropping connection: None

2016-10-13T09:11:32+0000 [Router 856 crossbar.router.protocol.WampWebSocketServerProtocol] Connection to/from tcp4:127.0.0.1:45340 was aborted locally

2016-10-13T09:11:32+0000 [Router 856 crossbar.router.protocol.WampWebSocketServerProtocol] _connectionLost: [Failure instance: Traceback (failure with no frames): <class ‘twisted.internet.error.ConnectionAborted’>: Connection was aborted locally, using.

]

2016-10-13T09:11:32+0000 [Router 856 crossbar.router.protocol.WampWebSocketServerProtocol] WAMP-over-WebSocket transport lost: wasClean=False, code=1006, reason=“connection was closed uncleanly (None)”

2016-10-13T09:11:33+0000 [Router 856 crossbar.router.protocol.WampWebSocketServerProtocol]

[(‘logOctets’, False, ‘WampWebSocketServerFactory’),

(‘logFrames’, False, ‘WampWebSocketServerFactory’),

(‘trackTimings’, False, ‘WampWebSocketServerFactory’),

(‘utf8validateIncoming’, True, ‘WampWebSocketServerFactory’),

(‘applyMask’, True, ‘WampWebSocketServerFactory’),

(‘maxFramePayloadSize’, 0, ‘WampWebSocketServerFactory’),

(‘maxMessagePayloadSize’, 0, ‘WampWebSocketServerFactory’),

(‘autoFragmentSize’, 0, ‘WampWebSocketServerFactory’),

(‘failByDrop’, False, ‘WampWebSocketServerFactory’),

(‘echoCloseCodeReason’, False, ‘WampWebSocketServerFactory’),

(‘openHandshakeTimeout’, 0.0, ‘WampWebSocketServerFactory’),

(‘closeHandshakeTimeout’, 0.0, ‘WampWebSocketServerFactory’),

(‘tcpNoDelay’, True, ‘WampWebSocketServerFactory’),

(‘autoPingInterval’, 5.0, ‘WampWebSocketServerFactory’),

(‘autoPingTimeout’, 10.0, ‘WampWebSocketServerFactory’),

(‘autoPingSize’, 4, ‘WampWebSocketServerFactory’),

(‘versions’, [8, 13], ‘WampWebSocketServerFactory’),

(‘webStatus’, True, ‘WampWebSocketServerFactory’),

(‘requireMaskedClientFrames’, True, ‘WampWebSocketServerFactory’),

(‘maskServerFrames’, False, ‘WampWebSocketServerFactory’),

(‘perMessageCompressionAccept’,

<function at 0x7f720a7abc08>,

‘WampWebSocketServerFactory’),

(‘serveFlashSocketPolicy’, False, ‘WampWebSocketServerFactory’),

(‘flashSocketPolicy’,

u’\n \n\x00’,

‘WampWebSocketServerFactory’),

(‘allowedOrigins’,

[u’ws://*’, u’http://*’, u’file://*’, u’https://*’],

‘WampWebSocketServerFactory’),

(‘allowedOriginsPatterns’,

[<_sre.SRE_Pattern object at 0x7f720a822870>,

<_sre.SRE_Pattern object at 0x7f720a7bb1d0>,

<_sre.SRE_Pattern object at 0x7f720a7bb100>,

<_sre.SRE_Pattern object at 0x7f720a84e618>],

‘WampWebSocketServerFactory’),

(‘allowNullOrigin’, True, ‘WampWebSocketServerFactory’),

(‘maxConnections’, 0, ‘WampWebSocketServerFactory’)]

0 Likes