Autobahn-Python: Connection drops due to JSON serializer sending garbage?

Hi everybody,

we are using Autobahn-Python on the client side together with the Crossbar router to send smaller amounts of data (roughly around 15 messages/second) consisting of smaller Python dictionaries that are serialized and sent as JSON.

We experience irregular connection drops (every couple of minutes or more often) due to exceptions thrown at the router side:

failing WebSocket connection (code=1007): "encountered invalid UTF-8 while processing text message at payload octet index 134"
failing WebSocket connection (code=1007): "UTF-8 text message payload ended within Unicode code point at payload octet index 134"

I recorded the traffic with Wireshark and indeed the message that causes the trouble contains some (seemingly) garbage at the end (that these message normally don’t have, normally there is just a trailing \030):

[16,1281,{"acknowledge":true},"topic_uri_name",[{"data":{"secs":1759,"nsecs":175000000}}]]97165}}@p¸¥ÚUXßZ«ùÿÿÿÿy´a§ÚU@ЧÚU±m¬ù±m¬ùÐRùÐRùàq¸¥ÚUÔRQùÿÿÿÿ='ô8|ù@DôÐÝ]§ÚU±m¬ù±m¬ùupp`

On the client side what we do is (our pub options are PublishOptions(acknowledge=True)):

autobahn_session.publish(self._wamp_uri, json_msg, options=self._wamp_pub_options)

I printed and compared the dictionary that we put in as payload and it looks fine:

{'data': {'secs': 1759, 'nsecs': 175000000}}

What could cause our troubles? Why does the serialized payload contain more garbage that leads to processing errors on router side?

Our router config and client application is pretty much following the example configs and code. Let me know if I can give any more important details.

Thanks in advance! :slight_smile:

Best, Jacob

That does seem really odd. There also looks to be a comma missing before “97165”.

If you have some simple code that repeats this, posting it could be helpful. What versions of Autobahn and Crossbar do you have? Are you running the client with python3 as well?

The 97165 is not part of the actual message anymore I think.

We are bound to Python 2.7 on the client side for now and we are using v17.10.1 of python-autobahn (twisted is v17.9.0). Those are rather old versions, I know, but that shouldn’t be an issue in itself…?

The Crossbar router is v20.8.1 and runs in Python 3.7.4

I will see if I can provide a minimal example to reproduce and get back to you.

Autobahn is not supported anymore on Python 2 and we don’t accept bug reports or will spend time on that. Sorry.

Rgd differences: one difference is the handling of binaries in WAMP when using wamp.2.json: only every worked on Python 3 (I seem to remember).

I will see if I can provide a minimal example to reproduce and get back to you.

Don’t bother: as said, we won’t spend time on python 2 or autobahn v17.10.1 :wink:

As a final note to potential readers: I was finally able to find the bug and as I already suspected it was me using autobahn wrongly and had nothing to do with library or python version. The problem turned out to be that I called session.publish() from a wrong thread (not the twisted reactor thread).

1 Like

ah yes, this can be a trap: all autobahn functions like publish, subscribe, etc MUST be called on the main thread only, because that is where the twisted reactor or the (default) asyncio event doing network IO is running. in twisted, this can be done from a background thread using “callFromThread”.

there is another trap: thread-safety of objects. when you create and only ever modify an object on your background thread, using that object to eg publish it on the main thread is fine.

however, you should not manipulate it on the main thread.

lastly, if you want to communicate from the main thread to a background thread, and you don’t want to use “deferToThread”, because the background thread should be kept running, then you should use a threadsafe and lockfree data structure like deque.

anyways, threads are evil;) but sometimes necessary. and always tricky …