ApplicationSession socket.publish() unable to publish to Crossbar WAMP router after "a while"

#1

Hi All,

I have recently upgraded my Autobahn/Twisted libraries, and I am now using Crossbar as my WAMP router.

As part of the upgrade, my publishing code now is run by Crossbar as an ApplicationSession instance.

I use WAMP to support some real-time web applications, and it supports a fair number of users. This is a Pub/Sub model, also using AutobahnJS libraries for the client.

I am currently having a problem where my application is eventually no longer able to publish messages to the crossbar router for delivery. Clients are still able to connect and subscribe to topics no problem, they just stop getting updates due to the inability to publish.

If it makes any difference, this seems to always occur when I get up in between 20,000 to 30,000 client connections. I am publishing an average of 1500 messages per second.

The basic publishing code is as follows:

def on_message(self, headers, message):
“”"
STOMP message handler, when I get a message from the queue, dispatch it
to my users
“”"
self.log.debug(‘received a STOMP message %s’, message)

    try:
        response = json.loads(message)

        destination = '%s.%s' % (
            response['topic'],
            response['destination_id']
        )

        self.socket.publish(
            destination,
            json.dumps(response)
        )

    except Exception as e:
        self.log.error(str(e))

``

str(e) is an empty string, I am working right now to gather any useful information from the exception,

I have made sure to follow all of the production Crossbar.io configuration suggestions, and the server itself is not under heavy load when this occurs.

Is this expected behavior, have I just hit some kind of known limit? I was hoping to support more throughput per instance, but I can certainly load balance if that is the answer. What can I do to prevent, or recover from this when it happens?

Thanks all,

Cam

0 Likes

#2

Further information:

I wrote a nagios plugin to attempt publishing to crossbar and alert if it throws the exception. The nagios monitor is still able to publish to crossbar when the ApplicationSession is unable to publish to crossbar.

0 Likes

#3

I haven’t been able to get much more info on the exception unfortunately. At this point I think we will probably just go back to WAMP V1 and the built in router and leave it.

Python 2.7.7
Twisted==15.1.0autobahn==0.10.2cffi==1.0.2characteristic==14.3.0cryptography==0.9enum34==1.0.4idna==2.0ipaddress==1.0.7java-config==2.2.0javatoolkit==0.3.0mod-pywebsocket==0.7.9ndg-httpsclient==0.4.0portage==2.2.14pyOpenSSL==0.15.1pyasn1==0.1.7pyasn1-modules==0.0.5pycparser==2.13requests==2.7.0service-identity==14.0.0six==1.9.0virtualenv==1.11wsgiref==0.1.2zope.interface==4.1.1

0 Likes

#4

To get feedback on publish, you can enable “acknowledged publication”: http://autobahn.ws/python/reference/autobahn.wamp.html#autobahn.wamp.interfaces.IPublisher.publish

Rgd performance, Crossbar.io is able to route 30k calls/sec on a single core router worker process …

···

Am Dienstag, 7. Juli 2015 21:46:29 UTC+2 schrieb Camron Levanger:

Hi All,

I have recently upgraded my Autobahn/Twisted libraries, and I am now using Crossbar as my WAMP router.

As part of the upgrade, my publishing code now is run by Crossbar as an ApplicationSession instance.

I use WAMP to support some real-time web applications, and it supports a fair number of users. This is a Pub/Sub model, also using AutobahnJS libraries for the client.

I am currently having a problem where my application is eventually no longer able to publish messages to the crossbar router for delivery. Clients are still able to connect and subscribe to topics no problem, they just stop getting updates due to the inability to publish.

If it makes any difference, this seems to always occur when I get up in between 20,000 to 30,000 client connections. I am publishing an average of 1500 messages per second.

The basic publishing code is as follows:

def on_message(self, headers, message):
“”"
STOMP message handler, when I get a message from the queue, dispatch it
to my users
“”"
self.log.debug(‘received a STOMP message %s’, message)

    try:
        response = json.loads(message)


        destination = '%s.%s' % (
            response['topic'],
            response['destination_id']
        )


        self.socket.publish(
            destination,
            json.dumps(response)
        )


    except Exception as e:
        self.log.error(str(e))

``

str(e) is an empty string, I am working right now to gather any useful information from the exception,

I have made sure to follow all of the production Crossbar.io configuration suggestions, and the server itself is not under heavy load when this occurs.

Is this expected behavior, have I just hit some kind of known limit? I was hoping to support more throughput per instance, but I can certainly load balance if that is the answer. What can I do to prevent, or recover from this when it happens?

Thanks all,

Cam

0 Likes

#5

Thank you for the hint Tobias! I have made that change and restarted the service. With any luck I can get some more useful information to report back soon.

···

On Thursday, July 9, 2015 at 5:26:19 AM UTC-6, Tobias Oberstein wrote:

To get feedback on publish, you can enable “acknowledged publication”: http://autobahn.ws/python/reference/autobahn.wamp.html#autobahn.wamp.interfaces.IPublisher.publish

Rgd performance, Crossbar.io is able to route 30k calls/sec on a single core router worker process …

Am Dienstag, 7. Juli 2015 21:46:29 UTC+2 schrieb Camron Levanger:

Hi All,

I have recently upgraded my Autobahn/Twisted libraries, and I am now using Crossbar as my WAMP router.

As part of the upgrade, my publishing code now is run by Crossbar as an ApplicationSession instance.

I use WAMP to support some real-time web applications, and it supports a fair number of users. This is a Pub/Sub model, also using AutobahnJS libraries for the client.

I am currently having a problem where my application is eventually no longer able to publish messages to the crossbar router for delivery. Clients are still able to connect and subscribe to topics no problem, they just stop getting updates due to the inability to publish.

If it makes any difference, this seems to always occur when I get up in between 20,000 to 30,000 client connections. I am publishing an average of 1500 messages per second.

The basic publishing code is as follows:

def on_message(self, headers, message):
“”"
STOMP message handler, when I get a message from the queue, dispatch it
to my users
“”"
self.log.debug(‘received a STOMP message %s’, message)

    try:
        response = json.loads(message)


        destination = '%s.%s' % (
            response['topic'],
            response['destination_id']
        )


        self.socket.publish(
            destination,
            json.dumps(response)
        )


    except Exception as e:
        self.log.error(str(e))

``

str(e) is an empty string, I am working right now to gather any useful information from the exception,

I have made sure to follow all of the production Crossbar.io configuration suggestions, and the server itself is not under heavy load when this occurs.

Is this expected behavior, have I just hit some kind of known limit? I was hoping to support more throughput per instance, but I can certainly load balance if that is the answer. What can I do to prevent, or recover from this when it happens?

Thanks all,

Cam

0 Likes

#6

Hi Camron,

just to be sure: without acknowledge, publish will return nothing and not raise. If you enable acknowledge, it'll return a Twisted Deferred or asyncio Future, and you need to attach an errback handler to that.

Or you can use co-routine style

@inlineCallbacks
def on_message(self, headers, message):
    ...
    try:
       yield self.socket.publish(...)
    except Exception as e:
       ...

···

==

unrelated note:

             self.socket.publish(
                 destination,
                 json.dumps(response)
             )

Why conversion to string again?

             self.socket.publish(
                 destination,
                 response
             )

should be fine.

Q (just curious): why are you using STOMP on the backend and not WAMP "end-to-end"? I guess you have some AMQP broker sitting around?

Cheers,
/Tobias

Am 09.07.2015 um 22:52 schrieb Camron Levanger:

Thank you for the hint Tobias! I have made that change and restarted the
service. With any luck I can get some more useful information to report
back soon.

On Thursday, July 9, 2015 at 5:26:19 AM UTC-6, Tobias Oberstein wrote:

    To get feedback on publish, you can enable "acknowledged
    publication":
    http://autobahn.ws/python/reference/autobahn.wamp.html#autobahn.wamp.interfaces.IPublisher.publish

    Rgd performance, Crossbar.io is able to route 30k calls/sec on a
    single core router worker process ..

    Am Dienstag, 7. Juli 2015 21:46:29 UTC+2 schrieb Camron Levanger:

        Hi All,

        I have recently upgraded my Autobahn/Twisted libraries, and I am
        now using Crossbar as my WAMP router.

        As part of the upgrade, my publishing code now is run by
        Crossbar as an ApplicationSession instance.

        I use WAMP to support some real-time web applications, and it
        supports a fair number of users. This is a Pub/Sub model, also
        using AutobahnJS libraries for the client.

        I am currently having a problem where my application is
        eventually no longer able to publish messages to the crossbar
        router for delivery. Clients are still able to connect and
        subscribe to topics no problem, they just stop getting updates
        due to the inability to publish.

        If it makes any difference, this seems to always occur when I
        get up in between 20,000 to 30,000 client connections. I am
        publishing an average of 1500 messages per second.

        The basic publishing code is as follows:

        >
        defon_message(self,headers,message):
        """
                 STOMP message handler, when I get a message from the
        queue, dispatch it
                 to my users
                 """
        self.log.debug('received a STOMP message %s',message)

        try:
                     response =json.loads(message)

                     destination ='%s.%s'%(
                         response['topic'],
                         response['destination_id']
        )

        self.socket.publish(
                         destination,
                         json.dumps(response)
        )

        exceptExceptionase:
        self.log.error(str(e))
        >

        str(e) is an empty string, I am working right now to gather any
        useful information from the exception,

        I have made sure to follow all of the production Crossbar.io
        configuration suggestions, and the server itself is not under
        heavy load when this occurs.

        Is this expected behavior, have I just hit some kind of known
        limit? I was hoping to support more throughput per instance, but
        I can certainly load balance if that is the answer. What can I
        do to prevent, or recover from this when it happens?

        Thanks all,

        Cam

--
You received this message because you are subscribed to the Google
Groups "Autobahn" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to autobahnws+...@googlegroups.com
<mailto:autobahnws+...@googlegroups.com>.
To post to this group, send email to autob...@googlegroups.com
<mailto:autob...@googlegroups.com>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/autobahnws/2855bcb2-fd69-4ced-a228-b5da0d43eb58%40googlegroups.com
<https://groups.google.com/d/msgid/autobahnws/2855bcb2-fd69-4ced-a228-b5da0d43eb58%40googlegroups.com?utm_medium=email&utm_source=footer>.
For more options, visit https://groups.google.com/d/optout.

0 Likes

#7

Thanks again Tobias.

I do get an error now.

repr(e) returns:

TransportLost()

But that is it.

Any ideas to move me further along?

···

On Thursday, July 9, 2015 at 5:26:19 AM UTC-6, Tobias Oberstein wrote:

To get feedback on publish, you can enable “acknowledged publication”: http://autobahn.ws/python/reference/autobahn.wamp.html#autobahn.wamp.interfaces.IPublisher.publish

Rgd performance, Crossbar.io is able to route 30k calls/sec on a single core router worker process …

Am Dienstag, 7. Juli 2015 21:46:29 UTC+2 schrieb Camron Levanger:

Hi All,

I have recently upgraded my Autobahn/Twisted libraries, and I am now using Crossbar as my WAMP router.

As part of the upgrade, my publishing code now is run by Crossbar as an ApplicationSession instance.

I use WAMP to support some real-time web applications, and it supports a fair number of users. This is a Pub/Sub model, also using AutobahnJS libraries for the client.

I am currently having a problem where my application is eventually no longer able to publish messages to the crossbar router for delivery. Clients are still able to connect and subscribe to topics no problem, they just stop getting updates due to the inability to publish.

If it makes any difference, this seems to always occur when I get up in between 20,000 to 30,000 client connections. I am publishing an average of 1500 messages per second.

The basic publishing code is as follows:

def on_message(self, headers, message):
“”"
STOMP message handler, when I get a message from the queue, dispatch it
to my users
“”"
self.log.debug(‘received a STOMP message %s’, message)

    try:
        response = json.loads(message)


        destination = '%s.%s' % (
            response['topic'],
            response['destination_id']
        )


        self.socket.publish(
            destination,
            json.dumps(response)
        )


    except Exception as e:
        self.log.error(str(e))

``

str(e) is an empty string, I am working right now to gather any useful information from the exception,

I have made sure to follow all of the production Crossbar.io configuration suggestions, and the server itself is not under heavy load when this occurs.

Is this expected behavior, have I just hit some kind of known limit? I was hoping to support more throughput per instance, but I can certainly load balance if that is the answer. What can I do to prevent, or recover from this when it happens?

Thanks all,

Cam

0 Likes

#8

Thanks again Tobias.

I do get an error now.

repr(e) returns:

TransportLost()

This means, you can't publish since there is no connection anymore.

But that is it.

Any ideas to move me further along?

Override onLeave/onDisconnect on your ApplicationSession object (the self.socket thing I suppose).

http://autobahn.ws/python/wamp/programming.html#session-lifecycle

A transport might get lost for different reasons. E.g. network intermediaries timing out an idle underlying TCP. In this case, WebSocket auto-ping/pong helps (avail in Autobahn and Crossbar.io).

···

Am 10.07.2015 um 00:08 schrieb Camron Levanger:

On Thursday, July 9, 2015 at 5:26:19 AM UTC-6, Tobias Oberstein wrote:

    To get feedback on publish, you can enable "acknowledged
    publication":
    http://autobahn.ws/python/reference/autobahn.wamp.html#autobahn.wamp.interfaces.IPublisher.publish

    Rgd performance, Crossbar.io is able to route 30k calls/sec on a
    single core router worker process ..

    Am Dienstag, 7. Juli 2015 21:46:29 UTC+2 schrieb Camron Levanger:

        Hi All,

        I have recently upgraded my Autobahn/Twisted libraries, and I am
        now using Crossbar as my WAMP router.

        As part of the upgrade, my publishing code now is run by
        Crossbar as an ApplicationSession instance.

        I use WAMP to support some real-time web applications, and it
        supports a fair number of users. This is a Pub/Sub model, also
        using AutobahnJS libraries for the client.

        I am currently having a problem where my application is
        eventually no longer able to publish messages to the crossbar
        router for delivery. Clients are still able to connect and
        subscribe to topics no problem, they just stop getting updates
        due to the inability to publish.

        If it makes any difference, this seems to always occur when I
        get up in between 20,000 to 30,000 client connections. I am
        publishing an average of 1500 messages per second.

        The basic publishing code is as follows:

        >
        defon_message(self,headers,message):
        """
                 STOMP message handler, when I get a message from the
        queue, dispatch it
                 to my users
                 """
        self.log.debug('received a STOMP message %s',message)

        try:
                     response =json.loads(message)

                     destination ='%s.%s'%(
                         response['topic'],
                         response['destination_id']
        )

        self.socket.publish(
                         destination,
                         json.dumps(response)
        )

        exceptExceptionase:
        self.log.error(str(e))
        >

        str(e) is an empty string, I am working right now to gather any
        useful information from the exception,

        I have made sure to follow all of the production Crossbar.io
        configuration suggestions, and the server itself is not under
        heavy load when this occurs.

        Is this expected behavior, have I just hit some kind of known
        limit? I was hoping to support more throughput per instance, but
        I can certainly load balance if that is the answer. What can I
        do to prevent, or recover from this when it happens?

        Thanks all,

        Cam

--
You received this message because you are subscribed to the Google
Groups "Autobahn" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to autobahnws+...@googlegroups.com
<mailto:autobahnws+...@googlegroups.com>.
To post to this group, send email to autob...@googlegroups.com
<mailto:autob...@googlegroups.com>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/autobahnws/057a4bf3-7b23-44f1-a3db-32383e997964%40googlegroups.com
<https://groups.google.com/d/msgid/autobahnws/057a4bf3-7b23-44f1-a3db-32383e997964%40googlegroups.com?utm_medium=email&utm_source=footer>.
For more options, visit https://groups.google.com/d/optout.

0 Likes