Handling large data via RPC (crossbar + autobahn|python)

#1

Hello,

I am trying to transmit large data through websockets using crossbar/autobahn’s RPC. My setup is as follow:

  • Python 2.7

  • A crossbar router (version 17.8.1.post1)

  • A back-end that will try to send a large pandas DataFrame as a json string

  • A front-end that will want to receive this string

In essence my front-end is doing that and the function data.get will return a large string.

class MyComponent(ApplicationSession):

@inlineCallbacks
def onJoin(self, details):
    print("session ready")
    try:
        res = yield self.call(u'data.get')

``

And I get this error:

2017-08-09T16:38:10+0200 session closed with reason wamp.close.transport_lost [WAMP transport was lost without closing the session before]
2017-08-09T16:38:10+0200 Cancelling 1 outstanding requests
2017-08-09T16:38:10+0200 call error: ApplicationError(error=<wamp.close.transport_lost>, args=[u’WAMP transport was lost without closing the session before’], kwargs={}, enc_algo=None)

It seems crossbar is kicking me out because my client session looks dead to him, but I thought that autobahn would chunk my data and that the call would not block the client reactor.

I enabled a few things in my crossbar configuration to improve websocket treatment; thanks to that I was able to transmit larger amount of data but eventually I would hit a limit (config file largely copied and pasted from sam & max).

                        "options": {
                            "enable_webstatus": false,
                            "max_frame_size": 16777216,
                            "auto_fragment_size": 65536,
                            "fail_by_drop": true,
                            "open_handshake_timeout": 2500,
                            "close_handshake_timeout": 1000,
                            "auto_ping_interval": 10000,
                            "auto_ping_timeout": 5000,
                            "auto_ping_size": 4,
                            "compression": {
                                "deflate": {
                                    "request_no_context_takeover": false,
                                    "request_max_window_bits": 11,
                                    "no_context_takeover": false,
                                    "max_window_bits": 11,
                                    "memory_level": 4
                               }
                            }
                        }

Any ideas, takes, things that I am doing wrong?

Thank you,

Adrien

crossbar-test.tar (10 KB)

0 Likes

#2

This sounds like a job for progressive results?

See http://crossbar.io/docs/Progressive-Call-Results/

···

On Wednesday, August 9, 2017 at 4:52:40 PM UTC+2, Ciclet Adrien wrote:

Hello,

I am trying to transmit large data through websockets using crossbar/autobahn’s RPC. My setup is as follow:

  • Python 2.7
  • A crossbar router (version 17.8.1.post1)
  • A back-end that will try to send a large pandas DataFrame as a json string
  • A front-end that will want to receive this string

In essence my front-end is doing that and the function data.get will return a large string.

class MyComponent(ApplicationSession):

@inlineCallbacks
def onJoin(self, details):
    print("session ready")
    try:
        res = yield self.call(u'data.get')

``

And I get this error:

2017-08-09T16:38:10+0200 session closed with reason wamp.close.transport_lost [WAMP transport was lost without closing the session before]
2017-08-09T16:38:10+0200 Cancelling 1 outstanding requests
2017-08-09T16:38:10+0200 call error: ApplicationError(error=<wamp.close.transport_lost>, args=[u’WAMP transport was lost without closing the session before’], kwargs={}, enc_algo=None)

It seems crossbar is kicking me out because my client session looks dead to him, but I thought that autobahn would chunk my data and that the call would not block the client reactor.

I enabled a few things in my crossbar configuration to improve websocket treatment; thanks to that I was able to transmit larger amount of data but eventually I would hit a limit (config file largely copied and pasted from sam & max).

                        "options": {
                            "enable_webstatus": false,
                            "max_frame_size": 16777216,
                            "auto_fragment_size": 65536,
                            "fail_by_drop": true,
                            "open_handshake_timeout": 2500,
                            "close_handshake_timeout": 1000,
                            "auto_ping_interval": 10000,
                            "auto_ping_timeout": 5000,
                            "auto_ping_size": 4,
                            "compression": {
                                "deflate": {
                                    "request_no_context_takeover": false,
                                    "request_max_window_bits": 11,
                                    "no_context_takeover": false,
                                    "max_window_bits": 11,
                                    "memory_level": 4
                               }
                            }
                        }

Any ideas, takes, things that I am doing wrong?

Thank you,

Adrien

0 Likes

#3

This sounds like a job for progressive results?

See http://crossbar.io/docs/Progressive-Call-Results/

yes, exactly. this is the right way to approach "large file download via WAMP".

side note: for the opposite direction, that is "large file upload", is missing the "progressive call arguments" feature in CB still (and in the WAMP spec). it is straight forward, but needs work/time.

···

Am 10.08.2017 um 09:54 schrieb Adam Jorgensen:

On Wednesday, August 9, 2017 at 4:52:40 PM UTC+2, Ciclet Adrien wrote:

Hello,

I am trying to transmit large data through websockets using
crossbar/autobahn's RPC. My setup is as follow:
- Python 2.7
- A crossbar router (version 17.8.1.post1)
- A back-end that will try to send a large pandas DataFrame as a json
string
- A front-end that will want to receive this string

In essence my front-end is doing that and the function data.get will
return a large string.

class MyComponent(ApplicationSession):

     @inlineCallbacks
     def onJoin(self, details):
         print("session ready")
         try:
             res = yield self.call(u'data.get')

And I get this error:

2017-08-09T16:38:10+0200 session closed with reason
wamp.close.transport_lost [WAMP transport was lost without closing the
session before]
2017-08-09T16:38:10+0200 Cancelling 1 outstanding requests
2017-08-09T16:38:10+0200 call error:
ApplicationError(error=<wamp.close.transport_lost>, args=[u'WAMP transport
was lost without closing the session before'], kwargs={}, enc_algo=None)

It seems crossbar is kicking me out because my client session looks dead
to him, but I thought that autobahn would chunk my data and that the call
would not block the client reactor.

I enabled a few things in my crossbar configuration to improve websocket
treatment; thanks to that I was able to transmit larger amount of data but
eventually I would hit a limit (config file largely copied and pasted from
sam & max).

                             "options": {
                                 "enable_webstatus": false,
                                 "max_frame_size": 16777216,
                                 "auto_fragment_size": 65536,
                                 "fail_by_drop": true,
                                 "open_handshake_timeout": 2500,
                                 "close_handshake_timeout": 1000,
                                 "auto_ping_interval": 10000,
                                 "auto_ping_timeout": 5000,
                                 "auto_ping_size": 4,
                                 "compression": {
                                     "deflate": {
                                         "request_no_context_takeover":
false,
                                         "request_max_window_bits": 11,
                                         "no_context_takeover": false,
                                         "max_window_bits": 11,
                                         "memory_level": 4
                                    }
                                 }
                             }

Any ideas, takes, things that I am doing wrong?

Thank you,

Adrien

0 Likes

#4

While progressive call parameters are missing you could probably mimic it by reversing the direction with a workflow like:

caller calls x.y.z.upload and includes a parameter that contains a WAMP URI for an end-point it has registered to handle the upload

callee for x.y.z.upload end-point process initial upload details and then calls the URI it received from the initial caller in order to retrieve the chunks for the upload part

Once callee’s call to the custom URI is done it completes and the original caller continues executing past it’s call to x.y.z.upload and maybe unregisters it’s custom end-point for the “upload”?

I imagine I could probably wrap this functionality up in a decorator :slight_smile:

···

On Thursday, August 10, 2017 at 10:10:08 AM UTC+2, Tobias Oberstein wrote:

Am 10.08.2017 um 09:54 schrieb Adam Jorgensen:

This sounds like a job for progressive results?

See http://crossbar.io/docs/Progressive-Call-Results/

yes, exactly. this is the right way to approach “large file download via
WAMP”.

side note: for the opposite direction, that is “large file upload”, is
missing the “progressive call arguments” feature in CB still (and in the
WAMP spec). it is straight forward, but needs work/time.

On Wednesday, August 9, 2017 at 4:52:40 PM UTC+2, Ciclet Adrien wrote:

Hello,

I am trying to transmit large data through websockets using

crossbar/autobahn’s RPC. My setup is as follow:

  • Python 2.7
  • A crossbar router (version 17.8.1.post1)
  • A back-end that will try to send a large pandas DataFrame as a json

string

  • A front-end that will want to receive this string

In essence my front-end is doing that and the function data.get will

return a large string.

class MyComponent(ApplicationSession):

 @inlineCallbacks
 def onJoin(self, details):
     print("session ready")
     try:
         res = yield self.call(u'data.get')

And I get this error:

2017-08-09T16:38:10+0200 session closed with reason

wamp.close.transport_lost [WAMP transport was lost without closing the

session before]

2017-08-09T16:38:10+0200 Cancelling 1 outstanding requests

2017-08-09T16:38:10+0200 call error:

ApplicationError(error=<wamp.close.transport_lost>, args=[u’WAMP transport

was lost without closing the session before’], kwargs={}, enc_algo=None)

It seems crossbar is kicking me out because my client session looks dead

to him, but I thought that autobahn would chunk my data and that the call

would not block the client reactor.

I enabled a few things in my crossbar configuration to improve websocket

treatment; thanks to that I was able to transmit larger amount of data but

eventually I would hit a limit (config file largely copied and pasted from

sam & max).

                         "options": {
                             "enable_webstatus": false,
                             "max_frame_size": 16777216,
                             "auto_fragment_size": 65536,
                             "fail_by_drop": true,
                             "open_handshake_timeout": 2500,
                             "close_handshake_timeout": 1000,
                             "auto_ping_interval": 10000,
                             "auto_ping_timeout": 5000,
                             "auto_ping_size": 4,
                             "compression": {
                                 "deflate": {
                                     "request_no_context_takeover":

false,

                                     "request_max_window_bits": 11,
                                     "no_context_takeover": false,
                                     "max_window_bits": 11,
                                     "memory_level": 4
                                }
                             }
                         }

Any ideas, takes, things that I am doing wrong?

Thank you,

Adrien

0 Likes

#5

While progressive call parameters are missing you could probably mimic it
by reversing the direction with a workflow like:

yeah, right. that would work;) it is a slight hack still .. but a clever one

···

Am 10.08.2017 um 10:33 schrieb Adam Jorgensen:

caller calls x.y.z.upload and includes a parameter that contains a WAMP URI
for an end-point it has registered to handle the upload
callee for x.y.z.upload end-point process initial upload details and then
calls the URI it received from the initial caller in order to retrieve the
chunks for the upload part
Once callee's call to the custom URI is done it completes and the original
caller continues executing past it's call to x.y.z.upload and maybe
unregisters it's custom end-point for the "upload"?

I imagine I could probably wrap this functionality up in a decorator :slight_smile:

On Thursday, August 10, 2017 at 10:10:08 AM UTC+2, Tobias Oberstein wrote:

Am 10.08.2017 um 09:54 schrieb Adam Jorgensen:

This sounds like a job for progressive results?

See http://crossbar.io/docs/Progressive-Call-Results/

yes, exactly. this is the right way to approach "large file download via
WAMP".

side note: for the opposite direction, that is "large file upload", is
missing the "progressive call arguments" feature in CB still (and in the
WAMP spec). it is straight forward, but needs work/time.

On Wednesday, August 9, 2017 at 4:52:40 PM UTC+2, Ciclet Adrien wrote:

Hello,

I am trying to transmit large data through websockets using
crossbar/autobahn's RPC. My setup is as follow:
- Python 2.7
- A crossbar router (version 17.8.1.post1)
- A back-end that will try to send a large pandas DataFrame as a json
string
- A front-end that will want to receive this string

In essence my front-end is doing that and the function data.get will
return a large string.

class MyComponent(ApplicationSession):

      @inlineCallbacks
      def onJoin(self, details):
          print("session ready")
          try:
              res = yield self.call(u'data.get')

And I get this error:

2017-08-09T16:38:10+0200 session closed with reason
wamp.close.transport_lost [WAMP transport was lost without closing the
session before]
2017-08-09T16:38:10+0200 Cancelling 1 outstanding requests
2017-08-09T16:38:10+0200 call error:
ApplicationError(error=<wamp.close.transport_lost>, args=[u'WAMP

transport

was lost without closing the session before'], kwargs={},

enc_algo=None)

It seems crossbar is kicking me out because my client session looks

dead

to him, but I thought that autobahn would chunk my data and that the

call

would not block the client reactor.

I enabled a few things in my crossbar configuration to improve

websocket

treatment; thanks to that I was able to transmit larger amount of data

but

eventually I would hit a limit (config file largely copied and pasted

from

sam & max).

                              "options": {
                                  "enable_webstatus": false,
                                  "max_frame_size": 16777216,
                                  "auto_fragment_size": 65536,
                                  "fail_by_drop": true,
                                  "open_handshake_timeout": 2500,
                                  "close_handshake_timeout": 1000,
                                  "auto_ping_interval": 10000,
                                  "auto_ping_timeout": 5000,
                                  "auto_ping_size": 4,
                                  "compression": {
                                      "deflate": {
                                          "request_no_context_takeover":
false,
                                          "request_max_window_bits": 11,
                                          "no_context_takeover": false,
                                          "max_window_bits": 11,
                                          "memory_level": 4
                                     }
                                  }
                              }

Any ideas, takes, things that I am doing wrong?

Thank you,

Adrien

0 Likes

#6

I think on the level of hackyness it’s not thaaaaaaaaaat bad :wink:

I think it’s more a side-effect of the flexibility of WAMP and the way it decouples your application components from concepts like server-side and client-side :slight_smile:

···

On Thursday, August 10, 2017 at 10:40:18 AM UTC+2, Tobias Oberstein wrote:

Am 10.08.2017 um 10:33 schrieb Adam Jorgensen:

While progressive call parameters are missing you could probably mimic it

by reversing the direction with a workflow like:

yeah, right. that would work;) it is a slight hack still … but a clever one

caller calls x.y.z.upload and includes a parameter that contains a WAMP URI

for an end-point it has registered to handle the upload

callee for x.y.z.upload end-point process initial upload details and then

calls the URI it received from the initial caller in order to retrieve the

chunks for the upload part

Once callee’s call to the custom URI is done it completes and the original

caller continues executing past it’s call to x.y.z.upload and maybe

unregisters it’s custom end-point for the “upload”?

I imagine I could probably wrap this functionality up in a decorator :slight_smile:

On Thursday, August 10, 2017 at 10:10:08 AM UTC+2, Tobias Oberstein wrote:

Am 10.08.2017 um 09:54 schrieb Adam Jorgensen:

This sounds like a job for progressive results?

See http://crossbar.io/docs/Progressive-Call-Results/

yes, exactly. this is the right way to approach "large file download via

WAMP".

side note: for the opposite direction, that is “large file upload”, is

missing the “progressive call arguments” feature in CB still (and in the

WAMP spec). it is straight forward, but needs work/time.

On Wednesday, August 9, 2017 at 4:52:40 PM UTC+2, Ciclet Adrien wrote:

Hello,

I am trying to transmit large data through websockets using

crossbar/autobahn’s RPC. My setup is as follow:

  • Python 2.7
  • A crossbar router (version 17.8.1.post1)
  • A back-end that will try to send a large pandas DataFrame as a json

string

  • A front-end that will want to receive this string

In essence my front-end is doing that and the function data.get will

return a large string.

class MyComponent(ApplicationSession):

  @inlineCallbacks
  def onJoin(self, details):
      print("session ready")
      try:
          res = yield self.call(u'data.get')

And I get this error:

2017-08-09T16:38:10+0200 session closed with reason

wamp.close.transport_lost [WAMP transport was lost without closing the

session before]

2017-08-09T16:38:10+0200 Cancelling 1 outstanding requests

2017-08-09T16:38:10+0200 call error:

ApplicationError(error=<wamp.close.transport_lost>, args=[u’WAMP

transport

was lost without closing the session before’], kwargs={},

enc_algo=None)

It seems crossbar is kicking me out because my client session looks

dead

to him, but I thought that autobahn would chunk my data and that the

call

would not block the client reactor.

I enabled a few things in my crossbar configuration to improve

websocket

treatment; thanks to that I was able to transmit larger amount of data

but

eventually I would hit a limit (config file largely copied and pasted

from

sam & max).

                          "options": {
                              "enable_webstatus": false,
                              "max_frame_size": 16777216,
                              "auto_fragment_size": 65536,
                              "fail_by_drop": true,
                              "open_handshake_timeout": 2500,
                              "close_handshake_timeout": 1000,
                              "auto_ping_interval": 10000,
                              "auto_ping_timeout": 5000,
                              "auto_ping_size": 4,
                              "compression": {
                                  "deflate": {
                                      "request_no_context_takeover":

false,

                                      "request_max_window_bits": 11,
                                      "no_context_takeover": false,
                                      "max_window_bits": 11,
                                      "memory_level": 4
                                 }
                              }
                          }

Any ideas, takes, things that I am doing wrong?

Thank you,

Adrien

0 Likes

#7

I think on the level of hackyness it's not thaaaaaaaaaat bad :wink:

I think it's more a side-effect of the flexibility of WAMP and the way it
decouples your application components from concepts like server-side and
client-side :slight_smile:

yeah, absolutely, I agree;)

this, lets call it work-around, is only possible because the artificial "client vs server" distinction is gone with WAMP.

it seems somewhat hard to grasp for developers in general, because we've got used to this client/server thinking so much.

but it is one of the major strength of WAMP. it's all about components talking peer-to-peer (at logical level), rather "client/server".

···

Am 10.08.2017 um 10:42 schrieb Adam Jorgensen:

On Thursday, August 10, 2017 at 10:40:18 AM UTC+2, Tobias Oberstein wrote:

Am 10.08.2017 um 10:33 schrieb Adam Jorgensen:

While progressive call parameters are missing you could probably mimic

it

by reversing the direction with a workflow like:

yeah, right. that would work;) it is a slight hack still .. but a clever
one

caller calls x.y.z.upload and includes a parameter that contains a WAMP

URI

for an end-point it has registered to handle the upload
callee for x.y.z.upload end-point process initial upload details and

then

calls the URI it received from the initial caller in order to retrieve

the

chunks for the upload part
Once callee's call to the custom URI is done it completes and the

original

caller continues executing past it's call to x.y.z.upload and maybe
unregisters it's custom end-point for the "upload"?

I imagine I could probably wrap this functionality up in a decorator :slight_smile:

On Thursday, August 10, 2017 at 10:10:08 AM UTC+2, Tobias Oberstein >> wrote:

Am 10.08.2017 um 09:54 schrieb Adam Jorgensen:

This sounds like a job for progressive results?

See http://crossbar.io/docs/Progressive-Call-Results/

yes, exactly. this is the right way to approach "large file download

via

WAMP".

side note: for the opposite direction, that is "large file upload", is
missing the "progressive call arguments" feature in CB still (and in

the

WAMP spec). it is straight forward, but needs work/time.

On Wednesday, August 9, 2017 at 4:52:40 PM UTC+2, Ciclet Adrien wrote:

Hello,

I am trying to transmit large data through websockets using
crossbar/autobahn's RPC. My setup is as follow:
- Python 2.7
- A crossbar router (version 17.8.1.post1)
- A back-end that will try to send a large pandas DataFrame as a json
string
- A front-end that will want to receive this string

In essence my front-end is doing that and the function data.get will
return a large string.

class MyComponent(ApplicationSession):

       @inlineCallbacks
       def onJoin(self, details):
           print("session ready")
           try:
               res = yield self.call(u'data.get')

And I get this error:

2017-08-09T16:38:10+0200 session closed with reason
wamp.close.transport_lost [WAMP transport was lost without closing

the

session before]
2017-08-09T16:38:10+0200 Cancelling 1 outstanding requests
2017-08-09T16:38:10+0200 call error:
ApplicationError(error=<wamp.close.transport_lost>, args=[u'WAMP

transport

was lost without closing the session before'], kwargs={},

enc_algo=None)

It seems crossbar is kicking me out because my client session looks

dead

to him, but I thought that autobahn would chunk my data and that the

call

would not block the client reactor.

I enabled a few things in my crossbar configuration to improve

websocket

treatment; thanks to that I was able to transmit larger amount of

data

but

eventually I would hit a limit (config file largely copied and pasted

from

sam & max).

                               "options": {
                                   "enable_webstatus": false,
                                   "max_frame_size": 16777216,
                                   "auto_fragment_size": 65536,
                                   "fail_by_drop": true,
                                   "open_handshake_timeout": 2500,
                                   "close_handshake_timeout": 1000,
                                   "auto_ping_interval": 10000,
                                   "auto_ping_timeout": 5000,
                                   "auto_ping_size": 4,
                                   "compression": {
                                       "deflate": {
                                           

"request_no_context_takeover":

false,
                                           "request_max_window_bits":

11,

                                           "no_context_takeover":

false,

                                           "max_window_bits": 11,
                                           "memory_level": 4
                                      }
                                   }
                               }

Any ideas, takes, things that I am doing wrong?

Thank you,

Adrien

0 Likes

#8

Indeed, the peer-to-peer nature of WAMP and Crossbar is one of the reasons I’m so keen on it. And in practise it’s proven to be really powerful to work with :slight_smile:

···

On Thu, Aug 10, 2017 at 10:49 AM Tobias Oberstein tobias.o...@gmail.com wrote:

Am 10.08.2017 um 10:42 schrieb Adam Jorgensen:

I think on the level of hackyness it’s not thaaaaaaaaaat bad :wink:

I think it’s more a side-effect of the flexibility of WAMP and the way it

decouples your application components from concepts like server-side and

client-side :slight_smile:

yeah, absolutely, I agree;)

this, lets call it work-around, is only possible because the artificial

“client vs server” distinction is gone with WAMP.

it seems somewhat hard to grasp for developers in general, because we’ve

got used to this client/server thinking so much.

but it is one of the major strength of WAMP. it’s all about components

talking peer-to-peer (at logical level), rather “client/server”.

On Thursday, August 10, 2017 at 10:40:18 AM UTC+2, Tobias Oberstein wrote:

Am 10.08.2017 um 10:33 schrieb Adam Jorgensen:

While progressive call parameters are missing you could probably mimic

it

by reversing the direction with a workflow like:

yeah, right. that would work;) it is a slight hack still … but a clever

one

caller calls x.y.z.upload and includes a parameter that contains a WAMP

URI

for an end-point it has registered to handle the upload

callee for x.y.z.upload end-point process initial upload details and

then

calls the URI it received from the initial caller in order to retrieve

the

chunks for the upload part

Once callee’s call to the custom URI is done it completes and the

original

caller continues executing past it’s call to x.y.z.upload and maybe

unregisters it’s custom end-point for the “upload”?

I imagine I could probably wrap this functionality up in a decorator :slight_smile:

On Thursday, August 10, 2017 at 10:10:08 AM UTC+2, Tobias Oberstein > > >> wrote:

Am 10.08.2017 um 09:54 schrieb Adam Jorgensen:

This sounds like a job for progressive results?

See http://crossbar.io/docs/Progressive-Call-Results/

yes, exactly. this is the right way to approach "large file download

via

WAMP".

side note: for the opposite direction, that is “large file upload”, is

missing the “progressive call arguments” feature in CB still (and in

the

WAMP spec). it is straight forward, but needs work/time.

On Wednesday, August 9, 2017 at 4:52:40 PM UTC+2, Ciclet Adrien wrote:

Hello,

I am trying to transmit large data through websockets using

crossbar/autobahn’s RPC. My setup is as follow:

  • Python 2.7
  • A crossbar router (version 17.8.1.post1)
  • A back-end that will try to send a large pandas DataFrame as a json

string

  • A front-end that will want to receive this string

In essence my front-end is doing that and the function data.get will

return a large string.

class MyComponent(ApplicationSession):

   @inlineCallbacks
   def onJoin(self, details):
       print("session ready")
       try:
           res = yield self.call(u'data.get')

And I get this error:

2017-08-09T16:38:10+0200 session closed with reason

wamp.close.transport_lost [WAMP transport was lost without closing

the

session before]

2017-08-09T16:38:10+0200 Cancelling 1 outstanding requests

2017-08-09T16:38:10+0200 call error:

ApplicationError(error=<wamp.close.transport_lost>, args=[u’WAMP

transport

was lost without closing the session before’], kwargs={},

enc_algo=None)

It seems crossbar is kicking me out because my client session looks

dead

to him, but I thought that autobahn would chunk my data and that the

call

would not block the client reactor.

I enabled a few things in my crossbar configuration to improve

websocket

treatment; thanks to that I was able to transmit larger amount of

data

but

eventually I would hit a limit (config file largely copied and pasted

from

sam & max).

                           "options": {
                               "enable_webstatus": false,
                               "max_frame_size": 16777216,
                               "auto_fragment_size": 65536,
                               "fail_by_drop": true,
                               "open_handshake_timeout": 2500,
                               "close_handshake_timeout": 1000,
                               "auto_ping_interval": 10000,
                               "auto_ping_timeout": 5000,
                               "auto_ping_size": 4,
                               "compression": {
                                   "deflate": {

“request_no_context_takeover”:

false,

                                       "request_max_window_bits":

11,

                                       "no_context_takeover":

false,

                                       "max_window_bits": 11,
                                       "memory_level": 4
                                  }
                               }
                           }

Any ideas, takes, things that I am doing wrong?

Thank you,

Adrien

0 Likes

#9

Thank you for the suggestion, it was indeed the solution. I am just sad to have to chunk the result myself.

For other having the same question the link to the example is https://github.com/crossbario/autobahn-python/tree/master/examples/twisted/wamp/rpc/progress

···

Le mercredi 9 août 2017 16:52:40 UTC+2, Ciclet Adrien a écrit :

Hello,

I am trying to transmit large data through websockets using crossbar/autobahn’s RPC. My setup is as follow:

  • Python 2.7
  • A crossbar router (version 17.8.1.post1)
  • A back-end that will try to send a large pandas DataFrame as a json string
  • A front-end that will want to receive this string

In essence my front-end is doing that and the function data.get will return a large string.

class MyComponent(ApplicationSession):

@inlineCallbacks
def onJoin(self, details):
    print("session ready")
    try:
        res = yield self.call(u'data.get')

``

And I get this error:

2017-08-09T16:38:10+0200 session closed with reason wamp.close.transport_lost [WAMP transport was lost without closing the session before]
2017-08-09T16:38:10+0200 Cancelling 1 outstanding requests
2017-08-09T16:38:10+0200 call error: ApplicationError(error=<wamp.close.transport_lost>, args=[u’WAMP transport was lost without closing the session before’], kwargs={}, enc_algo=None)

It seems crossbar is kicking me out because my client session looks dead to him, but I thought that autobahn would chunk my data and that the call would not block the client reactor.

I enabled a few things in my crossbar configuration to improve websocket treatment; thanks to that I was able to transmit larger amount of data but eventually I would hit a limit (config file largely copied and pasted from sam & max).

                        "options": {
                            "enable_webstatus": false,
                            "max_frame_size": 16777216,
                            "auto_fragment_size": 65536,
                            "fail_by_drop": true,
                            "open_handshake_timeout": 2500,
                            "close_handshake_timeout": 1000,
                            "auto_ping_interval": 10000,
                            "auto_ping_timeout": 5000,
                            "auto_ping_size": 4,
                            "compression": {
                                "deflate": {
                                    "request_no_context_takeover": false,
                                    "request_max_window_bits": 11,
                                    "no_context_takeover": false,
                                    "max_window_bits": 11,
                                    "memory_level": 4
                               }
                            }
                        }

Any ideas, takes, things that I am doing wrong?

Thank you,

Adrien

0 Likes

#10

Hi,

the configuration options in Crossbar.io are for automatic fragmentation of WebSocket messages.

However, a big WebSocket message, even when fragmented, will still "head-of-line" block other WebSocket messages that come after.

This is how (plain) WebSocket works (nothing Crossbar.io specific).

Since WAMP messages are mapped 1:1 to WebSocket messages, a big WAMP RPC return message will effectively hinder any other WAMP message (such as eg an EVENT) to arrive at the client until the formter WAMP message has been completely received.

And hence the progressive Call Results feature of WAMP.

It does not make sense IMO to do that inside a WAMP client library, because imagine you want to send a 1GB file.

What a user probably wants to do is NOT read the 1GB into RAM to chunk it, but read chunks directly from the file.

Or: imagine you have mmap'ed the file and want to get chunks from there.

etc etc

Cheers,
/Tobias

···

Am 10.08.2017 um 18:20 schrieb Ciclet Adrien:

Thank you for the suggestion, it was indeed the solution. I am just sad to
have to chunk the result myself.

For other having the same question the link to the example is
https://github.com/crossbario/autobahn-python/tree/master/examples/twisted/wamp/rpc/progress

Le mercredi 9 août 2017 16:52:40 UTC+2, Ciclet Adrien a écrit :

Hello,

I am trying to transmit large data through websockets using
crossbar/autobahn's RPC. My setup is as follow:
- Python 2.7
- A crossbar router (version 17.8.1.post1)
- A back-end that will try to send a large pandas DataFrame as a json
string
- A front-end that will want to receive this string

In essence my front-end is doing that and the function data.get will
return a large string.

class MyComponent(ApplicationSession):

     @inlineCallbacks
     def onJoin(self, details):
         print("session ready")
         try:
             res = yield self.call(u'data.get')

And I get this error:

2017-08-09T16:38:10+0200 session closed with reason
wamp.close.transport_lost [WAMP transport was lost without closing the
session before]
2017-08-09T16:38:10+0200 Cancelling 1 outstanding requests
2017-08-09T16:38:10+0200 call error:
ApplicationError(error=<wamp.close.transport_lost>, args=[u'WAMP transport
was lost without closing the session before'], kwargs={}, enc_algo=None)

It seems crossbar is kicking me out because my client session looks dead
to him, but I thought that autobahn would chunk my data and that the call
would not block the client reactor.

I enabled a few things in my crossbar configuration to improve websocket
treatment; thanks to that I was able to transmit larger amount of data but
eventually I would hit a limit (config file largely copied and pasted from
sam & max).

                             "options": {
                                 "enable_webstatus": false,
                                 "max_frame_size": 16777216,
                                 "auto_fragment_size": 65536,
                                 "fail_by_drop": true,
                                 "open_handshake_timeout": 2500,
                                 "close_handshake_timeout": 1000,
                                 "auto_ping_interval": 10000,
                                 "auto_ping_timeout": 5000,
                                 "auto_ping_size": 4,
                                 "compression": {
                                     "deflate": {
                                         "request_no_context_takeover":
false,
                                         "request_max_window_bits": 11,
                                         "no_context_takeover": false,
                                         "max_window_bits": 11,
                                         "memory_level": 4
                                    }
                                 }
                             }

Any ideas, takes, things that I am doing wrong?

Thank you,

Adrien

0 Likes