Unit testing wamp pub/sub applications

#1

I am new to autobahn and have some questions around unit testing strategies for a wamp based application that will include the following components:

  1. An Autobahn Wamp server
  2. Multiple python clients that are from remote systems that connect to the server and send remote updates.
  3. Web clients that are used to see the real time data changes.
    I am trying to figure out the best strategy to unit test these different components. I will be using pub/sub and rpc in different places. Any input from the gurus here would be really appreciated.

thanks,

Vijay

0 Likes

#2

I am new to autobahn and have some questions around unit testing
strategies for a wamp based application that will include the following

Important question!

components:

1. An Autobahn Wamp server
2. Multiple python clients that are from remote systems that connect to
    the server and send remote updates.
3. Web clients that are used to see the real time data changes.

I am trying to figure out the best strategy to unit test these different
components. I will be using pub/sub and rpc in different places. Any
input from the gurus here would be really appreciated.

It's a broad topic and there are different approaches .. some thoughts ..

Unit Testing

···

Am 10.09.2013 15:47, schrieb Vijay shan:
------------

(where "unit" are the smallest testable parts of an application):

For example, take your JS/HTML frontend code. The JS will probably do a WAMP subscribe on certain topics, and upon receiving events on those topics should execute certain code / perform certain (frontend) actions.

A unit test for the frontend could generate and inject a synthetic WAMP event and check that this triggers the desired actions in the frontend.

The unit test would be written completely in JS. For "injecting" events, there are 2 approaches:

1)
function myTopic1Handler(topic, event) {
  // your stuff that is tested
}

session.subscribe("http://mytopic1 ...", myTopic1Handler);

=> Have the unit test call myTopic1Handler directly.

2)
It would be nice if AutobahnJS had "hooks" for testing. E.g. inject a WAMP event. Since if you have

session.subscribe("http://mytopic1 ...", function (topic, event) {
  // your stuff that is tested
});

you cannot do as in 1).

Functional/Integration Testing
------------------------------

Obviously, above won't catch a lot of problems when developing distributed applications.

E.g. you want to test not only that a frontend behaves correctly _if_ it receives a certain event, but that the complete "chain" of app components works.

E.g. the Python WAMP client "A" correctly generates an event, your WAMP server correctly dispatches that, and your JS client "B" correctly processes the events.

If you wanna go (also) this route (which I think is desirable), here is how I'd approach that:

Write _one_ test client that opens _2_ WAMP connections to your WAMP server: one session running your code for "A" and one session running "B" code. This allows you to test "end-to-end".

This gets more involved, but tests really the distibuted app as a complete system.

/Tobias

thanks,
Vijay

--
You received this message because you are subscribed to the Google
Groups "Autobahn" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to autobahnws+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

0 Likes

#3

A few ideas:

Unit tests should be compact and simple. You probably shouldn’t use a WAMP server when running them. The reason is simple: When you run a test which calls some remote server via RPC which fetches some data from the Internet, and it fails, then you have about 500’000 places to check to see why. If the test just tests a single method and in total 20 lines of code are executed, then the problem must be in those 20 lines of code. It should be obvious which test is more stable and easier to fix.

The main problem with this approach is that it makes tests harder to write since you need to know exactly what you’re doing. That’s why many people believe the first kind of test (the 500’000 lines one) is “better”. They allow you to be sloppy, cut corners. Writing good unit tests takes some experience, code that is written to be tested and enough time to understand what you want to test, how you can test what you want and how much you want to test (which is less related to “code coverage” than most people believe).

So my approach would be to not use any Autobahn code in the application. Instead, wrap all this code in a helper class which encapsulates the whole communication.

In your unit tests, you can create a mock of this helper. That way, you can simulate any WAMP-based service in your tests, you can simulate errors, exceptions and corner cases easily.

And best of all: Your test API is stable even when WAMP changes. WAMP does a major API change? Just change the single communication wrapper and the mock and maybe a few tests and you’re done.

Regards,

A. Digulla

···

On Tuesday, September 10, 2013 at 3:47:13 PM UTC+2, Vijay shan wrote:

I am new to autobahn and have some questions around unit testing strategies for a wamp based application that will include the following components:

  1. An Autobahn Wamp server
  2. Multiple python clients that are from remote systems that connect to the server and send remote updates.
  3. Web clients that are used to see the real time data changes.
    I am trying to figure out the best strategy to unit test these different components. I will be using pub/sub and rpc in different places. Any input from the gurus here would be really appreciated.
0 Likes

#4

I am new to autobahn and have some questions around unit testing strategies for a wamp based application that will include the following components:

  1. An Autobahn Wamp server
  2. Multiple python clients that are from remote systems that connect to the server and send remote updates.
  3. Web clients that are used to see the real time data changes.
    I am trying to figure out the best strategy to unit test these different components. I will be using pub/sub and rpc in different places. Any input from the gurus here would be really appreciated.

A few ideas:

Unit tests should be compact and simple. You probably shouldn’t use a WAMP server when running them. The reason is simple: When you run a test which calls some remote server via RPC which fetches some data from the Internet, and it fails, then you have about 500’000 places to check to see why. If the test just tests a single method and in total 20 lines of code are executed, then the problem must be in those 20 lines of code. It should be obvious which test is more stable and easier to fix.

The main problem with this approach is that it makes tests harder to write since you need to know exactly what you’re doing. That’s why many people believe the first kind of test (the 500’000 lines one) is “better”. They allow you to be sloppy, cut corners. Writing good unit tests takes some experience, code that is written to be tested and enough time to understand what you want to test, how you can test what you want and how much you want to test (which is less related to “code coverage” than most people believe).

So my approach would be to not use any Autobahn code in the application. Instead, wrap all this code in a helper class which encapsulates the whole communication.

In your unit tests, you can create a mock of this helper. That way, you can simulate any WAMP-based service in your tests, you can simulate errors, exceptions and corner cases easily.

And best of all: Your test API is stable even when WAMP changes. WAMP does a major API change? Just change the single communication wrapper and the mock and maybe a few tests and you’re done.

The above would be more how I envisage testing of WAMP server components as well. As I’m very much interested in testing for the project I’m doing now (control software for a mineral analysis machine), I’d like to ask: Does anyone know of a project using Autobahn / Crossbar that does rigorous testing?

Since the machine we’re building will be used in mines, essentially 20 hours per day, with little access to it, we have kind of strict requirements on testing for the code that will be running out in the field: 100% line coverage for the unit tests (in addition to any integration/system tests).

Most frameworks I’ve worked with before where testing is non-trivial has at least something in their docs on how to go about it.

Cheers,

Elvis

···

On Monday, February 16, 2015 at 9:46:07 PM UTC+1, Aaron Digulla wrote:

On Tuesday, September 10, 2013 at 3:47:13 PM UTC+2, Vijay shan wrote:

Regards,

A. Digulla

0 Likes

#5

Trial works very well for testing code. This is the tool that is bundled with Twisted. As outlined above, the network code and the application code should be completely separate. Here is one solution.

Application component

class SomeComponent:
def init(self, session):
self.session = session
self.session.set_component(self)

def some_method(self):
    # DO NOT use session.call here
    self.session.make_rpc_call("some_other_method")

Network interface layer

class SomeComponentSession(ApplicationSession):
def set_component(self, component):
self.component = component

@wamp.register("some_uri")
def some_method(self):
    return self.component.some_method()

def make_rpc_call(self, method):
    self.call(method)

Testing harness

class FakeSession:
def set_component(self, component):
self.component = component

def some_method(self):
    return self.component.some_method()

def make_rpc_call(self, method):
    pass

``

Now you assume Autobahn will work as advertised and test only SomeComponent. When you test, instead of passing in an ApplicationSession, pass in a FakeSession object that logs calls. Unfortunately, this adds a level of complexity. The upshot is your application code can be tested independently of what network stack you use.

If you do use this, make sure session.component is set early, or avoid the race condition somehow.

I use a variant of this in production. It works reasonably well.

-Yury

···

On Tuesday, March 24, 2015 at 2:59:42 AM UTC-4, Elvis Stansvik wrote:

On Monday, February 16, 2015 at 9:46:07 PM UTC+1, Aaron Digulla wrote:

On Tuesday, September 10, 2013 at 3:47:13 PM UTC+2, Vijay shan wrote:

I am new to autobahn and have some questions around unit testing strategies for a wamp based application that will include the following components:

  1. An Autobahn Wamp server
  2. Multiple python clients that are from remote systems that connect to the server and send remote updates.
  3. Web clients that are used to see the real time data changes.
    I am trying to figure out the best strategy to unit test these different components. I will be using pub/sub and rpc in different places. Any input from the gurus here would be really appreciated.

A few ideas:

Unit tests should be compact and simple. You probably shouldn’t use a WAMP server when running them. The reason is simple: When you run a test which calls some remote server via RPC which fetches some data from the Internet, and it fails, then you have about 500’000 places to check to see why. If the test just tests a single method and in total 20 lines of code are executed, then the problem must be in those 20 lines of code. It should be obvious which test is more stable and easier to fix.

The main problem with this approach is that it makes tests harder to write since you need to know exactly what you’re doing. That’s why many people believe the first kind of test (the 500’000 lines one) is “better”. They allow you to be sloppy, cut corners. Writing good unit tests takes some experience, code that is written to be tested and enough time to understand what you want to test, how you can test what you want and how much you want to test (which is less related to “code coverage” than most people believe).

So my approach would be to not use any Autobahn code in the application. Instead, wrap all this code in a helper class which encapsulates the whole communication.

In your unit tests, you can create a mock of this helper. That way, you can simulate any WAMP-based service in your tests, you can simulate errors, exceptions and corner cases easily.

And best of all: Your test API is stable even when WAMP changes. WAMP does a major API change? Just change the single communication wrapper and the mock and maybe a few tests and you’re done.

The above would be more how I envisage testing of WAMP server components as well. As I’m very much interested in testing for the project I’m doing now (control software for a mineral analysis machine), I’d like to ask: Does anyone know of a project using Autobahn / Crossbar that does rigorous testing?

Since the machine we’re building will be used in mines, essentially 20 hours per day, with little access to it, we have kind of strict requirements on testing for the code that will be running out in the field: 100% line coverage for the unit tests (in addition to any integration/system tests).

Most frameworks I’ve worked with before where testing is non-trivial has at least something in their docs on how to go about it.

Cheers,

Elvis

Regards,

A. Digulla

0 Likes

#6

Trial works very well for testing code. This is the tool that is bundled with Twisted. As outlined above, the network code and the application code should be completely separate. Here is one solution.

Application component

class SomeComponent:
def init(self, session):
self.session = session
self.session.set_component(self)

def some_method(self):
    # DO NOT use session.call here
    self.session.make_rpc_call("some_other_method")

Network interface layer

class SomeComponentSession(ApplicationSession):
def set_component(self, component):
self.component = component

@wamp.register("some_uri")
def some_method(self):
    return self.component.some_method()

def make_rpc_call(self, method):
    self.call(method)

Testing harness

class FakeSession:
def set_component(self, component):
self.component = component

def some_method(self):
    return self.component.some_method()

def make_rpc_call(self, method):
    pass

``

Now you assume Autobahn will work as advertised and test only SomeComponent. When you test, instead of passing in an ApplicationSession, pass in a FakeSession object that logs calls. Unfortunately, this adds a level of complexity. The upshot is your application code can be tested independently of what network stack you use.

If you do use this, make sure session.component is set early, or avoid the race condition somehow.

I use a variant of this in production. It works reasonably well.

Thanks a lot for the example Yury. I think I’ll experiment some with this approach. Perhaps in some cases I could make use of the unittesting.mock package (or mock package from PyPI if on Python 2) to simplify the mocking.

Elvis

···

On Thursday, March 26, 2015 at 5:04:52 PM UTC+1, Yury Sobolev wrote:

-Yury

On Tuesday, March 24, 2015 at 2:59:42 AM UTC-4, Elvis Stansvik wrote:

On Monday, February 16, 2015 at 9:46:07 PM UTC+1, Aaron Digulla wrote:

On Tuesday, September 10, 2013 at 3:47:13 PM UTC+2, Vijay shan wrote:

I am new to autobahn and have some questions around unit testing strategies for a wamp based application that will include the following components:

  1. An Autobahn Wamp server
  2. Multiple python clients that are from remote systems that connect to the server and send remote updates.
  3. Web clients that are used to see the real time data changes.
    I am trying to figure out the best strategy to unit test these different components. I will be using pub/sub and rpc in different places. Any input from the gurus here would be really appreciated.

A few ideas:

Unit tests should be compact and simple. You probably shouldn’t use a WAMP server when running them. The reason is simple: When you run a test which calls some remote server via RPC which fetches some data from the Internet, and it fails, then you have about 500’000 places to check to see why. If the test just tests a single method and in total 20 lines of code are executed, then the problem must be in those 20 lines of code. It should be obvious which test is more stable and easier to fix.

The main problem with this approach is that it makes tests harder to write since you need to know exactly what you’re doing. That’s why many people believe the first kind of test (the 500’000 lines one) is “better”. They allow you to be sloppy, cut corners. Writing good unit tests takes some experience, code that is written to be tested and enough time to understand what you want to test, how you can test what you want and how much you want to test (which is less related to “code coverage” than most people believe).

So my approach would be to not use any Autobahn code in the application. Instead, wrap all this code in a helper class which encapsulates the whole communication.

In your unit tests, you can create a mock of this helper. That way, you can simulate any WAMP-based service in your tests, you can simulate errors, exceptions and corner cases easily.

And best of all: Your test API is stable even when WAMP changes. WAMP does a major API change? Just change the single communication wrapper and the mock and maybe a few tests and you’re done.

The above would be more how I envisage testing of WAMP server components as well. As I’m very much interested in testing for the project I’m doing now (control software for a mineral analysis machine), I’d like to ask: Does anyone know of a project using Autobahn / Crossbar that does rigorous testing?

Since the machine we’re building will be used in mines, essentially 20 hours per day, with little access to it, we have kind of strict requirements on testing for the code that will be running out in the field: 100% line coverage for the unit tests (in addition to any integration/system tests).

Most frameworks I’ve worked with before where testing is non-trivial has at least something in their docs on how to go about it.

Cheers,

Elvis

Regards,

A. Digulla

0 Likes