AutobahnPython + multiprocessing

#1

Hello,
I have WAMP application which connects to crossbar.io route. WAMP application is running with asyncio which is pretty cool for I/O delays, but it is still single threaded. Is it possible to make my WAMP app running an many cores? One eventloop per core. So it will run ok for I/O bound task and CPU-bound too. Any example? Thanks

0 Likes

#2

Actually I need many independent workers in separate processes

···

понедельник, 15 февраля 2016 г., 10:12:43 UTC+2 пользователь Oleg Korsak написал:

Hello,
I have WAMP application which connects to crossbar.io route. WAMP application is running with asyncio which is pretty cool for I/O delays, but it is still single threaded. Is it possible to make my WAMP app running an many cores? One eventloop per core. So it will run ok for I/O bound task and CPU-bound too. Any example? Thanks

0 Likes

#3

Hi Oleg,

Crossbar.io already has a multi-process / worker architecture

http://crossbar.io/docs/Architecture/

Not everything is implemented already, but eg here is how to scale (wrt to CPU) a worker on multiple processes

https://github.com/crossbario/crossbarexamples/tree/master/sharedregs/python

and here is how the Web service built into Crossbar.io scales on multi-core

https://github.com/crossbario/crossbarexamples/tree/master/benchmark/web

We are working on scaling router workers on multi-core too!

Cheers,
/Tobias

···

Am 15.02.2016 um 09:27 schrieb Oleg Korsak:

Actually I need many independent workers in separate processes

понедельник, 15 февраля 2016 г., 10:12:43 UTC+2 пользователь Oleg Korsak > написал:

    Hello,
    I have WAMP application which connects to crossbar.io
    <http://crossbar.io> route. WAMP application is running with asyncio
    which is pretty cool for I/O delays, but it is still single
    threaded. Is it possible to make my WAMP app running an many cores?
    One eventloop per core. So it will run ok for I/O bound task and
    CPU-bound too. Any example? Thanks

--
You received this message because you are subscribed to the Google
Groups "Autobahn" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to autobahnws+...@googlegroups.com
<mailto:autobahnws+...@googlegroups.com>.
To post to this group, send email to autob...@googlegroups.com
<mailto:autob...@googlegroups.com>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/autobahnws/9114c821-f788-4546-b515-833c44c212de%40googlegroups.com
<https://groups.google.com/d/msgid/autobahnws/9114c821-f788-4546-b515-833c44c212de%40googlegroups.com?utm_medium=email&utm_source=footer>.
For more options, visit https://groups.google.com/d/optout.

0 Likes

#4

Thanks for an examples, but I’m talking here more about WAMP component scaling. These examples assume that crossbar launches components that are stored nearby. In my case WAMP app is connecting to crossbar and registers it’s “endpoints”. They even run different Python versions. So crossbar router and WAMP app are different and separate instances. And I want to run wamp app on multiple cores

···

2016-02-15 11:38 GMT+02:00 Tobias Oberstein tobias.o...@gmail.com:

Hi Oleg,

Crossbar.io already has a multi-process / worker architecture

http://crossbar.io/docs/Architecture/

Not everything is implemented already, but eg here is how to scale (wrt to CPU) a worker on multiple processes

https://github.com/crossbario/crossbarexamples/tree/master/sharedregs/python

and here is how the Web service built into Crossbar.io scales on multi-core

https://github.com/crossbario/crossbarexamples/tree/master/benchmark/web

We are working on scaling router workers on multi-core too!

Cheers,

/Tobias

Am 15.02.2016 um 09:27 schrieb Oleg Korsak:

Actually I need many independent workers in separate processes

понедельник, 15 февраля 2016 г., 10:12:43 UTC+2 пользователь Oleg Korsak

написал:

Hello,

I have WAMP application which connects to [crossbar.io](http://crossbar.io)

<[http://crossbar.io](http://crossbar.io)> route. WAMP application is running with asyncio

which is pretty cool for I/O delays, but it is still single

threaded. Is it possible to make my WAMP app running an many cores?

One eventloop per core. So it will run ok for I/O bound task and

CPU-bound too. Any example? Thanks

You received this message because you are subscribed to the Google

Groups “Autobahn” group.

To unsubscribe from this group and stop receiving emails from it, send

an email to autobahnws+...@googlegroups.com

mailto:autobahnws+...@googlegroups.com.

To post to this group, send email to autob...@googlegroups.com

mailto:autob...@googlegroups.com.

To view this discussion on the web visit

https://groups.google.com/d/msgid/autobahnws/9114c821-f788-4546-b515-833c44c212de%40googlegroups.com

<https://groups.google.com/d/msgid/autobahnws/9114c821-f788-4546-b515-833c44c212de%40googlegroups.com?utm_medium=email&utm_source=footer>.

For more options, visit https://groups.google.com/d/optout.

You received this message because you are subscribed to a topic in the Google Groups “Autobahn” group.

To unsubscribe from this topic, visit https://groups.google.com/d/topic/autobahnws/6awZsc8bqXw/unsubscribe.

To unsubscribe from this group and all its topics, send an email to autobahnws+...@googlegroups.com.

To post to this group, send email to autob...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/autobahnws/56C19CA0.2010903%40gmail.com.

For more options, visit https://groups.google.com/d/optout.

0 Likes

#5

Thanks for an examples, but I'm talking here more about WAMP component
scaling. These examples assume that crossbar launches components that
are stored nearby. In my case WAMP app is connecting to crossbar and
registers it's "endpoints". They even run different Python versions. So
crossbar router and WAMP app are different and separate instances. And I
want to run wamp app on multiple cores

Sure. It's not required that Crossbar.io starts your components to use "shared registrations".

Just start multiple instances of your component and use shared regs.

Crossbar.io does not care _how_ your components are started, and what languages/run-times they use. You can even mix (!) languages and have 1 procedure being registered (shared) by components in different languages/run-times.

Cheers,
/Tobias

···

Am 15.02.2016 um 10:51 schrieb Oleg Korsak:

2016-02-15 11:38 GMT+02:00 Tobias Oberstein <tobias.o...@gmail.com
<mailto:tobias.o...@gmail.com>>:

    Hi Oleg,

    Crossbar.io already has a multi-process / worker architecture

    http://crossbar.io/docs/Architecture/

    Not everything is implemented already, but eg here is how to scale
    (wrt to CPU) a worker on multiple processes

    https://github.com/crossbario/crossbarexamples/tree/master/sharedregs/python

    and here is how the Web service built into Crossbar.io scales on
    multi-core

    https://github.com/crossbario/crossbarexamples/tree/master/benchmark/web

    We are working on scaling router workers on multi-core too!

    Cheers,
    /Tobias

    Am 15.02.2016 um 09:27 schrieb Oleg Korsak:

        Actually I need many independent workers in separate processes

        понедельник, 15 февраля 2016 г., 10:12:43 UTC+2 пользователь
        Oleg Korsak
        написал:

             Hello,
             I have WAMP application which connects to crossbar.io
        <http://crossbar.io>
             <http://crossbar.io> route. WAMP application is running
        with asyncio
             which is pretty cool for I/O delays, but it is still single
             threaded. Is it possible to make my WAMP app running an
        many cores?
             One eventloop per core. So it will run ok for I/O bound
        task and
             CPU-bound too. Any example? Thanks

        --
        You received this message because you are subscribed to the Google
        Groups "Autobahn" group.
        To unsubscribe from this group and stop receiving emails from
        it, send
        an email to autobahnws+...@googlegroups.com
        <mailto:autobahnws%2...@googlegroups.com>
        <mailto:autobahnws+...@googlegroups.com
        <mailto:autobahnws%2...@googlegroups.com>>.
        To post to this group, send email to autob...@googlegroups.com
        <mailto:autob...@googlegroups.com>
        <mailto:autob...@googlegroups.com
        <mailto:autob...@googlegroups.com>>.
        To view this discussion on the web visit
        https://groups.google.com/d/msgid/autobahnws/9114c821-f788-4546-b515-833c44c212de%40googlegroups.com
        <https://groups.google.com/d/msgid/autobahnws/9114c821-f788-4546-b515-833c44c212de%40googlegroups.com?utm_medium=email&utm_source=footer>.
        For more options, visit https://groups.google.com/d/optout.

    --
    You received this message because you are subscribed to a topic in
    the Google Groups "Autobahn" group.
    To unsubscribe from this topic, visit
    https://groups.google.com/d/topic/autobahnws/6awZsc8bqXw/unsubscribe.
    To unsubscribe from this group and all its topics, send an email to
    autobahnws+...@googlegroups.com
    <mailto:autobahnws%2...@googlegroups.com>.
    To post to this group, send email to autob...@googlegroups.com
    <mailto:autob...@googlegroups.com>.
    To view this discussion on the web visit
    https://groups.google.com/d/msgid/autobahnws/56C19CA0.2010903%40gmail.com.
    For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "Autobahn" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to autobahnws+...@googlegroups.com
<mailto:autobahnws+...@googlegroups.com>.
To post to this group, send email to autob...@googlegroups.com
<mailto:autob...@googlegroups.com>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/autobahnws/CALGa3ynr-caTRcP425-O-x4zVsedThXgXFZWSubhgtdBTmk2eQ%40mail.gmail.com
<https://groups.google.com/d/msgid/autobahnws/CALGa3ynr-caTRcP425-O-x4zVsedThXgXFZWSubhgtdBTmk2eQ%40mail.gmail.com?utm_medium=email&utm_source=footer>.
For more options, visit https://groups.google.com/d/optout.

0 Likes

#6

Yeah! That’s an amazing feature! Thanks! :slight_smile:

···

2016-02-15 12:00 GMT+02:00 Tobias Oberstein tobias.o...@gmail.com:

Thanks for an examples, but I’m talking here more about WAMP component

scaling. These examples assume that crossbar launches components that

are stored nearby. In my case WAMP app is connecting to crossbar and

registers it’s “endpoints”. They even run different Python versions. So

crossbar router and WAMP app are different and separate instances. And I

want to run wamp app on multiple cores
Am 15.02.2016 um 10:51 schrieb Oleg Korsak:

Sure. It’s not required that Crossbar.io starts your components to use “shared registrations”.

Just start multiple instances of your component and use shared regs.

Crossbar.io does not care how your components are started, and what languages/run-times they use. You can even mix (!) languages and have 1 procedure being registered (shared) by components in different languages/run-times.

Cheers,

/Tobias

2016-02-15 11:38 GMT+02:00 Tobias Oberstein <tobias.o...@gmail.com

mailto:tobias.o...@gmail.com>:

Hi Oleg,



Crossbar.io already has a multi-process / worker architecture



[http://crossbar.io/docs/Architecture/](http://crossbar.io/docs/Architecture/)



Not everything is implemented already, but eg here is how to scale

(wrt to CPU) a worker on multiple processes



[https://github.com/crossbario/crossbarexamples/tree/master/sharedregs/python](https://github.com/crossbario/crossbarexamples/tree/master/sharedregs/python)



and here is how the Web service built into Crossbar.io scales on

multi-core



[https://github.com/crossbario/crossbarexamples/tree/master/benchmark/web](https://github.com/crossbario/crossbarexamples/tree/master/benchmark/web)



We are working on scaling router workers on multi-core too!



Cheers,

/Tobias



Am 15.02.2016 um 09:27 schrieb Oleg Korsak:



    Actually I need many independent workers in separate processes



    понедельник, 15 февраля 2016 г., 10:12:43 UTC+2 пользователь

    Oleg Korsak

    написал:



         Hello,

         I have WAMP application which connects to [crossbar.io](http://crossbar.io)

    <[http://crossbar.io](http://crossbar.io)>

         <[http://crossbar.io](http://crossbar.io)> route. WAMP application is running

    with asyncio

         which is pretty cool for I/O delays, but it is still single

         threaded. Is it possible to make my WAMP app running an

    many cores?

         One eventloop per core. So it will run ok for I/O bound

    task and

         CPU-bound too. Any example? Thanks



    --

    You received this message because you are subscribed to the Google

    Groups "Autobahn" group.

    To unsubscribe from this group and stop receiving emails from

    it, send

    an email to autobahnws+...@googlegroups.com

mailto:autobahnws%2...@googlegroups.com

    <mailto:autobahnws+...@googlegroups.com

    <mailto:autobahnws%2...@googlegroups.com>>.

    To post to this group, send email to autob...@googlegroups.com

    <mailto:autob...@googlegroups.com>

    <mailto:autob...@googlegroups.com

    <mailto:autob...@googlegroups.com>>.

    To view this discussion on the web visit

    [https://groups.google.com/d/msgid/autobahnws/9114c821-f788-4546-b515-833c44c212de%40googlegroups.com](https://groups.google.com/d/msgid/autobahnws/9114c821-f788-4546-b515-833c44c212de%40googlegroups.com)

    <[https://groups.google.com/d/msgid/autobahnws/9114c821-f788-4546-b515-833c44c212de%40googlegroups.com?utm_medium=email&utm_source=footer](https://groups.google.com/d/msgid/autobahnws/9114c821-f788-4546-b515-833c44c212de%40googlegroups.com?utm_medium=email&utm_source=footer)>.

    For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).





--

You received this message because you are subscribed to a topic in

the Google Groups "Autobahn" group.

To unsubscribe from this topic, visit

[https://groups.google.com/d/topic/autobahnws/6awZsc8bqXw/unsubscribe](https://groups.google.com/d/topic/autobahnws/6awZsc8bqXw/unsubscribe).

To unsubscribe from this group and all its topics, send an email to

autobahnws+...@googlegroups.com

<mailto:autobahnws%2...@googlegroups.com>.

To post to this group, send email to autob...@googlegroups.com

<mailto:autob...@googlegroups.com>.

To view this discussion on the web visit

[https://groups.google.com/d/msgid/autobahnws/56C19CA0.2010903%40gmail.com](https://groups.google.com/d/msgid/autobahnws/56C19CA0.2010903%40gmail.com).

For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

You received this message because you are subscribed to the Google

Groups “Autobahn” group.

To unsubscribe from this group and stop receiving emails from it, send

an email to autobahnws+...@googlegroups.com

mailto:autobahnws+...@googlegroups.com.

To post to this group, send email to autob...@googlegroups.com

mailto:autob...@googlegroups.com.

To view this discussion on the web visit

https://groups.google.com/d/msgid/autobahnws/CALGa3ynr-caTRcP425-O-x4zVsedThXgXFZWSubhgtdBTmk2eQ%40mail.gmail.com

<https://groups.google.com/d/msgid/autobahnws/CALGa3ynr-caTRcP425-O-x4zVsedThXgXFZWSubhgtdBTmk2eQ%40mail.gmail.com?utm_medium=email&utm_source=footer>.

For more options, visit https://groups.google.com/d/optout.

You received this message because you are subscribed to a topic in the Google Groups “Autobahn” group.

To unsubscribe from this topic, visit https://groups.google.com/d/topic/autobahnws/6awZsc8bqXw/unsubscribe.

To unsubscribe from this group and all its topics, send an email to autobahnws+...@googlegroups.com.

To post to this group, send email to autob...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/autobahnws/56C1A1D5.2020401%40gmail.com.

For more options, visit https://groups.google.com/d/optout.

0 Likes

#7

At least for Twisted, that doesn't blend well will stdlib multiprocessing - and it's not needed anyway.

Simply fork your workers, and each will run a separate event loop.

On Twisted:

https://twistedmatrix.com/documents/current/api/twisted.internet.utils.getProcessOutput.html

with

excutable=sys.executable
args=['myworker.py']

where myworker.py would fire up your component using ApplicationRunner.

···

Am 16.02.2016 um 11:16 schrieb Oleg Korsak:

ok, looks like I've found a way to fix that:
https://bitbucket.org/snippets/oleg_korsak/5rRGr/revisions/

вторник, 16 февраля 2016 г., 12:10:59 UTC+2 пользователь Oleg Korsak > написал:

    Here is my playground:
    https://bitbucket.org/snippets/oleg_korsak/5rRGr
    <https://bitbucket.org/snippets/oleg_korsak/5rRGr>
    While it still spawns multiple processes and they handle requests -
    I get random count of following errors on startup:
    Exception in callback BaseSelectorEventLoop._sock_connect_cb(<Future
    finished result=None>, <socket.socke...6.245', 8090)>,
    ('144.76.226.245', 8090))
    handle: <Handle BaseSelectorEventLoop._sock_connect_cb(<Future
    finished result=None>, <socket.socke...6.245', 8090)>,
    ('144.76.226.245', 8090))>
    Traceback (most recent call last):
       File "/usr/lib64/python3.5/asyncio/events.py", line 125, in _run
         self._callback(*self._args)
       File "/usr/lib64/python3.5/asyncio/selector_events.py", line 443,
    in _sock_connect_cb
         fut.set_result(None)
       File "/usr/lib64/python3.5/asyncio/futures.py", line 329, in
    set_result
         raise InvalidStateError('{}: {!r}'.format(self._state, self))
    asyncio.futures.InvalidStateError: FINISHED: <Future finished
    result=None>

    And I'm not sure that asyncio`s event loops are separate and being
    handled correctly in all processes in my case.
    Does this code (
    http://autobahn.ws/python/_modules/autobahn/asyncio/wamp.html#ApplicationRunner
    <http://autobahn.ws/python/_modules/autobahn/asyncio/wamp.html#ApplicationRunner>
    ) mean that there is only one global event-loop which fails to be
    used in multiple processes? (I'm weak at asyncio stuff)

    понедельник, 15 февраля 2016 г., 16:46:30 UTC+2 пользователь Oleg > Korsak написал:

        Yeah! That's an amazing feature! Thanks! :slight_smile:

        2016-02-15 12:00 GMT+02:00 Tobias Oberstein
        <tobias.o...@gmail.com <mailto:tobias.o...@gmail.com>>:

            Am 15.02.2016 um 10:51 schrieb Oleg Korsak:

                Thanks for an examples, but I'm talking here more about
                WAMP component
                scaling. These examples assume that crossbar launches
                components that
                are stored nearby. In my case WAMP app is connecting to
                crossbar and
                registers it's "endpoints". They even run different
                Python versions. So
                crossbar router and WAMP app are different and separate
                instances. And I
                want to run wamp app on multiple cores

            Sure. It's not required that Crossbar.io starts your
            components to use "shared registrations".

            Just start multiple instances of your component and use
            shared regs.

            Crossbar.io does not care _how_ your components are started,
            and what languages/run-times they use. You can even mix (!)
            languages and have 1 procedure being registered (shared) by
            components in different languages/run-times.

            Cheers,
            /Tobias

                2016-02-15 11:38 GMT+02:00 Tobias Oberstein
                <tobias.o...@gmail.com
                <mailto:tobias.o...@gmail.com>
                <mailto:tobias.o...@gmail.com
                <mailto:tobias.o...@gmail.com>>>:

                     Hi Oleg,

                     Crossbar.io already has a multi-process / worker
                architecture

                http://crossbar.io/docs/Architecture/
                <http://crossbar.io/docs/Architecture/>

                     Not everything is implemented already, but eg here
                is how to scale
                     (wrt to CPU) a worker on multiple processes

                https://github.com/crossbario/crossbarexamples/tree/master/sharedregs/python
                <https://github.com/crossbario/crossbarexamples/tree/master/sharedregs/python>

                     and here is how the Web service built into
                Crossbar.io scales on
                     multi-core

                https://github.com/crossbario/crossbarexamples/tree/master/benchmark/web
                <https://github.com/crossbario/crossbarexamples/tree/master/benchmark/web>

                     We are working on scaling router workers on
                multi-core too!

                     Cheers,
                     /Tobias

                     Am 15.02.2016 um 09:27 schrieb Oleg Korsak:

                         Actually I need many independent workers in
                separate processes

                         понедельник, 15 февраля 2016 г., 10:12:43 UTC+2
                пользователь
                         Oleg Korsak
                         написал:

                              Hello,
                              I have WAMP application which connects to
                crossbar.io <http://crossbar.io>
                         <http://crossbar.io>
                              <http://crossbar.io> route. WAMP
                application is running
                         with asyncio
                              which is pretty cool for I/O delays, but
                it is still single
                              threaded. Is it possible to make my WAMP
                app running an
                         many cores?
                              One eventloop per core. So it will run ok
                for I/O bound
                         task and
                              CPU-bound too. Any example? Thanks

                         --
                         You received this message because you are
                subscribed to the Google
                         Groups "Autobahn" group.
                         To unsubscribe from this group and stop
                receiving emails from
                         it, send
                         an email to
                autobahnws+...@googlegroups.com
                <mailto:autobahnws%2...@googlegroups.com>

                <mailto:autobahnws%2...@googlegroups.com
                <mailto:autobahnws%25...@googlegroups.com>>
                         <mailto:autobahnws+...@googlegroups.com
                <mailto:autobahnws%2...@googlegroups.com>

                <mailto:autobahnws%2...@googlegroups.com
                <mailto:autobahnws%25...@googlegroups.com>>>.
                         To post to this group, send email to
                autob...@googlegroups.com
                <mailto:autob...@googlegroups.com>
                         <mailto:autob...@googlegroups.com
                <mailto:autob...@googlegroups.com>>
                         <mailto:autob...@googlegroups.com
                <mailto:autob...@googlegroups.com>
                         <mailto:autob...@googlegroups.com
                <mailto:autob...@googlegroups.com>>>.
                         To view this discussion on the web visit
                https://groups.google.com/d/msgid/autobahnws/9114c821-f788-4546-b515-833c44c212de%40googlegroups.com
                <https://groups.google.com/d/msgid/autobahnws/9114c821-f788-4546-b515-833c44c212de%40googlegroups.com>

                <https://groups.google.com/d/msgid/autobahnws/9114c821-f788-4546-b515-833c44c212de%40googlegroups.com?utm_medium=email&utm_source=footer
                <https://groups.google.com/d/msgid/autobahnws/9114c821-f788-4546-b515-833c44c212de%40googlegroups.com?utm_medium=email&utm_source=footer>>.
                         For more options, visit
                https://groups.google.com/d/optout
                <https://groups.google.com/d/optout>.

                     --
                     You received this message because you are
                subscribed to a topic in
                     the Google Groups "Autobahn" group.
                     To unsubscribe from this topic, visit
                https://groups.google.com/d/topic/autobahnws/6awZsc8bqXw/unsubscribe
                <https://groups.google.com/d/topic/autobahnws/6awZsc8bqXw/unsubscribe>.
                     To unsubscribe from this group and all its topics,
                send an email to
                autobahnws+...@googlegroups.com
                <mailto:autobahnws%2...@googlegroups.com>
                     <mailto:autobahnws%2...@googlegroups.com
                <mailto:autobahnws%25...@googlegroups.com>>.
                     To post to this group, send email to
                autob...@googlegroups.com
                <mailto:autob...@googlegroups.com>
                     <mailto:autob...@googlegroups.com
                <mailto:autob...@googlegroups.com>>.
                     To view this discussion on the web visit
                https://groups.google.com/d/msgid/autobahnws/56C19CA0.2010903%40gmail.com
                <https://groups.google.com/d/msgid/autobahnws/56C19CA0.2010903%40gmail.com>.
                     For more options, visit
                https://groups.google.com/d/optout
                <https://groups.google.com/d/optout>.

                --
                You received this message because you are subscribed to
                the Google
                Groups "Autobahn" group.
                To unsubscribe from this group and stop receiving emails
                from it, send
                an email to autobahnws+...@googlegroups.com
                <mailto:autobahnws%2...@googlegroups.com>
                <mailto:autobahnws+...@googlegroups.com
                <mailto:autobahnws%2...@googlegroups.com>>.
                To post to this group, send email to
                autob...@googlegroups.com
                <mailto:autob...@googlegroups.com>
                <mailto:autob...@googlegroups.com
                <mailto:autob...@googlegroups.com>>.
                To view this discussion on the web visit
                https://groups.google.com/d/msgid/autobahnws/CALGa3ynr-caTRcP425-O-x4zVsedThXgXFZWSubhgtdBTmk2eQ%40mail.gmail.com
                <https://groups.google.com/d/msgid/autobahnws/CALGa3ynr-caTRcP425-O-x4zVsedThXgXFZWSubhgtdBTmk2eQ%40mail.gmail.com>
                <https://groups.google.com/d/msgid/autobahnws/CALGa3ynr-caTRcP425-O-x4zVsedThXgXFZWSubhgtdBTmk2eQ%40mail.gmail.com?utm_medium=email&utm_source=footer
                <https://groups.google.com/d/msgid/autobahnws/CALGa3ynr-caTRcP425-O-x4zVsedThXgXFZWSubhgtdBTmk2eQ%40mail.gmail.com?utm_medium=email&utm_source=footer>>.
                For more options, visit
                https://groups.google.com/d/optout
                <https://groups.google.com/d/optout>.

            --
            You received this message because you are subscribed to a
            topic in the Google Groups "Autobahn" group.
            To unsubscribe from this topic, visit
            https://groups.google.com/d/topic/autobahnws/6awZsc8bqXw/unsubscribe
            <https://groups.google.com/d/topic/autobahnws/6awZsc8bqXw/unsubscribe>.
            To unsubscribe from this group and all its topics, send an
            email to autobahnws+...@googlegroups.com
            <mailto:autobahnws%2...@googlegroups.com>.
            To post to this group, send email to
            autob...@googlegroups.com
            <mailto:autob...@googlegroups.com>.
            To view this discussion on the web visit
            https://groups.google.com/d/msgid/autobahnws/56C1A1D5.2020401%40gmail.com
            <https://groups.google.com/d/msgid/autobahnws/56C1A1D5.2020401%40gmail.com>.

            For more options, visit https://groups.google.com/d/optout
            <https://groups.google.com/d/optout>.

--
You received this message because you are subscribed to the Google
Groups "Autobahn" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to autobahnws+...@googlegroups.com
<mailto:autobahnws+...@googlegroups.com>.
To post to this group, send email to autob...@googlegroups.com
<mailto:autob...@googlegroups.com>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/autobahnws/f267e0bd-d723-4119-8203-374e7d5fcc9f%40googlegroups.com
<https://groups.google.com/d/msgid/autobahnws/f267e0bd-d723-4119-8203-374e7d5fcc9f%40googlegroups.com?utm_medium=email&utm_source=footer>.
For more options, visit https://groups.google.com/d/optout.

0 Likes