Minimal, self-contained Crossbar deployment for embedded systems

#1

First off, please forgive my ignorance about Python.

How would one go about creating a compact, self-contained Crossbar deployment for an embedded Linux-ARM system? Ideally, I’d like to be able to dump a “compiled” Crossbar to the target platform without having to install a bunch of Python libraries. Afterwards, if I want to remotely update to a newer version of Crossbar, I just extract a compressed archive to temporary folder and then swap it out with the old one.

I’d like to somehow bake the Python runtime into the Crossbar deployment. That way, if Crossbar depends on changes to Python itself, then I simply replace the runtime along with Crossbar.

Our WAMP-based product will be a headless network appliance, and we don’t expect our users to update library dependencies on their device. That’s why I want to have a compact Crossbar deployment that can be easily swapped out with a newer one when needed.

Any pointers on how one would go about doing this would be greatly appreciated. If you know of an unrelated Python app that can be deployed in the manner I’m referring to, please let me know.

Cheers,

Emile Cormier

0 Likes

#2

I was thinking about the instructions for the RaspberryPi when I originally posted. The problem I had with those instructions was that all the dependencies were installed “auto-magically” in an apparently non-self-contained manner.

After reading the instructions for installing from source on Ubuntu/Debian, I see that it’s possible to deploy everything in a “virtual environment”. I think this is the self-contained deployment package I’m looking. Whether or not it’s compact remains to be seen.

But a question still remains. For my intended usage, am I better off with CPython or PyPy? Apparently, PyPy has better execution performance, but is there a difference in the total size of the virtual environment? I guess I might have to try both ways and see for myself.

···

On Wednesday, June 1, 2016 at 4:23:03 PM UTC-3, Emile Cormier wrote:

First off, please forgive my ignorance about Python.

How would one go about creating a compact, self-contained Crossbar deployment for an embedded Linux-ARM system? Ideally, I’d like to be able to dump a “compiled” Crossbar to the target platform without having to install a bunch of Python libraries. Afterwards, if I want to remotely update to a newer version of Crossbar, I just extract a compressed archive to temporary folder and then swap it out with the old one.

I’d like to somehow bake the Python runtime into the Crossbar deployment. That way, if Crossbar depends on changes to Python itself, then I simply replace the runtime along with Crossbar.

Our WAMP-based product will be a headless network appliance, and we don’t expect our users to update library dependencies on their device. That’s why I want to have a compact Crossbar deployment that can be easily swapped out with a newer one when needed.

Any pointers on how one would go about doing this would be greatly appreciated. If you know of an unrelated Python app that can be deployed in the manner I’m referring to, please let me know.

Cheers,

Emile Cormier

0 Likes

#3

Hi Emile,

If you use CPython, then you could try something like “bbfreeze” to produce a binary package … not tried it myself … but in terms of a comparison, I’ve recently converted my Crossbar application to “pypy” and it uses a lot more memory. Previously I was running comfortably on a 512M Digital Ocean instance with no swap … for production I’m now considering a 1G instance “with” swap. So in my case, slightly over 2x usage would be a reasonable estimate. Can’t really say how well it performs, “pypy” feels slower, although I know it must be quicker, I suspect the first-time compilation is slower than CPython and I suspect that is what is making is “seem” a little slower … but once I stop reloading every 30 seconds while testing, I expect this issue to evaporate …

But if you’re working on RaspberryPi devices, I would think you’re pretty much going to be stuck on CPython … if you expose yourself to pypy memory usage with your swap on an SD card, speed aside, you’re going to stuff your SD card in fairly short order …

0 Likes

#4

Thanks, Gareth, that information is quite useful! My device is a BeagleBone Black with 512MB RAM, so I guess I’ll stick with CPython. I’ll give bbfreeze a try and will let you know how it works out. I’m not going to tackle this anytime soon; I was just planning ahead.

···

On Thursday, June 2, 2016 at 1:36:06 PM UTC-3, Gareth Bult wrote:

Hi Emile,

If you use CPython, then you could try something like “bbfreeze” to produce a binary package … not tried it myself … but in terms of a comparison, I’ve recently converted my Crossbar application to “pypy” and it uses a lot more memory. Previously I was running comfortably on a 512M Digital Ocean instance with no swap … for production I’m now considering a 1G instance “with” swap. So in my case, slightly over 2x usage would be a reasonable estimate. Can’t really say how well it performs, “pypy” feels slower, although I know it must be quicker, I suspect the first-time compilation is slower than CPython and I suspect that is what is making is “seem” a little slower … but once I stop reloading every 30 seconds while testing, I expect this issue to evaporate …

But if you’re working on RaspberryPi devices, I would think you’re pretty much going to be stuck on CPython … if you expose yourself to pypy memory usage with your swap on an SD card, speed aside, you’re going to stuff your SD card in fairly short order …

0 Likes

#5

Sure, I know things like bbfreeze “sound” good (and may well be if you’re on Windows) but if you’re using Linux, you may find it just as easy (or easier) to build your own package (.rpm/.deb etc) containing the required files and deploy that as opposed to a binary … you won’t be compiling as you develop, so if you deploy post-compiled code, you’re relying completely in your test suite, rather than ongoing developer testing you would normally have with Python apps.

Alternatively (!) I deploy to a raw device using Ansible, taking source files directly from a “git” tag … (I hear purists screaming in the background) but this works well for me and I don’t have to worry about keeping up different package builds for lots of different distributions … just a few extra conditionals in my Ansible scripts.

0 Likes

#6

There already are precompiled deb packages for desktop. Do you know if the build scripts that generate those are available? If so, then perhaps I could tweak them to generate deb packages for my ARM platform.

···

On Thursday, June 2, 2016 at 1:55:18 PM UTC-3, Gareth Bult wrote:

Sure, I know things like bbfreeze “sound” good (and may well be if you’re on Windows) but if you’re using Linux, you may find it just as easy (or easier) to build your own package (.rpm/.deb etc) containing the required files and deploy that as opposed to a binary … you won’t be compiling as you develop, so if you deploy post-compiled code, you’re relying completely in your test suite, rather than ongoing developer testing you would normally have with Python apps.

Alternatively (!) I deploy to a raw device using Ansible, taking source files directly from a “git” tag … (I hear purists screaming in the background) but this works well for me and I don’t have to worry about keeping up different package builds for lots of different distributions … just a few extra conditionals in my Ansible scripts.

0 Likes

#7

Mmm, not sure which bits you mean … my previous project involved Pi’s and Crossbar (door entry system) and I was able to install the whole thing using standard .deb packages and “pip” … (again, all via ansible). So if you’re using Pi’s, then everything you need should be available … minus your actual project code - which you would need to either deploy directly or via your own .deb package … (?)

0 Likes

#8

Hi Emile,

we already provide completely self-contained binary packages of Crossbar.io for Debian/Ubuntu x86-64. These bundle everything but the C run-time.

We might expand that to provide the same for Linux/ARM. Eg deb packages.

In the future, we likely will also provide snaps for Ubuntu Core. You might want to look into Ubuntu Core - the concepts and design makes a lot of sense.

In fact, I think what Ubuntu Core is doing is really the way forward for the "how to update embedded Linux devices and apps" problem.

Cheers,
/Tobias

···

Am 01.06.2016 um 21:23 schrieb Emile Cormier:

First off, please forgive my ignorance about Python.

How would one go about creating a compact, self-contained Crossbar
deployment for an embedded Linux-ARM system? Ideally, I'd like to be able
to dump a "compiled" Crossbar to the target platform without having to
install a bunch of Python libraries. Afterwards, if I want to remotely
update to a newer version of Crossbar, I just extract a compressed archive
to temporary folder and then swap it out with the old one.

I'd like to somehow bake the Python runtime into the Crossbar deployment.
That way, if Crossbar depends on changes to Python itself, then I simply
replace the runtime along with Crossbar.

Our WAMP-based product will be a headless network appliance, and we don't
expect our users to update library dependencies on their device. That's why
I want to have a compact Crossbar deployment that can be easily swapped out
with a newer one when needed.

Any pointers on how one would go about doing this would be greatly
appreciated. If you know of an unrelated Python app that can be deployed in
the manner I'm referring to, please let me know.

Cheers,
Emile Cormier

0 Likes

#9

Thanks, Gareth and Tobias. For some reason, I thought deb packages were tied to a specific architecture. But if the Crossbar deb package only contains Python script code, then it makes that you would be able to install it on the Raspberry Pi.

···

On Thu, Jun 2, 2016 at 2:25 PM, Tobias Oberstein tobias.o...@gmail.com wrote:

First off, please forgive my ignorance about Python.

How would one go about creating a compact, self-contained Crossbar

deployment for an embedded Linux-ARM system? Ideally, I’d like to be able

to dump a “compiled” Crossbar to the target platform without having to

install a bunch of Python libraries. Afterwards, if I want to remotely

update to a newer version of Crossbar, I just extract a compressed archive

to temporary folder and then swap it out with the old one.

I’d like to somehow bake the Python runtime into the Crossbar deployment.

That way, if Crossbar depends on changes to Python itself, then I simply

replace the runtime along with Crossbar.

Our WAMP-based product will be a headless network appliance, and we don’t

expect our users to update library dependencies on their device. That’s why

I want to have a compact Crossbar deployment that can be easily swapped out

with a newer one when needed.

Any pointers on how one would go about doing this would be greatly

appreciated. If you know of an unrelated Python app that can be deployed in

the manner I’m referring to, please let me know.

Cheers,

Emile Cormier

Hi Emile,

we already provide completely self-contained binary packages of Crossbar.io for Debian/Ubuntu x86-64. These bundle everything but the C run-time.

We might expand that to provide the same for Linux/ARM. Eg deb packages.

In the future, we likely will also provide snaps for Ubuntu Core. You might want to look into Ubuntu Core - the concepts and design makes a lot of sense.

In fact, I think what Ubuntu Core is doing is really the way forward for the “how to update embedded Linux devices and apps” problem.

Cheers,

/Tobias

Am 01.06.2016 um 21:23 schrieb Emile Cormier:

You received this message because you are subscribed to a topic in the Google Groups “Crossbar” group.

To unsubscribe from this topic, visit https://groups.google.com/d/topic/crossbario/FWZWJFrmFYU/unsubscribe.

To unsubscribe from this group and all its topics, send an email to crossbario+...@googlegroups.com.

To post to this group, send email to cross...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/crossbario/57506BF7.7050802%40gmail.com.

For more options, visit https://groups.google.com/d/optout.

0 Likes

#10

Hi, they are, for obvious reasons. Python however (and other scripting languages) are generally architecture agnostic and for such occurrences we have an architecture labelled as “noarch”. (this is also used for documentation, “man” pages and the like) If you have a Python application that will run on any system, typically when you build the package, choose “noarch” as the architecture and it will install on any platform.

If you use the stock Ubuntu for Raspberry Pi, it should come with access to “apt” and packages built for ARM architecture, including things like “pip”.

I must admit I didn’t actually run Crossbar “on” a Pi, we used Pi/Autobahn clients with a cloud based Crossbar … but I was sort of expecting “pip” to be able to pull crossbar down and compile it up … (??)

0 Likes

#11

The deb package is arch dependent … as it bundles PyPy and also binary extensions. So what we need is to extend our build infra to produce ARM based debs as well …

···

Sent from Mobile

Am 02.06.2016 21:46 schrieb “Emile Cormier” emile.co...@gmail.com:

Thanks, Gareth and Tobias. For some reason, I thought deb packages were tied to a specific architecture. But if the Crossbar deb package only contains Python script code, then it makes that you would be able to install it on the Raspberry Pi.

On Thu, Jun 2, 2016 at 2:25 PM, Tobias Oberstein tobias.o...@gmail.com wrote:

Hi Emile,

we already provide completely self-contained binary packages of Crossbar.io for Debian/Ubuntu x86-64. These bundle everything but the C run-time.

We might expand that to provide the same for Linux/ARM. Eg deb packages.

In the future, we likely will also provide snaps for Ubuntu Core. You might want to look into Ubuntu Core - the concepts and design makes a lot of sense.

In fact, I think what Ubuntu Core is doing is really the way forward for the “how to update embedded Linux devices and apps” problem.

Cheers,

/Tobias

Am 01.06.2016 um 21:23 schrieb Emile Cormier:

First off, please forgive my ignorance about Python.

How would one go about creating a compact, self-contained Crossbar

deployment for an embedded Linux-ARM system? Ideally, I’d like to be able

to dump a “compiled” Crossbar to the target platform without having to

install a bunch of Python libraries. Afterwards, if I want to remotely

update to a newer version of Crossbar, I just extract a compressed archive

to temporary folder and then swap it out with the old one.

I’d like to somehow bake the Python runtime into the Crossbar deployment.

That way, if Crossbar depends on changes to Python itself, then I simply

replace the runtime along with Crossbar.

Our WAMP-based product will be a headless network appliance, and we don’t

expect our users to update library dependencies on their device. That’s why

I want to have a compact Crossbar deployment that can be easily swapped out

with a newer one when needed.

Any pointers on how one would go about doing this would be greatly

appreciated. If you know of an unrelated Python app that can be deployed in

the manner I’m referring to, please let me know.

Cheers,

Emile Cormier

You received this message because you are subscribed to a topic in the Google Groups “Crossbar” group.

To unsubscribe from this topic, visit https://groups.google.com/d/topic/crossbario/FWZWJFrmFYU/unsubscribe.

To unsubscribe from this group and all its topics, send an email to crossbario+...@googlegroups.com.

To post to this group, send email to cross...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/crossbario/57506BF7.7050802%40gmail.com.

For more options, visit https://groups.google.com/d/optout.

You received this message because you are subscribed to the Google Groups “Crossbar” group.

To unsubscribe from this group and stop receiving emails from it, send an email to crossbario+...@googlegroups.com.

To post to this group, send email to cross...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/crossbario/CAM70yxDCE8jpCYHOie6VcWGEgRGpKOCG5s8hryBpO8AGRrjnig%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

0 Likes

#12

If you do, please provide support for the gcc-arm-linux-gnueabihf architecture.

···

On Friday, June 3, 2016 at 12:07:39 PM UTC-3, Tobias Oberstein wrote:

The deb package is arch dependent … as it bundles PyPy and also binary extensions. So what we need is to extend our build infra to produce ARM based debs as well …

0 Likes

#13

Hi Emile,

The deb package is arch dependent .. as it bundles PyPy and also binary
extensions. So what we need is to extend our build infra to produce ARM
based debs as well ..

If you do, please provide support for the gcc-arm-linux-gnueabi*hf*
architecture.

sure. as far as I know the following holds true anyways:

- neither CPy nor PyPy runs on ARM softfp (ARM11 lacking HW FPU)
- the Pi 3 has a 64 bit ARM core, but a 32 bit userland

since our build system creates a completely self-contained binary packages (it even bundles its own OpenSSL), we only depend on the C run-time.

we can't ship that / paper over diffs in glibc between different Linuxes with incompat. C run-times, and (at least short term) will only be able to provide builds for glibc, not other C run-times ..

cheers,
/Tobias

PS: have you looked at Ubuntu Snappy Core? Opinions?

···

Am 03.06.2016 um 17:12 schrieb Emile Cormier:

On Friday, June 3, 2016 at 12:07:39 PM UTC-3, Tobias Oberstein wrote:

0 Likes

#14

Tobias,

I’ve only looked at Ubuntu Core very briefly. It seems promising; I’ll have to study it further when I have the chance.

For the BeagleBone, the primary distro being supported is Debian. I don’t know how well that platform will run Ubuntu Core. In theory, I should be able to use the same kernel provided with the BeagleBone Debian image, and install the Ubuntu Core distro on top of it.

We still have to decide whether we want to use apt as the mechanism to deploy updates, or roll our own. Our system has to very robust against failed or interrupted updates.

Cheers,

Emile

···

On Friday, June 3, 2016 at 1:41:48 PM UTC-3, Tobias Oberstein wrote:

Hi Emile,

Am 03.06.2016 um 17:12 schrieb Emile Cormier:

On Friday, June 3, 2016 at 12:07:39 PM UTC-3, Tobias Oberstein wrote:

The deb package is arch dependent … as it bundles PyPy and also binary

extensions. So what we need is to extend our build infra to produce ARM

based debs as well …

If you do, please provide support for the gcc-arm-linux-gnueabihf

architecture.

sure. as far as I know the following holds true anyways:

  • neither CPy nor PyPy runs on ARM softfp (ARM11 lacking HW FPU)

  • the Pi 3 has a 64 bit ARM core, but a 32 bit userland

since our build system creates a completely self-contained binary
packages (it even bundles its own OpenSSL), we only depend on the C
run-time.

we can’t ship that / paper over diffs in glibc between different Linuxes
with incompat. C run-times, and (at least short term) will only be able
to provide builds for glibc, not other C run-times …

cheers,

/Tobias

PS: have you looked at Ubuntu Snappy Core? Opinions?

0 Likes