I’m very new to Crossbar and Python and quite amazed on what could be done with it. Apologies if I’m asking the question that was asked before, but I tried to use google as much as I can to get an answer to no avail.
Here is my set up - I have a large read-only object that is used as basis for calculations for my workers. I would like to be able to share that object with all of them, there is no need to synchronize or do anything multi-core with it since all workers are using it read only - they just need to wait before it loads everything, I don’t want to use database to keep my blob because it is too time consuming and I have plenty of RAM to accomodate it. I’m using Linux for my development.
So how can I make this large in-memory “blob” be available to workers ?
Have all workers as “guests” that are spawned by a different process -> load large blob in the memory, fork and then register each worker with crossbar. As long as blob is indeed read only, linux should be able share it withe everyone without copying it to each process. Will have to coordinate launching guests and node.
Use memory mapped files, but deal with more basic data structures instead of full python on the “blob”. All workers could be native in Crossbar.io, data sharing is dealt by memory mapped files.
I’m wondering if anyone had any experience with this or if there are some examples that I have missed.