Harvesting data from servers via ssh

#1

Hi,

I’d like to use a client app to tell my server application (this is where I think of using RPC) to start collecting data from other machines…

The server needs to do so by executing (long-running) commands via ssh on those other machines.

I guess I would need something as in twisted-conch, where IIUC, it is possible to invoke a call-back when the command on the other machine has finished.

After that, my server application needs my (and others’) client to show the data obtained from those commands. This is where I think PubSub is playing its role.

Does anyone have some advice about which examples to look at to get started regarding running the ssh commands?

Maybe it’s comparable to executing long-running queries on a (remote) database?

I assumed the arduino-example is not really a fit as obtaining the data from sensors is not long-running (I might be wrong though)?

Thanks in advance,

Roger

0 Likes

#2

Hi Roger,

Hi,

I'd like to use a client app to tell my server application (this is
where I think of using RPC) to start collecting data from other machines..

The server needs to do so by executing (long-running) commands via ssh
on those other machines.
I guess I would need something as in twisted-conch, where IIUC, it is
possible to invoke a call-back when the command on the other machine has
finished.

Exactly. Twisted Conch is a jewel. And of course it's fully asynch., means it integrates seamlessly with Autobahn / WAMP.

After that, my server application needs my (and others') client to show
the data obtained from those commands. This is where I think PubSub is
playing its role.

Using RPC to trigger the start of a long-running operation, and using PubSub events to let that long-running operation notify others of it's progress sounds good.

The other option you might consider is using RPCs with "progressive results".

E.g. the long-running operation is started with an RPC. And that RPC will only finally return when the long-op is completely done. While the long-op is running, it can send progress to the client that issued the RPC.

https://github.com/tavendo/AutobahnPython/blob/master/examples/twisted/wamp/basic/rpc/progress/backend.py#L37

In this case, only the client that triggered start of long-op will get progress information. This is different from the first option of combining RPC+PubSub, where the progress information can be distributed to all interested parties (via PubSub).

Which of above is more appropriate: depends on your requirements / exactly what you want to have.

Does anyone have some advice about which examples to look at to get
started regarding running the ssh commands?

Conch is powerful, but can be difficult to approach. I'd recommend following Jean-Paul's advices here

http://stackoverflow.com/questions/4617507/best-way-to-run-remote-commands-thru-ssh-in-twisted

and probably start from the example here

http://twistedmatrix.com/documents/current/conch/howto/conch_client.html

Maybe it's comparable to executing long-running queries on a (remote)
database?
I assumed the arduino-example is not really a fit as obtaining the data
from sensors is not long-running (I might be wrong though)?

Essentially, you need to "bridge" 2 Twisted Protocol instances:

The one is talking SSH to the command being run and the other is talking WAMP to a WAMP router.

If you go with the RPC+PubSub variant, roughly:

class JobProtocol(Protocol):

    ## this class corresponds to NoiseProtocol example here

···

Am 27.06.2014 01:54, schrieb Roger Erens:
    ## http://twistedmatrix.com/documents/current/conch/howto/conch_client.html

    def connectionMade(self):
       ...

    def dataReceived(self, data):
       ## here we forward data received via SSH as WAMP event
       self.factory.session.publish(...)

class Job:

    def __init__(self, session, jobparams):
        self.session = session

    def run(self):
       endpoint = SSHCommandClientEndpoint(...)
       factory = Factory()
       factory.protocol = JobProtocol
       factory.session = self.session
       return endpoint.connect(factory)

class JobController(ApplicationSession):

    def onJoin(self, details):
       self.jobs = {}

    @wamp.register('com.myapp.start_job')
    def start_job(self, jobparams):
       id = newid()
       job = Job(self, jobparams)
       self.jobs[id] = job
       job.run()
       return id

Hope above is sufficient to get you started ..

Cheers,
/Tobias

Thanks in advance,

Roger

--
You received this message because you are subscribed to the Google
Groups "Autobahn" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to autobahnws+...@googlegroups.com
<mailto:autobahnws+...@googlegroups.com>.
For more options, visit https://groups.google.com/d/optout.

0 Likes