Beware of buffered output

I've been playing around with paramiko, a ssh2 implementation for python.  While running commands that quickly executed was easy, I had to solve a different problem.

Basically we have some perl scripts that can be called on a remote machine, and one of the easier ways to do this is through SSH.  So I took a quick look around the web, and it seemed like a lot of people liked paramiko, so I thought I would give it a twirl.  So I did the quick and easy setup that you usually see on other tutorial sites:

import paramiko

client = paramiko.SSHClient()

client.set_missing_host_key_policy(paramiko.AutoAddPolicy())

client.connect("127.0.0.1", username='root_beer', password='yeah_right')

So far so good.  Unfortunately, most of the examples you see on the web involve very quickly executed commands, like 'ls -al', or 'pwd'.  I needed to execute a long (sometimes 16+ hour script).  So if you naively try to do something like:

sin, sout, serr = client.exec_command("perl mylongscript.pl", 0) 

you won't actually see anything that would have printed on the local machine's stdout print out.  In fact, it appears that the command is blocking.  Now, what confused me is that if I ran something like:

=> ssh root_beer@127.0.0.1
=> Password:
=> *********
=> perl mylongscript.pl

I would immediately see the results from stdout.  So I assumed that something in paramiko was blocking, waiting for something.  I had turned off buffering when I executed command.  I even executed python with the -u option (unbuffered).  None of these worked, so then I tried using paramiko's raw Transport and Channel classes.

My first thought was that I would create a socket object, pass that to the Transport's constructor, and then call select on it....something like this:

s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

s.connect(("127.0.0.1", 22))

trans = paramiko.Transport(s)

chan = trans.open_session()

Now that I had a channel and a socket, I thought I could call select on the socket.

while True:

    sockin, sockout, sockerr = select.select([s],[],[])

    for input in sockin:

        print s.recv(120)

But of course, that didn't work either.  I tried setting the channel's blocking to false, and all kinds of other weird things.  I also had to worry about threading.  I needed a separate thread which would grab the process ID of the script I was running, so that when the remote script was done, I could finish up what I was doing in my main thread.  Since my concurrency skills are very rough around the edges, I thought maybe that was messing something up.


Eventually, something dawned on me that should have hit me on the head right away.  About 3 months back, I was writing some Java code that would launch a separate jython process (via the Runtime class's exec() method).  What was tricky was that this jython process could in turn launch other processes, such as a perl script.  That time also, I was never getting the output from the script.  What eventually dawned on me back then was that it wasn't jython or Java that was buffering the output....it was perl.

I eventually came up with an ugly perl hack to turn off buffering.  Basically, I slurped in a file, prepended the '$| = 1;' line which turns off buffering to the scalar that held the slurped in file, and I evaled it.


When I remembered this, I realized that what I was doing with paramiko was unnecessary, and the following was much simpler.  Now, instead of selecting on the socket, I could use the Channel object's recv_ready() function.

while True:

    if chan.recv_ready():

        print chan.recv(120)