Using PHP’s Program Execution Functions for SFTP

If you need to make use of an external program from within a PHP script, then this essay is for you. My example script is for managing an sftp connection (using OpenSSH), but the principals can be applied to any interaction that requires communication between your script and an external process.

One of the Penn Medical School’s business partners recently stopped allowing ftp connections to their servers for retrieving data files. They required us to switch to sftp (secure ftp). Those providing services for transferring sensitive data files over the internet have been steadily moving from ftp to sftp over the past couple of years, and from what I can see, the pace is accelerating. This poses a programming challenge if you have scripts that automate your ftp needs, as they’ll need to be re-written for sftp. This is not a trivial undertaking, especially if you’re programming in PHP. You can’t just swap out your PHP ftp function calls with sftp equivalents. Actually, you can, but you probably don’t want to, as you would have to upgrade to PHP 5 (adoption of which has been very slow across the PHP community) and you would have to install the PECL/ssh2 library, which – as noted on php.net – currently has no stable version.

So we had to roll our own sftp solution, which required using PHP’s program execution functions. The php.net documentation is good on this topic, but much of it is fully comprehensible only if you already know what you’re doing (this isn’t a criticism – it’s a documentation site after all, not a tutorial site). This annotated sample script will help you get started if you’re new to PHP’s program execution functions.

#!/usr/local/bin/php
<?php

$keyPath = 'path/to/your/ssh_key';
$login = 'your_username';
$server = 'your_sftp_server';
$connectionString = "Connecting to $server...\n";

$childPipes = array(
    0 => array("pipe", "r"), // stdin is a pipe that the child will read from
    1 => array("pipe", "w"), // stdout is a pipe that the child will write to
    2 => array("pipe", "w"), // stderr is a pipe that the child will write to
);

# turning off password authentication will avoid getting a password prompt if
# the key fails for any reason
$connection = proc_open(
    "sftp -oPasswordAuthentication=no -oIdentityFile={$keyPath} {$login}@{$server}",
    $childPipes, $parentPipes);

if ($connection === FALSE) {
    print "Cannot connect to $server.\n";
    exit;
}

PHP’s proc_open is a fork by another name. The $childPipes array is for setting up the communication channels from the child process perspective, and proc_open will set $parentPipes to a corresponding set of communication channels from the parent process perspective. Looking at the definition of $childPipes, the logic may seem backwards at first, but it’s not. For example, the parent process will write to the child’s stdin (element 0), which means the child process is reading that channel.

In the user contributed notes on the php.net proc_open page, most folks write out stderr to a file. But for our sftp script we need to see what’s coming through on stderr, so we’re not directing it to a file.

For establishing the connection, we turn off password authentication, which means we won’t get a password prompt if the key authentication fails. This is important, since the script cannot see or respond to such a prompt (the prompt goes directly to the terminal, so you can’t see it on stdin or stdout; you could see it if you want to do TTY buffering, but let’s not go there…).

# The "connecting..." message is written to stderr. Make sure there's nothing
# besides that in stderr before continuing.
$error = readError($parentPipes, TRUE);
sleep(3);
$error .= readError($parentPipes);

if ($error != $connectionString) {
    fclose($parentPipes[0]);
    fclose($parentPipes[1]);
    fclose($parentPipes[2]);
    $closeStatus = proc_close($connection);
    print $error;
    print "proc_close return value: $closeStatus\n";
    exit;
}

I don’t know if this is typical, but the sftp server we’re connecting to returns the “connecting…” welcome message on stderr (we’re reading stderr with a custom function named readError, which we’ll get to below). Having this message on stderr is problematic, since it’s not really an error message. An actual connection error, such as having a bad key, will come through on stderr after the “connecting…” message. This means we first look for the “connecting…” string (the TRUE argument to readError turns blocking on, so we’ll wait for it to appear – more on this below in the readError function), and then we have no choice but to sleep for a few seconds, to see if anything else comes through on stderr. And finally, to see if there was anything in stderr besides the “connecting…” message, we have no choice but to analyze the string 🙁 . This is an ugly solution, but dealing with stderr is difficult, since you never know when an error may or may not appear.

If we detect an error, we close the pipes before closing the connection. This is important for avoiding the possibility of a deadlock.

# gets us past the first "sftp>" prompt
$output = readOut($parentPipes);

After logging in, we’ll get an “sftp>” prompt on stdout. We’ll read from stdout to get past this prompt, using the custom function readOut (which is defined below).

# Get the directory listing and print it
writeIn($parentPipes, "ls -l");
$output .= readOut($parentPipes);
$error = readError($parentPipes);

if (strlen($error)) {
    fclose($parentPipes[0]);
    fclose($parentPipes[1]);
    fclose($parentPipes[2]);
    $closeStatus = proc_close($connection);
    print $error;
    print "proc_close return value: $closeStatus\n";
    exit;
}

print $output;

# close the sftp connection
writeIn($parentPipes, "quit");
fclose($parentPipes[0]);
fclose($parentPipes[1]);
fclose($parentPipes[2]);
$closeStatus = proc_close($connection);

if ($closeStatus != 0) {
    print "proc_close return value: $closeStatus\n";
}

This code just demonstrates getting a directory listing (using the custom function writeIn), printing it, and then closing the connection. You can use this as a template for any sftp commands you want to run.

function readOut($pipes, $end = 'sftp> ', $length = 1024) {
    stream_set_blocking($pipes[1], FALSE);

    while (!feof($pipes[1])) {
        $buffer = fgets($pipes[1], $length);
        $returnValue .= $buffer;

        if (substr_count($buffer, $end) > 0) {
            $pipes[1] = "" ;
            break;
        }
    }

    return $returnValue;
}

readOut loops over the stdout pipe until it sees an “sftp>” prompt, which is how we know that the server has finished writing to stdout. Note that we’ve turned off stream_set_blocking. This lets us define our own controls for reading from the stdout stream. In this case, we want readOut to return when the server has finished responding to a command. The best marker for that is the appearance of the “sftp>” after it finishes processing a command, so we set the while loop to break when it sees the prompt.

function readError($pipes, $blocking = FALSE, $length = 1024) {
    stream_set_blocking($pipes[2], $blocking);

    while (!feof($pipes[2])) {
        $buffer = fgets($pipes[2], $length);
        $returnValue .= $buffer;

        if ((!strlen($buffer) && $blocking === FALSE)
          || ($blocking === TRUE && substr_count($buffer, "\n") > 0)) {
            $pipes[2] = "" ;
            break;
        }
    }

    return $returnValue;
}

function writeIn($pipes, $string) {
    fwrite($pipes[0], $string . "\n");
}
?>

Reading from stderr is more complicated than reading from stdout, because 1. there is no equivalent to the “sftp>” prompt to let us know when the server is done writing to stdout, and 2. at any given time, there may or may not be an error. For most places in the script, we solve this problem by:

  1. Calling readError after calling readOut. This is based on the supposition – which has proved reliable – that the server will finish writing to stderr by the time it has finished writing to stdout.
  2. Setting stream_set_blocking to false. If we set it to true, the script would wait indefinitely for something to appear on stderr, and most of the time there will be nothing there.

The one situation when this approach doesn’t work is when we first log in, since the server writes to stderr before writing to stdout (as described above, it sends that “connecting…” message on stderr). In this case we turn blocking on, since we know the message is coming.

So far this script has been used with only one sftp server, so you may need to make some adjustments to make it work with your server (particularly with how it reads stderr when logging in). Also, I’d be interested in hearing from anyone who has a more elegant solution to handling the initial connection.

Stepping back from the specifics of sftp, the key thing to take away from this is that you will need to acquire a detailed knowledge of the behaviors of your external process so that your script can interact with it reliably. In particular, you need to test your handling of all the different kinds of errors the external process might throw at your script.

7 Comments

  1. Reply
    Paulus November 14, 2006

    Thank you very much for this article. These days I am relying far too much on Google’s first couple of result pages, but luckily I found this.

    Upgrading to PHP5 or installing the PECL/ssh2 library was not an option for me either so this is perfect.

    Just now receiving the error: “Couldn’t read packet: Connection reset by peer”. But I have now given up on SFTP via PHP.

  2. Reply
    Jared May 23, 2007

    Thanks for this example.

    One nitpick, why not create a function to handle the closing?

  3. Reply
    Jay Laseman June 29, 2007

    In your example, you have “#!/usr/local/bin/phpsh”

    Is this the interactive phpsh shell from phpsh.org?

    I ran your script on my system and am having problems.
    My sftp and ssh work fine from the command line so I know my keys ae good. But when I run the script I get prompted for a password ( I took the -o no password option out to see what was happening) I am running RH Core 5 with php 5.1.6

  4. Reply
    Mike June 30, 2007

    Hi Jay,

    It’s this version of phpsh:

    PHP 4.4.0 (cli) (built: Oct 4 2005 13:56:49)
    Copyright (c) 1997-2004 The PHP Group
    Zend Engine v1.3.0, Copyright (c) 1998-2004 Zend Technologies

    Have you confirmed that the path to your key file is correct in the script (especially if you’re using a relative path)? My example script could be improved with a file_exists check on $keyPath.

    If your key file path is fine, and you put the -o option back in, which error message do you get? I’m wondering if you’re failing to connect to the server, or if your connecting but getting messages from the server thta my script isn’t expecting.

    Mike T

  5. Reply
    Christian February 27, 2008

    Great read. Great code. Thanks!!

  6. Reply
    arkascha August 18, 2008

    Hello,
    just stumbled over this description when googling for some hints concerning a problem of mine. ..
    I used a similar solution some three now that worked flawless. The difference is only that I used select-calls instead of a sleep/polling strategy (performance) and that my solution can get along with an interactive authentification if required. This means you can use a password OR a key, the key with or without a passphrase. The whole solution is some kind of generic sftp shell in php that can be used interactive by a script.

    The strange thing is:
    The solution stopped working on newer systems (openSSH >= 4.2 and php4/php5) and I cannot fix it. So I want to ask if you have run into similar problems with your script:
    whyever, I only get the initial prompt back from the cli sftp when I first do *something* on the stdin of the client, like for example I send a linebreak. If I just wait for the prompt I run into a socket timeout and thats it.
    I cannot find any relevant changes in the ssh/sftp implementation, I am not aware of any basic changes in socket- or process handling in php, so I ask myself where the difference is ?
    Do you still use that solution ?

    arkascha

  7. Reply
    Mike August 24, 2008

    Hi arkascha – sorry for the delay writing back. Unfortunately, I haven’t looked at this code in ages. I’m no longer involved with the project I wrote it for. However, I will visit with the team currently responsible for it next week. I know they’ve done a PHP upgrade since I wrote it. I’ll find out if that caused any problems for them and let you know.

Leave a Reply