Unable to reliably run commands in tests over SSH backend if commands return error code 255

[azaghal]

When running tests against a remote server over SSH, it is not possible to reliably run tests that involve invocation of commands that may return an error code of 255. E.g. a test like this can not be carried out using the SSH backend:

def test_some_command(host):
    command = host.run("some_command")

    assert command.rc == 255

The ssh command itself will return either an error code of the remotely executed command, or 255 if there was an "internal" error with the SSH itself. Testinfra treats the error code 255 as a special case, and instead raises a runtime exception if encountered. Therefore, if trying to run such commands (that have return code of 255) over the SSH backend, you end-up with Testinfra in a way "masking" the real errors or preventing running of "negative" tests (that try to produce erroneous state and check that the return code is 255, for example).

So far, I've faced this issue while trying to test the following tools:

• ssh
• sieve-connect

In case of the ssh command, I was trying to verify that only public key authentication has been enabled on the OpenSSH server, using the command (yes, probably there is a better way, maybe using something like Paramiko, but this was actually the simplest I could come up with at that time):

ssh -v -o PreferredAuthentications=none -o NoHostAuthenticationForLocalhost=yes localhost

At this point I would basically check for presence of string debug1: Authentications that can continue: publickey in the standard error output to verify that the server is indeed offering just the public key authentication.

The sieve-connect is a small client tool that implements the ManageSieve protocol, and in particular I have used it to verify my Dovecot (providing IMAP/ManageSieve) installation. This tool will return error code 255 in case the authentication fails.

Now, depending on what you want to test, you may or may not care about the return codes for such tools. For example, I could easily see tests being written to validate the OpenSSH server (or even client) configuration that would involve negative tests (where you want to ensure certain types of authentication fails, or maybe that once a user is removed from some group, he/she cannot log-in any longer etc).

As a current workaround, I have resorted to using the construct || /bin/false at the end of command, but this would inevitably mask the real error code from the failing command. Again, depending on what you are doing, you may or may not care about the actual error code. This kind of construct also looks a bit ugly, and I'd feel it would be prone to more mistakes while writing a test.

I'm providing a simple reproduction scenario below to demonstrate the issue on a fresh dedicated machine.

Reproduction steps:

1. Start with a minimal installation of Debian 10 Buster (I've resorted to using Vagrant to speed things up). All reproduction steps should be run on this machine as a non-root user.

2. Install the necessary tools:

sudo apt-get update
sudo apt-get install virtualenvwrapper

3. Log-out and log-in back into the machine (just to activate the virtualenvwrapper).

4. Set-up login over SSH into the local machine and make sure it works correctly:

ssh-keygen -N '' -f ~/.ssh/id_rsa
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
chmod 600 ~/.ssh/authorized_keys
cat /etc/ssh/*.pub | sed -e 's/^/localhost,127.0.0.1,::1 /' >> ~/.ssh/known_hosts
ssh localhost date

5. Create the virtual environment for running Testinfra:

mkdir ~/testinfra-rc-255
mkvirtualenv -a ~/testinfra-rc-255 -p /usr/bin/python3 testinfra
pip install testinfra

6. Create reproduction test file ~/testinfra-rc-255/test_reproduction.py:

def test_can_login_as_same_user(host):
    command = host.run("ssh localhost date")

    assert command.rc == 0

def test_cannot_login_as_root(host):
    command = host.run("ssh root@localhost date")

    assert command.rc == 255

7. Run the test:

py.test --hosts='ssh://localhost'

Expected results:

1. All tests pass in step 7.

Actual results:

1. The test test_can_login_as_same_user in step 7 passes.

2. The test test_cannot_login_as_root in step 7 fails.