libssh2_userauth methods always indicate error when additional auth steps are required by server

[ion201]

Describe the bug When the ssh server is configured to accept multiple public keys, libssh2 returns an unexpected error for the first unlock attempt.

To Reproduce

1. Configure ssh server with the following option -

AuthenticationMethods publickey,publickey

2. Compile the below example, changing "username", "key1", and"key2" to suit the system. key1 and key2 are unique keys which are both in the user's authorized_keys file.

This file can be compiled with - gcc -g -Wall -Werror ./authtest.c -lssh2 -o ./authtest

#include <libssh2.h>
#include <stdio.h>
#include <sys/socket.h>
#include <netinet/in.h>

int main() {
  const char key1[] = "/home/user/.ssh/id_ed25519";
  const char key2[] = "/home/user/.ssh/id_ed25519_2";
  const char username[] = "user";
  int rc;

  libssh2_socket_t sock = socket(AF_INET, SOCK_STREAM, 0);
  struct sockaddr_in sin;
  sin.sin_family = AF_INET;
  sin.sin_port = htons(22);
  sin.sin_addr.s_addr = htonl(0x7F000001);
  connect(sock, (struct sockaddr*)(&sin), sizeof(struct sockaddr_in));

  libssh2_init(0);

  LIBSSH2_SESSION *session = libssh2_session_init();
  libssh2_session_set_blocking(session, 1);
  libssh2_session_handshake(session, sock);

  rc = libssh2_userauth_publickey_fromfile(session, username, NULL, key1, "");
  printf("key1 auth rc = %d\n", rc);

  rc = libssh2_userauth_publickey_fromfile(session, username, NULL, key2, "");
  printf("key2 auth rc = %d\n", rc);
}

3. Run the example, observe the following output -

$ ./authtest 
key1 auth rc = -19
key2 auth rc = 0

Expected behavior The first call to libssh2_userauth_publickey_fromfile should indicate partially successful authentication. I observe that this authentication request actually did succeed, because the connection is successfully established upon sending the second key, but the indicated error of -19 (LIBSSH2_ERROR_PUBLICKEY_UNVERIFIED) doesn't make any sense in context.

If the auth method is changed to password,publickey (replace first key unlock with libssh2_userauth_password) I similarly observe that the the error reported is indistinguishable from a normal incorrect password error, even though the password was accepted.

pw auth rc = -18
key2 auth rc = 0

Version (please complete the following information):

• OS and version: Ubuntu 24.04
• libssh2 version: 1.11.0-4.1build2
• crypto backend and version: openssl

Additional context For comparison, here is the logging from openssh with the same auth scenario -

debug1: Offering public key: /home/user/.ssh/id_ed25519 ED25519 ... agent
debug1: Server accepts key: /home/user/.ssh/id_ed25519 ED25519 ... agent
Authenticated using "publickey" with partial success.
debug1: Authentications that can continue: publickey
debug1: Offering public key: /home/user/.ssh/id_ed25519_2 RSA ... agent
debug1: Server accepts key: /home/user/.ssh/id_ed25519_2 RSA ... agent
Authenticated to 127.0.0.1 ([127.0.0.1]:22) using "publickey".
debug1: channel 0: new session [client-session] (inactive timeout: 0)

[LarsNordin-LNdata]

libssh2 is not checking the addition data in the SSH_MSG_USERAUTH_FAILURE message, in RFC4552 the USERAUTH_FAILURE can have a payload with additional auth possible AND a flag to indicate partial success.

byte SSH_MSG_USERAUTH_FAILURE string name-list authentications that can continue boolean partial success

This requires a new LIBSSH2 error code LIBSSH2_ERROR_PARTIAL_SUCCESS and changes to userauth.c to check and return this new error code if partial is set.

[Michael-Panic]

Authentication would definitely be easier with a partial success error code. Right now, we have to work around this in our client code by calling libssh2_userauth_authenticated() when the error is LIBSSH2_ERROR_PUBLICKEY_UNVERIFIED. As long as it doesn't return NULL and still contains publickey, we're good to try the next key.

[LarsNordin-LNdata]

I have tested the implementation using one password authentication and two public key, I get LIBSSH2_ERROR_PARTIAL_SUCCESS for the first two and a return of 0 in the third (sshd configured to require this with "AuthenticationMethods password,publickey,publickey"

This is in pull-request #1757

I havm't update the manual, what about:

Return 0 on success or negative on failure. A partial successful authentication returns LIBSSH2_ERROR_PARTIAL_SUCCESS and further authentication is needed. It returns LIBSSH2_ERROR_EAGAIN when it would otherwise block. While LIBSSH2_ERROR_PARTIAL_SUCCESS and LIBSSH2_ERROR_EAGAIN is a negative number, it is not really a failure per se.

[ion201]

The proposal in #1757 looks like exactly what I expected. Grammar aside, I think this suggestion for the docs is sufficient as well.

While LIBSSH2_ERROR_PARTIAL_SUCCESS and LIBSSH2_ERROR_EAGAIN are negative numbers, they are not really failures per se.

[LarsNordin-LNdata]

As this change is considered of breaking of the ABI it could take a while before in gets inte a release. Are you compiling from source and whould like to have this change implemented with an ifdef? I don't no if it's allowed by the maintenders.

[ion201]

Can only speak for myself, but I don't see any need to ifdef this feature to move it along faster. This is easy enough to work around or manually merge as needed. It's had this behavior for so long at this point, a bit more time wouldn't seem to make much difference.