st2 Fix PEM formatting bug when loading private SSH keys in _get_pkey

Summary

When providing RSA private keys (especially in single-line format or missing proper PEM line breaks), the _get_pkey_object method fails with paramiko.SSHException, even for valid keys.

Environment

StackStorm version: 3.8.1
Affected component: st2common/st2common/runners/paramiko_ssh.py key parsing
Python version: 3.8
OS: Ubuntu 24.04.1

Steps to Reproduce

Use an inline RSA private key with no line breaks, such as from an environment variable or API POST.
Execute an SSH action.

Jul 04 '25 05:07 amritupreti

Thanks for opening this issue. I think there needs to be some discussion before proceeding with an implementation. From the linked issue:

Problem

When using _get_pkey_object, the code assumed that the provided private key string was already correctly PEM-formatted. However, in many automation scenarios, users may provide the key in a single line or malformed format.

StackStorm passes key material directly to Paramiko for processing. StackStorm does no key material validation because that's not its responsibility. As such, StackStorm should not be attempting to manipulate user provided key material.

Jul 04 '25 07:07 nzlosh

Thanks for opening this issue. I think there needs to be some discussion before proceeding with an implementation. From the linked issue:

Problem When using _get_pkey_object, the code assumed that the provided private key string was already correctly PEM-formatted. However, in many automation scenarios, users may provide the key in a single line or malformed format.

StackStorm passes key material directly to Paramiko for processing. StackStorm does no key material validation deferring to Paramiko. As such, StackStorm should not be attempting to manipulate user provided key material.

Paramiko from_private_key classmethod requires that key when passed in string is properly formatted. As such st2 currently will not work when you copy/paste the key into the UI. So this is a valid bug and a fix is welcome.

Jul 04 '25 07:07 sulochan

Please provide an example of input for key material that is causing the issue.

Jul 04 '25 07:07 nzlosh

If we paste this key in UI, it will say invalid format. Currently key can only be passed as file.

Jul 04 '25 08:07 amritupreti

Do you have the same problem using the st2 cli passing the key as an argument?

Jul 04 '25 08:07 nzlosh

https://github.com/StackStorm/st2/blob/master/st2common/st2common/runners/paramiko_ssh.py#L744

Passing key as file work. That is a completely different path as given above. The problem lies when you key as KeyMaterial. That it needs to be in proper format for paramiko to use it otherwise it will give invalid key. There are number of issues filed in the repo or on general internet on this.

The issue with not being able to pass key material is that true automation when done by calling the api become difficult/not possible.

Jul 04 '25 08:07 amritupreti

The API works as expected with correctly formatted key material. Here curl is being used to send a core.remote execution:

curl -X POST -H  'Accept-Encoding: gzip, deflate, zstd' -H  'Accept: */*' -H  'Connection: keep-alive' -H  'X-Auth-Token: e55c20d5b5cb44608dce15735d89fa95' -H  'content-type: application/json' --data-binary '{"action": "core.remote", "parameters": { "hosts": "localhost", "username": "toto", "cmd": "date", "private_key": "-----BEGIN OPENSSH PRIVATE KEY-----\nb3BlbnNzaC1rZXktdjEAAAAABG5vbmUAAAAEbm9uZQAAAAAAAAABAAABlwAAAAdzc2gtcn\nNhAAAAAwEAAQAAAYEArFu4W6rSo4X7ehjVmlmoSK8eHKDJuCbNQh5sAQ0bRJU0YIDN0V+4\n8nWax0fENGoH2sYabr5/SgOIfM3urzKuViQ4KU+hwROGewubzVgLiIRWRK4OUEDHNdj6aK\nQUn3j9Y7ThTenMapriiA5LL9H2P1YCOFG320vq+8M/KyL47FewxM3Dt5NhezGrIEUVOPJy\n6lgQ9l3yX3TATgd3ZU8IbKK8Eo/44VpyUbNHymhzGu8DNFifEV3w70m2nCnrqOanPApYJX\nzLdVkvkHutwv7QEwZ6mfPO9A8409Z/yG+QUybRejYCnaGIHoRsu0Bw+rHY3Uoxie+e2ulv\nLXKGf8BxluxsFU2IqqD6OejYTAedtKyir+/1IEEFfceztR6CBudNZDGn+19RdRO6Focn8a\n3v4vlS+IONW0C7j/rLyYaSTWpTuG+2GHjylBF77HLBG2CnL8aSgPLVhmW2zr7JpFQLFgFn\nniSTLLejayiedgONsRSJdCweCnA9zl+0GRnUmk3nAAAFiI1O4q6NTuKuAAAAB3NzaC1yc2\nEAAAGBAKxbuFuq0qOF+3oY1ZpZqEivHhygybgmzUIebAENG0SVNGCAzdFfuPJ1msdHxDRq\nB9rGGm6+f0oDiHzN7q8yrlYkOClPocEThnsLm81YC4iEVkSuDlBAxzXY+mikFJ94/WO04U\n3pzGqa4ogOSy/R9j9WAjhRt9tL6vvDPysi+OxXsMTNw7eTYXsxqyBFFTjycupYEPZd8l90\nwE4Hd2VPCGyivBKP+OFaclGzR8pocxrvAzRYnxFd8O9Jtpwp66jmpzwKWCV8y3VZL5B7rc\nL+0BMGepnzzvQPONPWf8hvkFMm0Xo2Ap2hiB6EbLtAcPqx2N1KMYnvntrpby1yhn/AcZbs\nbBVNiKqg+jno2EwHnbSsoq/v9SBBBX3Hs7UeggbnTWQxp/tfUXUTuhaHJ/Gt7+L5UviDjV\ntAu4/6y8mGkk1qU7hvthh48pQRe+xywRtgpy/GkoDy1YZlts6+yaRUCxYBZ54kkyy3o2so\nnnYDjbEUiXQsHgpwPc5ftBkZ1JpN5wAAAAMBAAEAAAGAGKByUwaxWhQGvodV3dv5o4kB8G\n045UsGAPB/1hadUstO1IsS3Vuj/WC4YMjsS+DqhmPP2wr0/QpMMijqdbX0GvLJ430xS3kS\nufoKpOhznWnrOZz1Bpp63ELIZLMtDywmaFyr5IoHXyQKQg5ox8hkOuB7KLjkYD0UsixPlk\nZE8uAErdoIoO45+3Q5Uz8LrTtXGAzFP65ktybjY0LF2H/WYE9QOAbCNNDLjMjk9RK/f5Wv\nC4euErktd8M8iEulFFkASBUcWn7/eoLSFpdwHcyGZTAVzDt0yc3Lu3Jc1DUUhpMHWwIwkH\nr4BU944xkFI1/BsWarwyAxCXP/tf8XLc/Cgalad348iN//osmbhSPrdRYiIOXSTdnG5BkT\nyf5uuAECVl3OBBTHuB75Hk2lIbw7ibWaV6rupNocx3FVWGRwQPJnMYGgt9bxqliO27OKyO\nNZZEQfpUH5nEOiCdRUaTLBSWz9fmpTqBiSVnWE9eJkZ2hkEg8SuDYOupUMQWCLik6hAAAA\nwQCuNiJ0rsgibMLkH3lMdl9tY/WZs51glG29KI35tHa+Nl+sUPMag/A0sbLtyoPZAF01kM\nevDX8KEihDuwf82BtOd9c8KvaJRNMYteeKuVniXqt7yKPDfLSzq2cW3iepw1JMPyy/U9JH\niQyqHDBPuJ59xyJSMM+E0nTN07svGSQxli7UdVmRTTjWIfE6NSdjXvxMy4t2Yj5UGFB9tM\nYayvrjB62C67q0qEqBFz/BzKLfjtEnjBJjYHZuTRJX9J7E2dsAAADBANbMeNg8bKCNP7m9\nAfI9LoSciagjORU4yN1MGZWR+3Qrr6gsGaj3xQ9L8q//0kNkekOACYsrDpx1dN7+/z4/Pc\nm1KT9nXYiSrKUP+R9GkfGugPwAxkJl12uvamDN5RaJys/IAKO5Zj7/q+N420Bkwqkfc1CK\n9BzYJCjgGrJOG4SqU0sWV8/4QZ3eRR3ecc6ERJZj7VrZcDw+6CnE1nbeludF5eSbiD5j8x\n8MbugQrLg2c8f9KhKgVqBoHKJSk/vSNQAAAMEAzWs+H48NUg34jHN4YoJPKklFk6A4/l7e\n0KWC/4af0OLpcmuTXfwXsUxtudcWAvOPV2a4g2qrUavTik0c1C3BfKAFBt7zRuTX8tvC4t\nhue1JbQAT3br/SKUdr6TIVYaoamahpuVXEXZdbXoDrRjCldp8KDRbvo96KdNNC7E39QL/H\ndKdHD1G8oJ+pS2WoFbAIuEqW5RkZOgrb1h8RwVRcVpWzD2JGPpZUIzcaWx3K8UzgIYtyLu\nfhecGaIa2jZeMrAAAAC2Ftcml0QGFtcml0AQIDBAUGBw==\n-----END OPENSSH PRIVATE KEY-----\n\n"},"user": null}' http://127.0.0.1:9101/v1/executions

which produced the expected execution

st2 execution get 6867ad81cbc31d6a77d3a858
id: 6867ad81cbc31d6a77d3a858
action.ref: core.remote
context.user: st2admin
parameters: 
  cmd: date
  hosts: localhost
  private_key: '********'
  username: toto
status: succeeded (1s elapsed)
start_timestamp: Fri, 04 Jul 2025 10:31:29 UTC
end_timestamp: Fri, 04 Jul 2025 10:31:30 UTC
log: 
  - status: requested
    timestamp: '2025-07-04T10:31:29.445000Z'
  - status: scheduled
    timestamp: '2025-07-04T10:31:29.561000Z'
  - status: running
    timestamp: '2025-07-04T10:31:29.643000Z'
  - status: succeeded
    timestamp: '2025-07-04T10:31:30.072000Z'
result: 
  localhost:
    failed: false
    return_code: 0
    stderr: ''
    stdout: Fri 04 Jul 2025 10:31:29 AM UTC
    succeeded: true

This shows that the ST2 API processes the PEM file correctly when it is encoded as a JSON string. Encoding the PEM file as a JSON string is required since the API expects a JSON payload.

Jul 04 '25 10:07 nzlosh

And there in lies the problem. For a user, the normal behaviour is to copy past the key into the string or the stackstorm UI. The definition of "properly formatted" requires the "/n" at the end, which is not normal user action.

In order to get that curl command to work, did you have to put in the /n at the end as string/take an extra step to json encode it ?

Jul 04 '25 11:07 sulochan

If the webui isn't serialising multi-line input into a valid json encoded string as part of the call to the API then it seems like a bug to me since it's not preserving the original text format. Please open an issue / PR against https://github.com/StackStorm/st2web

Jul 04 '25 13:07 nzlosh

If the webui isn't serialising multi-line input into a valid json encoded string as part of the call to the API then it seems like a bug to me since it's not preserving the original text format. Please open an issue / PR against https://github.com/StackStorm/st2web

Yes, we can fix the StackStorm UI, but the root issue lies on the API side. API consumers should be notified separately, as the problem may go unnoticed on their end until it reaches production. A proper fix needs to be implemented at the API level, not just in the frontend.

Jul 04 '25 13:07 amritupreti

FYI the st2 cli handles the pem file correctly.

st2 run core.remote hosts=localhost cmd='date; date' username=toto private_key='-----BEGIN OPENSSH PRIVATE KEY-----
b3BlbnNzaC1rZXktdjEAAAAABG5vbmUAAAAEbm9uZQAAAAAAAAABAAABlwAAAAdzc2gtcn
NhAAAAAwEAAQAAAYEArFu4W6rSo4X7ehjVmlmoSK8eHKDJuCbNQh5sAQ0bRJU0YIDN0V+4
8nWax0fENGoH2sYabr5/SgOIfM3urzKuViQ4KU+hwROGewubzVgLiIRWRK4OUEDHNdj6aK
QUn3j9Y7ThTenMapriiA5LL9H2P1YCOFG320vq+8M/KyL47FewxM3Dt5NhezGrIEUVOPJy
6lgQ9l3yX3TATgd3ZU8IbKK8Eo/44VpyUbNHymhzGu8DNFifEV3w70m2nCnrqOanPApYJX
zLdVkvkHutwv7QEwZ6mfPO9A8409Z/yG+QUybRejYCnaGIHoRsu0Bw+rHY3Uoxie+e2ulv
LXKGf8BxluxsFU2IqqD6OejYTAedtKyir+/1IEEFfceztR6CBudNZDGn+19RdRO6Focn8a
3v4vlS+IONW0C7j/rLyYaSTWpTuG+2GHjylBF77HLBG2CnL8aSgPLVhmW2zr7JpFQLFgFn
niSTLLejayiedgONsRSJdCweCnA9zl+0GRnUmk3nAAAFiI1O4q6NTuKuAAAAB3NzaC1yc2
EAAAGBAKxbuFuq0qOF+3oY1ZpZqEivHhygybgmzUIebAENG0SVNGCAzdFfuPJ1msdHxDRq
B9rGGm6+f0oDiHzN7q8yrlYkOClPocEThnsLm81YC4iEVkSuDlBAxzXY+mikFJ94/WO04U
3pzGqa4ogOSy/R9j9WAjhRt9tL6vvDPysi+OxXsMTNw7eTYXsxqyBFFTjycupYEPZd8l90
wE4Hd2VPCGyivBKP+OFaclGzR8pocxrvAzRYnxFd8O9Jtpwp66jmpzwKWCV8y3VZL5B7rc
L+0BMGepnzzvQPONPWf8hvkFMm0Xo2Ap2hiB6EbLtAcPqx2N1KMYnvntrpby1yhn/AcZbs
bBVNiKqg+jno2EwHnbSsoq/v9SBBBX3Hs7UeggbnTWQxp/tfUXUTuhaHJ/Gt7+L5UviDjV
tAu4/6y8mGkk1qU7hvthh48pQRe+xywRtgpy/GkoDy1YZlts6+yaRUCxYBZ54kkyy3o2so
nnYDjbEUiXQsHgpwPc5ftBkZ1JpN5wAAAAMBAAEAAAGAGKByUwaxWhQGvodV3dv5o4kB8G
045UsGAPB/1hadUstO1IsS3Vuj/WC4YMjsS+DqhmPP2wr0/QpMMijqdbX0GvLJ430xS3kS
ufoKpOhznWnrOZz1Bpp63ELIZLMtDywmaFyr5IoHXyQKQg5ox8hkOuB7KLjkYD0UsixPlk
ZE8uAErdoIoO45+3Q5Uz8LrTtXGAzFP65ktybjY0LF2H/WYE9QOAbCNNDLjMjk9RK/f5Wv
C4euErktd8M8iEulFFkASBUcWn7/eoLSFpdwHcyGZTAVzDt0yc3Lu3Jc1DUUhpMHWwIwkH
r4BU944xkFI1/BsWarwyAxCXP/tf8XLc/Cgalad348iN//osmbhSPrdRYiIOXSTdnG5BkT
yf5uuAECVl3OBBTHuB75Hk2lIbw7ibWaV6rupNocx3FVWGRwQPJnMYGgt9bxqliO27OKyO
NZZEQfpUH5nEOiCdRUaTLBSWz9fmpTqBiSVnWE9eJkZ2hkEg8SuDYOupUMQWCLik6hAAAA
wQCuNiJ0rsgibMLkH3lMdl9tY/WZs51glG29KI35tHa+Nl+sUPMag/A0sbLtyoPZAF01kM
evDX8KEihDuwf82BtOd9c8KvaJRNMYteeKuVniXqt7yKPDfLSzq2cW3iepw1JMPyy/U9JH
iQyqHDBPuJ59xyJSMM+E0nTN07svGSQxli7UdVmRTTjWIfE6NSdjXvxMy4t2Yj5UGFB9tM
YayvrjB62C67q0qEqBFz/BzKLfjtEnjBJjYHZuTRJX9J7E2dsAAADBANbMeNg8bKCNP7m9
AfI9LoSciagjORU4yN1MGZWR+3Qrr6gsGaj3xQ9L8q//0kNkekOACYsrDpx1dN7+/z4/Pc
m1KT9nXYiSrKUP+R9GkfGugPwAxkJl12uvamDN5RaJys/IAKO5Zj7/q+N420Bkwqkfc1CK
9BzYJCjgGrJOG4SqU0sWV8/4QZ3eRR3ecc6ERJZj7VrZcDw+6CnE1nbeludF5eSbiD5j8x
8MbugQrLg2c8f9KhKgVqBoHKJSk/vSNQAAAMEAzWs+H48NUg34jHN4YoJPKklFk6A4/l7e
0KWC/4af0OLpcmuTXfwXsUxtudcWAvOPV2a4g2qrUavTik0c1C3BfKAFBt7zRuTX8tvC4t
hue1JbQAT3br/SKUdr6TIVYaoamahpuVXEXZdbXoDrRjCldp8KDRbvo96KdNNC7E39QL/H
dKdHD1G8oJ+pS2WoFbAIuEqW5RkZOgrb1h8RwVRcVpWzD2JGPpZUIzcaWx3K8UzgIYtyLu
fhecGaIa2jZeMrAAAAC2Ftcml0QGFtcml0AQIDBAUGBw==
-----END OPENSSH PRIVATE KEY-----
'

which produces the expected execution

id: 6867d75ccbc31d6a77d3a891
action.ref: core.remote
context.user: st2admin
parameters:
  cmd: date; date
  hosts: localhost
  private_key: '********'
  username: toto
status: succeeded
start_timestamp: Fri, 04 Jul 2025 13:30:04 UTC
end_timestamp: Fri, 04 Jul 2025 13:30:04 UTC
result:
  localhost:
    failed: false
    return_code: 0
    stderr: ''
    stdout: 'Fri 04 Jul 2025 01:30:04 PM UTC
      Fri 04 Jul 2025 01:30:04 PM UTC'
    succeeded: true

Jul 04 '25 13:07 nzlosh

And there in lies the problem. For a user, the normal behaviour is to copy past the key into the string or the stackstorm UI. The definition of "properly formatted" requires the "/n" at the end, which is not normal user action.

I need more details please. What "user"? What level of access do they have to st2? Are they copy/pasting keys into the st2 webui or some other interface like through ChatOps? If this is through the st2 web UI, what browser are they using?

What operating system is the user using? (Windows? Mac? Linux?) What flavor/version of operating system? Is their machine configured to use utf-8 or some other encoding/locale? Are they using a desktop/laptop or a mobile device?

Where is the user copying the keys from? Are they using the operating system's clipboard? If they copy a key and paste it into a plain text file, do the line endings get mangled or truncated or does the character encoding change?

Jul 04 '25 21:07 cognifloyd

This is not an API issue, this is a st2web issue, or rather issue with HTML.

Secret strings entered via the st2web are entered through a "password" HTML input type. This input type does not allow line break \n characters. If the key is passed into an input that's not marked as "secret" and therefore not a "password" input type, the line break characters are rendered correctly.

Jul 15 '25 15:07 fdrab

From UI perspective this can be remedied by using textareafield for secrets instead of type password and using webkit-text-security css property on all strings that are secret. This would achieve that whatever multiline string is preserved, while not affecting single line strings at all.

Jul 17 '25 16:07 fdrab

Thanks for the insight @fdrab. It sounds like a good solution to fix the issue. 👍

Jul 17 '25 17:07 nzlosh

Fix PEM formatting bug when loading private SSH keys in _get_pkey_object

Summary

Environment

Steps to Reproduce