jenkinsci
Jenkins helm values.yaml - failed to pass the basicSSHUserPrivateKey via terraform data.aws_secretsmanager_secret_version
Jenkins and plugins versions report
Environment
Jenkins: 2.332.2
OS: Linux - 5.10.109
---
ace-editor:1.1
ansicolor:1.0.1
antisamy-markup-formatter:2.7
apache-httpcomponents-client-4-api:4.5.13-1.0
authentication-tokens:1.4
aws-credentials:191.vcb_f183ce58b_9
aws-java-sdk:1.12.163-315.v2b_716ec8e4df
aws-java-sdk-cloudformation:1.12.246-349.v96b_b_f7eb_a_c3c
aws-java-sdk-codebuild:1.12.246-349.v96b_b_f7eb_a_c3c
aws-java-sdk-ec2:1.12.246-349.v96b_b_f7eb_a_c3c
aws-java-sdk-ecr:1.12.246-349.v96b_b_f7eb_a_c3c
aws-java-sdk-ecs:1.12.246-349.v96b_b_f7eb_a_c3c
aws-java-sdk-elasticbeanstalk:1.12.246-349.v96b_b_f7eb_a_c3c
aws-java-sdk-iam:1.12.246-349.v96b_b_f7eb_a_c3c
aws-java-sdk-logs:1.12.246-349.v96b_b_f7eb_a_c3c
aws-java-sdk-minimal:1.12.246-349.v96b_b_f7eb_a_c3c
aws-java-sdk-ssm:1.12.246-349.v96b_b_f7eb_a_c3c
bootstrap5-api:5.1.3-7
bouncycastle-api:2.26
branch-api:2.1046.v0ca_37783ecc5
caffeine-api:2.9.3-65.v6a_47d0f4d1fe
checks-api:1.7.4
cloudbees-folder:6.729.v2b_9d1a_74d673
command-launcher:84.v4a_97f2027398
configuration-as-code:1414.v878271fc496f
credentials:1087.1089.v2f1b_9a_b_040e4
credentials-binding:523.vd859a_4b_122e6
datadog:4.0.0
display-url-api:2.3.6
docker-commons:1.19
docker-java-api:3.2.13-37.vf3411c9828b9
docker-plugin:1.2.7
docker-workflow:1.28
durable-task:496.va67c6f9eefa7
ec2:1.68
echarts-api:5.3.3-1
extended-choice-parameter:346.vd87693c5a_86c
font-awesome-api:6.1.1-1
git:4.10.3
git-client:3.11.0
git-server:1.11
github:1.34.3
github-api:1.303-400.v35c2d8258028
github-branch-source:1598.v91207e9f9b_4a_
handlebars:3.0.8
jackson2-api:2.13.3-285.vc03c0256d517
javax-activation-api:1.2.0-3
javax-mail-api:1.6.2-6
jaxb:2.3.6-1
jdk-tool:1.0
jjwt-api:0.11.5-77.v646c772fddb_0
jnr-posix-api:3.1.7-3
job-dsl:1.79
jquery:1.12.4-1
jquery3-api:3.6.0-4
jsch:0.1.55.2
junit:1119.1121.vc43d0fc45561
kubernetes:1.31.3
kubernetes-cli:1.10.3
kubernetes-client-api:5.12.2-193.v26a_6078f65a_9
kubernetes-credentials:0.9.0
list-git-branches-parameter:0.0.11
lockable-resources:2.15
mailer:414.vcc4c33714601
matrix-auth:3.1.5
matrix-project:772.v494f19991984
metrics:4.1.6.2
mina-sshd-api-common:2.8.0-21.v493b_6b_db_22c6
mina-sshd-api-core:2.8.0-21.v493b_6b_db_22c6
momentjs:1.1.1
node-iterator-api:1.5.1
okhttp-api:4.9.3-105.vb96869f8ac3a
parameterized-trigger:2.44
pipeline-aws:1.43
pipeline-build-step:2.18
pipeline-github:2.8-138.d766e30bb08b
pipeline-github-lib:36.v4c01db_ca_ed16
pipeline-graph-analysis:195.v5812d95a_a_2f9
pipeline-groovy-lib:593.va_a_fc25d520e9
pipeline-input-step:449.v77f0e8b_845c4
pipeline-milestone-step:101.vd572fef9d926
pipeline-model-api:2.2097.v33db_b_de764b_e
pipeline-model-definition:2.2097.v33db_b_de764b_e
pipeline-model-extensions:2.2097.v33db_b_de764b_e
pipeline-rest-api:2.24
pipeline-stage-step:293.v200037eefcd5
pipeline-stage-tags-metadata:2.2097.v33db_b_de764b_e
pipeline-stage-view:2.24
plain-credentials:1.8
plugin-util-api:2.17.0
popper2-api:2.11.5-2
python:1.3
resource-disposer:0.19
role-strategy:3.2.0
saml:2.296.v0016349946db_
scm-api:608.vfa_f971c5a_a_e9
script-security:1175.v4b_d517d6db_f0
slack:608.v19e3b_44b_b_9ff
snakeyaml-api:1.30.2-76.vc104f7ce9870
ssh-credentials:277.v95c2fec1c047
ssh-slaves:1.821.vd834f8a_c390e
sshd:3.242.va_db_9da_b_26a_c3
structs:318.va_f3ccb_729b_71
terraform:1.0.10
token-macro:293.v283932a_0a_b_49
trilead-api:1.66.v49c6758b_b_360
uno-choice:2.6.1
variant:1.4
workflow-aggregator:2.6
workflow-api:1188.v0016b_4f29881
workflow-basic-steps:969.vc4ec3e4854b_f
workflow-cps:2729.vea_17b_79ed57a_
workflow-cps-global-lib:588.v576c103a_ff86
workflow-durable-task-step:1174.v73a_9a_17edce0
workflow-job:1189.va_d37a_e9e4eda_
workflow-multibranch:716.vc692a_e52371b_
workflow-scm-step:400.v6b_89a_1317c9a_
workflow-step-api:625.vd896b_f445a_f8
workflow-support:833.va_1c71061486b_
ws-cleanup:0.42
xml-job-to-job-dsl:0.1.13
What Operating System are you using (both controller, and any agents involved in the problem)?
The Jenkins running on EKS cluster and deployed by Helm chart with Terraform "helm_release"
Reproduction steps
- Create AWS secret manager with ssh-key for GitHub credentials as below:
-----BEGIN OPENSSH PRIVATE KEY-----
b3BlbnNzaC1rZXktdjEABCDEBG5vbmUAAAAEbm9uZQAAAAAAAAABAAABlwAAAAdzc2gtcn
NhAAAAAwEAAQAAAYEAxs6i6gjyvbqIMboLC7zQ3RB0UhGORL5idCIPhpupuQIMySUmZ1S+
-----END OPENSSH PRIVATE KEY-----
- Use data to call the secret:
data "aws_secretsmanager_secret_version" "github_token" {
secret_id = "github"
}
- Define the SSH_PRIVATE_KEY in the values.yaml -
credentials:
system:
domainCredentials:
- credentials:
- basicSSHUserPrivateKey:
scope: GLOBAL
id: github
username: github
description: "Credentials for GitHub repo"
privateKeySource:
directEntry:
privateKey: |
"${SSH_PRIVATE_KEY}"
- Pass the secret to the values.yaml with terraform templatefile function -
resource "helm_release" "jenkins" {
name = "jenkins"
namespace = kubernetes_namespace.jenkins.metadata[0].name
repository = "https://charts.jenkins.io"
chart = "jenkins"
version = var.chart_version
values = [
templatefile("${path.module}/${var.values}.yaml", {
SSH_PRIVATE_KEY : data.aws_secretsmanager_secret_version.github_token.secret_string
}
)
]
}
Expected Results
Helm upgrade action to be completed successfully and the ssh-key configured as expected in the Jenkins credentials.
Actual Results
Received the below error:
Error: ---> error converting YAML to JSON: yaml: line 433: could not find expected ':' # Default values for jenkins.
on this section:
credentials:
system:
domainCredentials:
- credentials:
- basicSSHUserPrivateKey:
scope: GLOBAL
id: github
username: github
description: "Credentials for public-cloud-infrastructure GitHub repo"
privateKeySource:
directEntry:
privateKey: |
"-----BEGIN OPENSSH PRIVATE KEY-----
b3BlbnNzaC1rZXktdjEABCDEBG5vbmUAAAAEbm9uZQAAAAAAAAABAAABlwAAAAdzc2gtcn
NhAAAAAwEAAQAAAYEAxs6i6gjyvbqIMboLC7zQ3RB0UhGORL5idCIPhpupuQIMySUmZ1S+
-----END OPENSSH PRIVATE KEY-----"
Anything else?
I tried to use this link to fix the syntax error of the yaml with no luck
Gotta be careful with yaml: https://yaml-multiline.info/
This should do and ensure the key saved in secret manager has an final newline.
privateKey: "${SSH_PRIVATE_KEY}"
Gotta be careful with yaml: https://yaml-multiline.info/
This should do and ensure the key saved in secret manager has an final newline.
privateKey: "${SSH_PRIVATE_KEY}"
Thanks for the quick response.
In this case, the helm upgrade is completed correctly, but the secret is created as one line which is wrong as the ssh-key is sensitive and should be created (pass) to the values.yaml as below:
credentials:
system:
domainCredentials:
- credentials:
- basicSSHUserPrivateKey:
scope: GLOBAL
id: github-public-cloud-infrastructure
username: github
description: "Credentials for public-cloud-infrastructure GitHub repo"
privateKeySource:
directEntry:
privateKey: |
-----BEGIN OPENSSH PRIVATE KEY-----
b3BlbnNzaC1rZXktdjEABCDBG5vbmUAAAAEbm9uZQAAAAAAAAABAAABlwAAAAdzc2gtcn
NhAAAAAwEAAQAAAYEAxs6i6gjyvbqIMboLC7zQ3RB0UhGORL5idCIPhpupuQIMySUmZ1S+
-----END OPENSSH PRIVATE KEY-----
Secret manager and terraform should perserve the multiline.
So will the expanded secret
Yes you can have multiline in yaml with quotes by escaping with \n in a literal.
Also the expanded secret will also perserve newlines by using \n in the raw binary to preserve the newlines.
Also why use terraform templating to expand secrets?
You do know you can use Jenkins and JCasC to directly connect to secret manager: https://plugins.jenkins.io/aws-secrets-manager-credentials-provider/#plugin-content-configuration-as-code-casc
Don't store secrets in helm revisions 😉
Sorry this is the plugin you want: https://github.com/jenkinsci/aws-secrets-manager-secret-source-plugin
Sorry, this is the plugin you want: https://github.com/jenkinsci/aws-secrets-manager-secret-source-plugin
IIRC there is no option to use IAM assume role with the plugin, right? as we manage all our secrets on different AWS account
Secret manager and terraform should perserve the multiline.
So will the expanded secret
Yes you can have multiline in yaml with quotes by escaping with
\nin a literal.Also the expanded secret will also perserve newlines by using
\nin the raw binary to preserve the newlines.
By adding the /n the issue is solved.
@jetersen Thanks a lot for your help
IIRC there is no option to use IAM assume role with the plugin, right? as we manage all our secrets on different AWS account
There is, it uses the AWS Java SDK it supports assume role. So simply make sure your Jenkins server assumes the appropriate role
This is related to this subject. I can't seem to get credentials to work in the values.yaml file. I can get it to work if I upload to container filesystem and do a JCasC "Replace configuration source with:"
What is the proper syntax for credentials in the values.yaml? I have the below code in my values.yaml and the AuthorizedStrategy and SecurityRealms work, but not Credentials
JCasC:
enabled: true
credentials:
system:
domainCredentials:
- credentials:
- string:
description: "XXXXXXXXXXXXXXXX"
id: "XXXXXXXXXXXXXXXXXXXX"
scope: GLOBAL
secret: "{XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX}"
authorizationStrategy: |-
roleBased:
roles:
global:
- assignments:
...
@wesleung not at all related, however please see docs: https://github.com/jenkinsci/configuration-as-code-plugin/blob/master/docs/features/secrets.adoc#passing-secrets-through-variables
Can also look at demos: https://github.com/jenkinsci/configuration-as-code-plugin/tree/master/demos/credentials
Hi,
Sorry to piggy back on this but my issue is closely related to this.
I am having a similar issue when trying to pass an SSH private key (value, not from a file) from terraform building an entire Jenkins environment, and then trying to automatically configure the EC2 plugin so that environment is already hooked into AWS, all within a jenkins.yaml file, that is then consumed by a cloud-init script when building the Jenkins manager. I've tried every syntax I can think of to make sure that the syntax is right and I keep getting errors when the JCasC gets parsed and its related to me trying to create the basicSSHUserPrivateKey credential. I send over the values into my cloud-config and then use them to insert values into my JCasC config (as I create that file with cloud-config).
For some reason, I keep having issues with the private key pem that I am inserting into the credentials area in the JCasC file. I've tried a ton of diff syntaxes, but no dice. "${ec2_private_key}" , ${"$"}{decodeBase64:${ec2_private_key}}, the list goes on. @jetersen I've seen a ton of comments from you that are SORT of around this subject but not exactly related. Do you have any ideas of something I could try?
This attribute that I am using as the ec2_private_key value is a typical PEM file - so I am not sure what I am doing wrong. It must all be in the syntax of how I am trying to inject it. I've reviewed all of the documentation and all of the Git issues I could find and I am stuck.
-----BEGIN RSA PRIVATE KEY----- MIIJKgIBAAKCAgEAsnJWD4Qr3r/i6ZGnYRuP5/BhOWkvg6IwO6lfxPxvo/yBfeUl ... /X602AYAC6KYLuIsUo4x0Idp1EOD+vQeEhskmBVgkoerSd4tvXcgo6O8TUZbGdko -----END RSA PRIVATE KEY-----
@ryanberger-az I don't think it is relevant to post here.
Seems your issue is basic yaml syntax. Which could be caused by your templating the wrong yaml.
@jetersen Do you have any advice on how I could begin to troubleshoot this? The value that is being sent into the JCasC file is the exact value the documentation says it supports - so I am not sure why I am receiving the error message that I am when it is parsed. Sorry, I am not an expert at YAML syntaxing but I am trying to learn.
Can you share your template? Since that should not contain secrets.
Or the section around line 64 and 65. Read the error message 😉
This might also be helpful, I think your issue is multiline: https://yaml-multiline.info/
@jetersen I got this resolved. This was an indentation issue and I ended up having to use the terraform indent function. Your indentation count may vary, in my case I build my jenkins.yaml files using a cloud-init script, so my indentations were a little farther over because of the nested automation.
- basicSSHUserPrivateKey:
description: "Private key for the Windows Agents keypair"
id: "aws-ec2-windows-agents-pk"
privateKeySource:
directEntry:
privateKey: |
${indent(22, ec2_private_key)}
scope: GLOBAL
username: "administrator"