concourse
JWT for Container Runtime
This is my first contribution to Concourse - I'm very open to developing in Golang but unfamiliar with the code base at github.com/concourse/concourse and need some help figuring out where to put the code and how to pass along the payload to the container properly. Figured this would be a good place to start talking about the change before beginning work on it.
This also sounds like it's solving the same problem as
var_sources. Is it not? https://concourse-ci.org/vars.html#var-sources
That implementation, while an improvement on the current non-experimental implementation of Vault integration, is not ideal for the following reason: it does not adequately solve the first credential issue. All of the currently supported authentication methods for var_sources (approle, cert, token) are themselves dependent on secrets that Concourse must then attempt to store securely on a per-pipeline basis. JWT auth removes the need for Concourse to store any secrets at all for the connection to Vault, aside from standard TLSv1.3 communication (which is solved by both Concourse and Vault trusting the same PKI/CA).
From the Concourse perspective, no actual local configuration will need to be done that is specific to Vault. Concourse:
- Provides a signed JWT to each job that Concourse itself defines, generates, signs and delivers
- Provides a JWKS validation endpoint to any external system that wants to validate the JWT
...and that's it. No implementing and storing a certificate specifically for Vault authentication. No storing AppRole credentials. No token renewal maintenance. Nothing specific to Vault at all in this implementation (so far). Everything is dynamically generated per run, creating a unique signed JWT that is tightly scoped to the job lifetime and doesn't have to be pre-configured.
Potentially, var_sources could leverage this authentication method as an additional source type. But that's a different discussion.
Then, on the Vault side, all the configuration happens. Vault is set up to trust the JWKS endpoint that Concourse provides. When a pipeline job or other container gives the signed JWT to Vault, Vault is able to validate the job is part of the pipeline it says it is, and should have access to secrets based on the job's team, pipeline, or whatever other data points the Vault administrator chooses to bind claims to. The specific configuration of JWT on the Vault side is somewhat irrelevant to this discussion.
EDIT: The other authentication methods are also more prone to human error. Human must handle credentials (not ideal), configure them on the right pipeline, and ideally ensure the credentials are not re-used on multiple pipelines in a way that would encourage role privilege creep. The credentials are also static, and must be rotated. Eventually (and, to be fair, this is speculation) Concourse itself will be asked to handle the automatic rotation of those credentials by giving Concourse itself another role that can do those actions in Vault and Concourse, or people will build their own master CI configuration tools and roles that do this for them. JWT avoids all this. Because the credential is just-in-time, ephemeral, unique to the job run, and a validation endpoint is provided, external systems like Vault simply get configured to trust the ecosystem. No humans handling credentials, no need for rotation effort, no (speculative) creep towards another role that manages static credentials on Concourse and Vault.
I think I'm starting to understand this. This DOES sound like a nice improvement over the current vault setup/integration.
I'll re-read the proposal soon.
Apologies, was on vacation, working on a response now...
EDIT: responded to all inline questions @taylorsilva :smile:
Enabling JWT even opens up the ability to allow Concourse pipelines to authenticate and authorize to arbitrary custom APIs if desired.
This is the part that excites me the most - if Concourse would expose the pubkey it uses for JWT signing, it would have some concept of workload identity, which would allow key-less authentication to AWS/GCP, similar to https://scalesec.com/blog/identity-federation-for-github-actions-on-aws/.
In this RFC, there's also a lot about Vault authentication and secret handling, which confused me at first. I'm not sure I understood correctly, but this essentially proposes using these JWTs to also interact with the Vault credential manager? Should this probably be moved to its own section/separate RFC? Having some component creating these JWTs is a precondition to propagating them when asking for secrets…
Enabling JWT even opens up the ability to allow Concourse pipelines to authenticate and authorize to arbitrary custom APIs if desired.
This is the part that excites me the most - if Concourse would expose the pubkey it uses for JWT signing, it would have some concept of workload identity, which would allow key-less authentication to AWS/GCP, similar to https://scalesec.com/blog/identity-federation-for-github-actions-on-aws/.
In this RFC, there's also a lot about Vault authentication and secret handling, which confused me at first. I'm not sure I understood correctly, but this essentially proposes using these JWTs to also interact with the Vault credential manager? Should this probably be moved to its own section/separate RFC? Having some component creating these JWTs is a precondition to propagating them when asking for secrets…
Yes, so selfishly, I'm approaching this from the perspective of a Hashicorp Vault Implementation Partner, and JWT/OIDC is a great way to establish workload identity, but of course, the standard can be implemented anywhere. The use case is not limited to Hashicorp Vault, but is instead one of many use cases, and I framed it this way because it is the situation I am most familiar with.
I've had a re-write of this RFC on my mind for a while now, but this is great feedback, and I will make it more generic and list out separate use cases in a way that doesn't make it look like Hashicorp Vault is the only use case (or even a "primary" use case).