kubelet: add credential provider feature-gate

[samjo-nyang]

What I'd like: Allow arbitrary kubelet feature-gates settings to the kubelet configuration file, such as

featureGates:
  RotateKubeletServerCertificate: true
  CSIMigration: false
{{#if settings.kubernetes.feature-gates}}
{{#each settings.kubernetes.feature-gates as |v k|}}  {{k}}: {{v}}
{{/each}}
{{/if}}

Any alternatives you've considered: (nothing)

FYI: I'd like to try https://kubernetes.io/docs/tasks/kubelet-credential-provider/kubelet-credential-provider/ , but it requires to enable feature-gates on kubelet. I have internal patch, but there are no equivalent features on the upstream.

[samuelkarp]

Hi @samjo-nyang, thanks for opening this issue. Is the Kubelet credential provider the only feature gate you're looking to enable or are there others that you're interested in as well?

[samjo-nyang]

currently, i am only interested in the credential provider feature

[samuelkarp]

Hey @samjo-nyang, thanks for clarifying! We've discussed this and at this time we're not planning to add arbitrary feature-gates; Bottlerocket's validated settings API is one of the mechanisms we believe helps with both security and predictability of configuration at scale. However, we are happy to consider adding individual feature gates such as the credential provider feature. I've re-titled this issue to reflect the request for the credential provider feature gate.

Can you tell us a bit more about how you'd like to use that feature? Are you interested in adopting a credential provider for a registry other than Amazon ECR?

[samjo-nyang]

Yes, I have a private oci image registry and I want to implement a custom credential provider for the registry.

[stmcginnis]

Hey @samjo-nyang - As part of some other changes I am working on to enable credential providers, that will bring in the ability to enable the KubeletCredentialProviders feature gate.

I'm wondering if that change is enough to address this and close out this issue? Or is there a use case beyond that that you would still need addressed? There are some concerns about exposing this arbitrary set of feature gates, but we can discuss more if there is something you need to do that is prevented by not having this.

Thanks!

[samjo-nyang]

Hi, @stmcginnis For now, it is enough to enable (or give ability to enable) KubeletCredentialProviders. Thanks!

[stmcginnis]

Thanks, let's track this with #2310 then. If there ends up being any other feature gates we would like, we can open specific issues for those use cases.

Thanks!

[stmcginnis]

Sorry for the noise. Going to reopen this issue to track the work of adding credential provider support. Then use https://github.com/bottlerocket-os/bottlerocket/issues/2310 to track adding IAM Roles Anywhere to extend the work done here to support that additional use case.

[stmcginnis]

Status update on this work... I think I have most things in place with #2377, but having some trouble validating things.

I've verified with the team that it looks like all the right configuration is in place, but things are not working as expected. The current theory is there is some conflict between our use of the in-tree AWS cloud provider and this newer functionality that may actually need the out-of-tree cloud provider. Still working on trying to validate that assumption.

Since there is still a bit of work to do here, it doesn't look like this will make it into the 1.10.0 release. Retargeting this to 1.11.0 and will update as we find out more.

[stmcginnis]

Just an update, this is currently blocked on this upstream issue:

https://github.com/kubernetes/kubernetes/issues/112842

I will track that and follow up here as that progresses.

[kdaula]

I am taking the 1.11.0 release label from this issue since @stmcginnis will be working with upstream kubernetes driving to resolve https://github.com/kubernetes/kubernetes/issues/112842. As soon as the upstream issue is fixed, we will work to getting this into a release.