[Bug] AL2023 self-managed nodegroups should support `kubeletExtraConfig`

[TiberiuGC]

Currently, there's a validation in place that prevents the use case described in the title, e.g.

Error: could not create cluster provider from options: kubeletExtraConfig is not supported for AmazonLinux2023 nodegroups (path=nodeGroups[0].kubeletExtraConfig)

For self-managed AL2023 nodes, we need to build the kubelet config file and pass it to the nodeadm process via nodeConfig.kubelet.flags i.e. --config.

[punkwalker]

@TiberiuGC nodeadm does not allow passing just kubelet config via file, It has to be the entire NodeConfig and it has to be passed as nodeadm config --config-source. However, doing so would require executing nodeadm via user data.

EKS Optimized AMIs have Service Unit files for nodeadm /etc/systemd/system/nodeadm-config.service & /etc/systemd/system/nodeadm-run.service and it utilizes user data as source for NodeConfig.

Therefore, we should be building NodeConfig in UserData

[TiberiuGC]

@punkwalker thanks for the feedback!

Based on these references - https://awslabs.github.io/amazon-eks-ami/nodeadm/doc/api/#kubeletoptions and https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/, I was assuming we could pass something along the lines below to the userdata 👇🏻

MIME-Version: 1.0
Content-Type: multipart/mixed; boundary=//

--//
Content-Type: application/node.eks.aws

apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
spec:
  cluster:
    ...
  kubelet:
    config:
      clusterDNS:
      - 10.100.0.10
    flags:
    ...
    - --config=config.yaml

--//--

What do you think about this? Will this not work?

[punkwalker]

@TiberiuGC ~~I see, I think this should work. I will try to test it.~~ I just walked through the nodeadm code and I think, using --config in nodeConfig.spec.kubelet.flags may not work.

Nodeadm is using the values of nodeConfig.spec.kubelet.flags similar to KUBELET_EXTRA_ARGS of AL2 bootstrap.sh. The flags from config are concanated and stored as ENVIRONMENT variable "NODEADM_KUBELET_ARGS" in a File which is referenced in kubelet unit file. Ref

And Nodeadm also sets --config which later gets added into NODEADM_KUBELET_ARGS. Ref

Setting --config flag again might either overwrite the value of --config set by Nodeadm or it may break kubelet due to duplicate flag (not sure about the later). Even if the nodeadm flag value is overwritten by eksctl --config flag, we would have to write the entire kubelet config.json in that file as the --config value is the file from where kubelet will pick base/default config.

So, IMO we should use nodeConfig.spec.kubelet.config via UserData for adding the kubeletExtraConfig. What do you think?

[TiberiuGC]

@punkwalker thanks for investigating this option. Lmk if you'd like to work on the fix you've suggested, if your time allows.

[punkwalker]

Sure @TiberiuGC I will work on it.