[Bug] eksctl scale nodegroup fails with Error: overrideBootstrapCommand overwrites container runtime....

[steffakasid]

What were you trying to accomplish?

I was trying to scale my nodegroups together with my clusterconfig: eksctl scale nodegroup -f -

What happened?

The first nodegroup was scaled but then the command failed with the following error:

Error: overrideBootstrapCommand overwrites container runtime setting; please use --container-runtime in the bootsrap script instead

This seems to be related to: https://github.com/weaveworks/eksctl/pull/5365

How to reproduce it?

Create a cluster config using unmanagedNodegroups with customAMI and overrideBootstrapCommand then run ekcstl scale nodegroup -f cluster.yml with this cluster configuration.

Logs

eksctl scale nodegroup -f - --verbose 4
2022-07-14 11:34:40 [▶]  role ARN for the current session is "arn:aws:sts::638377377767:assumed-role/ServiceTeam/aws-go-sdk-1657791279532821000"
2022-07-14 11:34:42 [▶]  cluster = &types.Cluster{Arn:(*string)(0x1400063f320), CertificateAuthority:(*types.Certificate)(0x1400063f2b0), ClientRequestToken:(*string)(nil), ConnectorConfig:(*types.ConnectorConfigResponse)(nil), CreatedAt:time.Date(2020, time.November, 26, 12, 54, 27, 582000000, time.UTC), EncryptionConfig:[]types.EncryptionConfig{types.EncryptionConfig{Provider:(*types.Provider)(0x1400063f2f0), Resources:[]string{"secrets"}, noSmithyDocumentSerde:document.NoSerde{}}}, Endpoint:(*string)(0x1400063f2a0), Identity:(*types.Identity)(0x1400063f240), KubernetesNetworkConfig:(*types.KubernetesNetworkConfigResponse)(0x14000800750), Logging:(*types.Logging)(0x14000f46720), Name:(*string)(0x1400063f310), PlatformVersion:(*string)(0x1400063f290), ResourcesVpcConfig:(*types.VpcConfigResponse)(0x140001583f0), RoleArn:(*string)(0x1400063f330), Status:"ACTIVE", Tags:map[string]string{}, Version:(*string)(0x1400063f270), noSmithyDocumentSerde:document.NoSerde{}}
2022-07-14 11:34:42 [ℹ]  scaling nodegroup "bizhub-1a-blue" in cluster eks-test
2022-07-14 11:34:45 [▶]  nodegroups = [0x14000e6a6c0 0x14000b3c480 0x14000e6b560 0x14000b3d320 0x14000b3de60 0x14000d886c0 0x14000b8b320 0x14000b8bd40 0x14000e3b320 0x14000a5ca20 0x1400057e480 0x14000be87e0 0x14000be9200 0x14000be9c20]
2022-07-14 11:34:48 [ℹ]  nodegroup successfully scaled
Error: overrideBootstrapCommand overwrites container runtime setting; please use --container-runtime in the bootsrap script instead

Versions

$ eksctl info
eksctl version: 0.105.0-dev+aa76f1d4.2022-07-08T14:38:11Z
kubectl version: v1.24.3
OS: darwin

[Himangini]

@steffakasid can you share the contents of cluster.yml [sensitive info redacted of-course] so we can investigate this further, please?

[steffakasid]

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
  name: eks-cluster
  version: "1.20"
  region: eu-central-1
iam:
  withOIDC: true
  serviceAccounts:
    - metadata:
        name: aws-load-balancer-controller
        namespace: kube-system
      wellKnownPolicies:
        awsLoadBalancerController: true
vpc:
  nat:
    gateway: Disable
  subnets:
    private:
      eu-central-1a:
        id: subnet-1a
      eu-central-1b:
        id: subnet-1b
      eu-central-1c:
        id: subnet-1c

  extraCIDRs:
    - 0.0.0.0/0
  clusterEndpoints:
    publicAccess: false
    privateAccess: true
cloudWatch:
  clusterLogging:
    enableTypes:
      - api
      - audit
      - authenticator
      - controllerManager
      - scheduler
secretsEncryption:
  keyARN: arn:aws:kms:eu-central-1:12345676:key/asdfas-bbb--dddd
addons:
- name: coredns
  version: v1.8.0
- name: kube-proxy
  version: v1.20.7
nodeGroups:
  - name: type1-1a-blue
    ami: ami-12345
    maxPodsPerNode: 44
    overrideBootstrapCommand: |
      #!/bin/bash
      source /var/lib/cloud/scripts/eksctl/bootstrap.helper.sh
      # Note "--node-labels=${NODE_LABELS}" needs the above helper sourced to work, otherwise will have to be defined manually.
      /etc/eks/bootstrap.sh eks-cluster --container-runtime containerd --kubelet-extra-args "--node-labels=${NODE_LABELS} --register-with-taints=${NODE_TAINTS}"
    instanceType: m5.2xlarge
    minSize: 1
    desiredCapacity: 1
    maxSize: 2
    availabilityZones:
      - eu-central-1a
    privateNetworking: true
    labels:
      type: type1
      aws-vpc/eniConfig: eu-central-1eu-central-1a
    tags:
      Environment: Test

    iam:
      withAddonPolicies:
        externalDNS: true
        ebs: true
        efs: true
        autoScaler: true
        cloudWatch: true
      attachPolicyARNs:
        - arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly
        - arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy
        - arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy
        - arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore
        - arn:aws:iam::12345:policy/UseKmsKey
  - name: type2-1b-blue
    ami: ami-12345
    maxPodsPerNode: 44
    overrideBootstrapCommand: |
      #!/bin/bash
      source /var/lib/cloud/scripts/eksctl/bootstrap.helper.sh
      # Note "--node-labels=${NODE_LABELS}" needs the above helper sourced to work, otherwise will have to be defined manually.
      /etc/eks/bootstrap.sh eks-cluster --container-runtime containerd --kubelet-extra-args "--node-labels=${NODE_LABELS} --register-with-taints=${NODE_TAINTS}"
    instanceType: m5.2xlarge
    minSize: 1
    desiredCapacity: 1
    maxSize: 2
    availabilityZones:
      - eu-central-1b
    privateNetworking: true
    labels:
      type: type2
      aws-vpc/eniConfig: eu-central-1eu-central-1b
    tags:
      Environment: Test

    iam:
      withAddonPolicies:
        externalDNS: true
        ebs: true
        efs: true
        autoScaler: true
        cloudWatch: true
      attachPolicyARNs:
        - arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly
        - arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy
        - arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy
        - arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore
        - arn:aws:iam::12345:policy/UseKmsKey
  - name: type2-1c-blue
    ami: ami-12345
    maxPodsPerNode: 44
    overrideBootstrapCommand: |
      #!/bin/bash
      source /var/lib/cloud/scripts/eksctl/bootstrap.helper.sh
      # Note "--node-labels=${NODE_LABELS}" needs the above helper sourced to work, otherwise will have to be defined manually.
      /etc/eks/bootstrap.sh eks-cluster --container-runtime containerd --kubelet-extra-args "--node-labels=${NODE_LABELS} --register-with-taints=${NODE_TAINTS}"
    instanceType: m5.2xlarge
    minSize: 1
    desiredCapacity: 1
    maxSize: 2
    availabilityZones:
      - eu-central-1c
    privateNetworking: true
    labels:
      type: type2
      aws-vpc/eniConfig: eu-central-1eu-central-1c
    tags:
      Environment: Test

    iam:
      withAddonPolicies:
        externalDNS: true
        ebs: true
        efs: true
        autoScaler: true
        cloudWatch: true
      attachPolicyARNs:
        - arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly
        - arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy
        - arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy
        - arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore
        - arn:aws:iam::12345:policy/UseKmsKey
  - name: type2-1a-blue
    ami: ami-12345
    maxPodsPerNode: 205
    overrideBootstrapCommand: |
      #!/bin/bash
      source /var/lib/cloud/scripts/eksctl/bootstrap.helper.sh
      # Note "--node-labels=${NODE_LABELS}" needs the above helper sourced to work, otherwise will have to be defined manually.
      /etc/eks/bootstrap.sh eks-cluster --container-runtime containerd --kubelet-extra-args "--node-labels=${NODE_LABELS} --register-with-taints=${NODE_TAINTS}"
    instanceType: m5.4xlarge
    minSize: 1
    desiredCapacity: 1
    maxSize: 3
    availabilityZones:
      - eu-central-1a
    privateNetworking: true
    taints:
      - key: type
        value: type2
        effect: NoSchedule
    labels:
      type: type2
      aws-vpc/eniConfig: eu-central-1eu-central-1a
    tags:
      Environment: Test

    iam:
      withAddonPolicies:
        externalDNS: true
        ebs: true
        efs: true
        autoScaler: true
        cloudWatch: true
      attachPolicyARNs:
        - arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly
        - arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy
        - arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy
        - arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore
        - arn:aws:iam::12345:policy/UseKmsKey
  - name: type2-1b-blue
    ami: ami-12345
    maxPodsPerNode: 205
    overrideBootstrapCommand: |
      #!/bin/bash
      source /var/lib/cloud/scripts/eksctl/bootstrap.helper.sh
      # Note "--node-labels=${NODE_LABELS}" needs the above helper sourced to work, otherwise will have to be defined manually.
      /etc/eks/bootstrap.sh eks-cluster --container-runtime containerd --kubelet-extra-args "--node-labels=${NODE_LABELS} --register-with-taints=${NODE_TAINTS}"
    instanceType: m5.4xlarge
    minSize: 1
    desiredCapacity: 1
    maxSize: 3
    availabilityZones:
      - eu-central-1b
    privateNetworking: true
    taints:
      - key: type
        value: type2
        effect: NoSchedule
    labels:
      type: type2
      aws-vpc/eniConfig: eu-central-1eu-central-1b
    tags:
      Environment: Test

    iam:
      withAddonPolicies:
        externalDNS: true
        ebs: true
        efs: true
        autoScaler: true
        cloudWatch: true
      attachPolicyARNs:
        - arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly
        - arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy
        - arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy
        - arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore
        - arn:aws:iam::12345:policy/UseKmsKey
  - name: type2-1c-blue
    ami: ami-12345
    maxPodsPerNode: 205
    overrideBootstrapCommand: |
      #!/bin/bash
      source /var/lib/cloud/scripts/eksctl/bootstrap.helper.sh
      # Note "--node-labels=${NODE_LABELS}" needs the above helper sourced to work, otherwise will have to be defined manually.
      /etc/eks/bootstrap.sh eks-cluster --container-runtime containerd --kubelet-extra-args "--node-labels=${NODE_LABELS} --register-with-taints=${NODE_TAINTS}"
    instanceType: m5.4xlarge
    minSize: 1
    desiredCapacity: 1
    maxSize: 3
    availabilityZones:
      - eu-central-1c
    privateNetworking: true
    taints:
      - key: type
        value: type2
        effect: NoSchedule
    labels:
      type: type2
      aws-vpc/eniConfig: eu-central-1eu-central-1c
    tags:
      Environment: Test

    iam:
      withAddonPolicies:
        externalDNS: true
        ebs: true
        efs: true
        autoScaler: true
        cloudWatch: true
      attachPolicyARNs:
        - arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly
        - arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy
        - arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy
        - arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore
        - arn:aws:iam::12345:policy/UseKmsKey
  - name: type2-gw-1a-blue
    ami: ami-12345
    maxPodsPerNode: 44
    overrideBootstrapCommand: |
      #!/bin/bash
      source /var/lib/cloud/scripts/eksctl/bootstrap.helper.sh
      # Note "--node-labels=${NODE_LABELS}" needs the above helper sourced to work, otherwise will have to be defined manually.
      /etc/eks/bootstrap.sh eks-cluster --container-runtime containerd --kubelet-extra-args "--node-labels=${NODE_LABELS} --register-with-taints=${NODE_TAINTS}"
    instanceType: c5.4xlarge
    minSize: 1
    desiredCapacity: 1
    maxSize: 3
    availabilityZones:
      - eu-central-1a
    privateNetworking: true
    taints:
      - key: type
        value: type2-gw
        effect: NoSchedule
    labels:
      type: type2-gw
      aws-vpc/eniConfig: eu-central-1eu-central-1a
    tags:
      Environment: Test

    iam:
      withAddonPolicies:
        externalDNS: true
        ebs: true
        efs: true
        autoScaler: true
        cloudWatch: true
      attachPolicyARNs:
        - arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly
        - arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy
        - arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy
        - arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore
        - arn:aws:iam::12345:policy/UseKmsKey
  - name: type2-gw-1b-blue
    ami: ami-12345
    maxPodsPerNode: 44
    overrideBootstrapCommand: |
      #!/bin/bash
      source /var/lib/cloud/scripts/eksctl/bootstrap.helper.sh
      # Note "--node-labels=${NODE_LABELS}" needs the above helper sourced to work, otherwise will have to be defined manually.
      /etc/eks/bootstrap.sh eks-cluster --container-runtime containerd --kubelet-extra-args "--node-labels=${NODE_LABELS} --register-with-taints=${NODE_TAINTS}"
    instanceType: c5.4xlarge
    minSize: 1
    desiredCapacity: 1
    maxSize: 3
    availabilityZones:
      - eu-central-1b
    privateNetworking: true
    taints:
      - key: type
        value: type2-gw
        effect: NoSchedule
    labels:
      type: type2-gw
      aws-vpc/eniConfig: eu-central-1eu-central-1b
    tags:
      Environment: Test

    iam:
      withAddonPolicies:
        externalDNS: true
        ebs: true
        efs: true
        autoScaler: true
        cloudWatch: true
      attachPolicyARNs:
        - arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly
        - arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy
        - arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy
        - arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore
        - arn:aws:iam::12345:policy/UseKmsKey
  - name: type2-gw-1c-blue
    ami: ami-12345
    maxPodsPerNode: 44
    overrideBootstrapCommand: |
      #!/bin/bash
      source /var/lib/cloud/scripts/eksctl/bootstrap.helper.sh
      # Note "--node-labels=${NODE_LABELS}" needs the above helper sourced to work, otherwise will have to be defined manually.
      /etc/eks/bootstrap.sh eks-cluster --container-runtime containerd --kubelet-extra-args "--node-labels=${NODE_LABELS} --register-with-taints=${NODE_TAINTS}"
    instanceType: c5.4xlarge
    minSize: 1
    desiredCapacity: 1
    maxSize: 3
    availabilityZones:
      - eu-central-1c
    privateNetworking: true
    taints:
      - key: type
        value: type2-gw
        effect: NoSchedule
    labels:
      type: type2-gw
      aws-vpc/eniConfig: eu-central-1eu-central-1c
    tags:
      Environment: Test

    iam:
      withAddonPolicies:
        externalDNS: true
        ebs: true
        efs: true
        autoScaler: true
        cloudWatch: true
      attachPolicyARNs:
        - arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly
        - arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy
        - arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy
        - arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore
        - arn:aws:iam::12345:policy/UseKmsKey
  - name: type3-1a-blue
    ami: ami-12345
    maxPodsPerNode: 24
    overrideBootstrapCommand: |
      #!/bin/bash
      source /var/lib/cloud/scripts/eksctl/bootstrap.helper.sh
      # Note "--node-labels=${NODE_LABELS}" needs the above helper sourced to work, otherwise will have to be defined manually.
      /etc/eks/bootstrap.sh eks-cluster --container-runtime containerd --kubelet-extra-args "--node-labels=${NODE_LABELS} --register-with-taints=${NODE_TAINTS}"
    instanceType: t3.large
    minSize: 1
    desiredCapacity: 1
    maxSize: 3
    availabilityZones:
      - eu-central-1a
    privateNetworking: true
    taints:
      - key: type
        value: type3
        effect: NoSchedule
    labels:
      type: type3
      aws-vpc/eniConfig: eu-central-1eu-central-1a
    tags:
      Environment: Test

    iam:
      withAddonPolicies:
        externalDNS: true
        ebs: true
        efs: true
        autoScaler: true
        cloudWatch: true
      attachPolicyARNs:
        - arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly
        - arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy
        - arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy
        - arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore
        - arn:aws:iam::12345:policy/UseKmsKey
  - name: type3-1b-blue
    ami: ami-12345
    maxPodsPerNode: 24
    overrideBootstrapCommand: |
      #!/bin/bash
      source /var/lib/cloud/scripts/eksctl/bootstrap.helper.sh
      # Note "--node-labels=${NODE_LABELS}" needs the above helper sourced to work, otherwise will have to be defined manually.
      /etc/eks/bootstrap.sh eks-cluster --container-runtime containerd --kubelet-extra-args "--node-labels=${NODE_LABELS} --register-with-taints=${NODE_TAINTS}"
    instanceType: t3.large
    minSize: 1
    desiredCapacity: 1
    maxSize: 3
    availabilityZones:
      - eu-central-1b
    privateNetworking: true
    taints:
      - key: type
        value: type3
        effect: NoSchedule
    labels:
      type: type3
      aws-vpc/eniConfig: eu-central-1eu-central-1b
    tags:
      Environment: Test

    iam:
      withAddonPolicies:
        externalDNS: true
        ebs: true
        efs: true
        autoScaler: true
        cloudWatch: true
      attachPolicyARNs:
        - arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly
        - arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy
        - arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy
        - arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore
        - arn:aws:iam::12345:policy/UseKmsKey
  - name: type3-1c-blue
    ami: ami-12345
    maxPodsPerNode: 24
    overrideBootstrapCommand: |
      #!/bin/bash
      source /var/lib/cloud/scripts/eksctl/bootstrap.helper.sh
      # Note "--node-labels=${NODE_LABELS}" needs the above helper sourced to work, otherwise will have to be defined manually.
      /etc/eks/bootstrap.sh eks-cluster --container-runtime containerd --kubelet-extra-args "--node-labels=${NODE_LABELS} --register-with-taints=${NODE_TAINTS}"
    instanceType: t3.large
    minSize: 1
    desiredCapacity: 1
    maxSize: 3
    availabilityZones:
      - eu-central-1c
    privateNetworking: true
    taints:
      - key: type
        value: type3
        effect: NoSchedule
    labels:
      type: type3
      aws-vpc/eniConfig: eu-central-1eu-central-1c
    tags:
      Environment: Test

    iam:
      withAddonPolicies:
        externalDNS: true
        ebs: true
        efs: true
        autoScaler: true
        cloudWatch: true
      attachPolicyARNs:
        - arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly
        - arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy
        - arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy
        - arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore
        - arn:aws:iam::12345:policy/UseKmsKey

Here it is.

Regards Steffen

[TiberiuGC]

Hey @steffakasid , are you sure you've shared the correct cluster.yml file. Asking because checking the output of your command I see that the first nodegroup that was successfully scaled is called bizhub-1a-blue, whereas in the provided config file there's no nodegroup named like this.

Now the error you're seeing is a validation error, which makes me think that you've been setting containerRuntime for your nodegroup together with overrideBootstrapCommand. I may be wrong but could you please double check?

[Himangini]

Hey @steffakasid , are you sure you've shared the correct cluster.yml file. Asking because checking the output of your command I see that the first nodegroup that was successfully scaled is called bizhub-1a-blue, whereas in the provided config file there's no nodegroup named like this.

Now the error you're seeing is a validation error, which makes me think that you've been setting containerRuntime for your nodegroup together with overrideBootstrapCommand. I may be wrong but could you please double check?

@steffakasid we are unable to reproduce this on our side., can you post any updates to this? please close it if this is no longer an issue you are facing. 👍🏻

[github-actions[bot]]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[github-actions[bot]]

This issue was closed because it has been stalled for 5 days with no activity.