[EKS] [request]: Bring Start-EKSBootstrap.ps1 to feature parity with its Linux equivalent

[tynril]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Tell us about your request

The EKS bootstrap script for Windows nodes fell out of parity compared to its Linux equivalent. I would like to request for parity to be restored.

Which service(s) is this request for?

This is for EKS' Windows support.

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?

The current implementation of the Windows EKS boostrap script, Start-EKSBootstrap.ps1, is missing many of the parameters that can be passed to its Linux equivalent, bootstrap.sh.

This makes it more challenging to, for example, configure Docker via its daemon.json file, on Windows EKS nodes, than it is on Linux nodes.

Are you currently working around this issue?

While it is possible to execute things before and after the bootstrap script, its scope of responsibility is so wide that for some use-cases, that is simply not an option.

Additional context

The Start-EKSBootstrap.ps1 script's header contains the following line:

EKS bootstrap script. Should maintain close parity with https://github.com/awslabs/amazon-eks-ami/blob/master/files/bootstrap.sh

It seems like the intent is for parity to be achieved, but as currently implemented, it is not.

For comparison, here are the parameters defined for Linux (from bootstrap.sh):

function print_help {
    echo "usage: $0 [options] <cluster-name>"
    echo "Bootstraps an instance into an EKS cluster"
    echo ""
    echo "-h,--help print this help"
    echo "--use-max-pods Sets --max-pods for the kubelet when true. (default: true)"
    echo "--b64-cluster-ca The base64 encoded cluster CA content. Only valid when used with --apiserver-endpoint. Bypasses calling \"aws eks describe-cluster\""
    echo "--apiserver-endpoint The EKS cluster API Server endpoint. Only valid when used with --b64-cluster-ca. Bypasses calling \"aws eks describe-cluster\""
    echo "--kubelet-extra-args Extra arguments to add to the kubelet. Useful for adding labels or taints."
    echo "--enable-docker-bridge Restores the docker default bridge network. (default: false)"
    echo "--aws-api-retry-attempts Number of retry attempts for AWS API call (DescribeCluster) (default: 3)"
    echo "--docker-config-json The contents of the /etc/docker/daemon.json file. Useful if you want a custom config differing from the default one in the AMI"
    echo "--dns-cluster-ip Overrides the IP address to use for DNS queries within the cluster. Defaults to 10.100.0.10 or 172.20.0.10 based on the IP address of the primary interface"
}

And here are the parameters accepted by Start-EKSBoostrap.ps1:

[CmdletBinding()]
param(
  [Parameter(Mandatory=$true)]
  [string]$EKSClusterName,
  [string]$KubeletExtraArgs,
  [string]$Endpoint,
  [string]$APIServerEndpoint,
  [string]$Base64ClusterCA
)

[sstoyanovucsd]

Hi @mikestef9 ,

The absence of "--dns-cluster-ip" parameter alternative for Windows nodes leads to Windows pods using the default DNS IP address (10.100.0.10 or 172.20.0.10) even when cluster is installed with custom service CIDR. This breaks DNS resolution for Windows pods on such clusters.

AWS support replicated the issue and advised us to use a custom userdata script for the Windows nodes to configure DNS resolver config properly as a temporary workaround. Since they said "temporary" I guess a fix is being worked on. If so, it would be nice to:

1. Get periodic updates or even an ETA here.
2. Until fixed, add a note to the considerations on this page that using custom services CIDR leads to broken DNS resolution within Windows pods. (the one about custom networking not supported on Windows nodes doesn't seem to be relevant to this issue)

Thanks! Stoyan

[ChrisMcKee]

The Windows powershell script in the 2019 ami contains -DNSClusterIP

<#
.SYNOPSIS
EKS bootstrap script. Should maintain close parity with https://github.com/awslabs/amazon-eks-ami/blob/master/files/bootstrap.sh
.PARAMETER EKSClusterName
Specifies the EKS cluster name which this worker node to be joined.
.PARAMETER KubeletExtraArgs
Specifies the extra arguments for kubelet(optional).
.PARAMETER KubeProxyExtraArgs
Specifies the extra arguments for kube-proxy(optional).
.PARAMETER Endpoint
Specifies the EKS cluster endpoint(optional). Default is production endpoint.
.PARAMETER APIServerEndpoint
The EKS cluster API Server endpoint(optional). Only valid when used with -Base64ClusterCA. Bypasses calling "Get-EKSCluster".
.PARAMETER Base64ClusterCA
The base64 encoded cluster CA content(optional). Only valid when used with -APIServerEndpoint. Bypasses calling "Get-EKSCluster".
.PARAMETER DNSClusterIP
Overrides the IP address to use for DNS queries within the cluster(optional). Defaults to 10.100.0.10 or 172.20.0.10 based on the IP address of the primary interface.
.PARAMETER ServiceCIDR
Overrides the Kubernetes Service IP Address range from which cluster services are addressed. Defaults to 172.20.0.0/16 or 10.100.0.0/16 based on the IP address of the primary interface.
.PARAMETER ContainerRuntime
Specifies the container runtime to be used. On EKS 1.21 and below, it defaults to 'docker'. On EKS 1.24 and above, defaults to 'containerd'.
#>
[CmdletBinding()]
param(
  [Parameter(Mandatory=$true)]
  [string]$EKSClusterName,
  [string]$KubeletExtraArgs,
  [string]$KubeProxyExtraArgs,
  [string]$Endpoint,
  [string]$APIServerEndpoint,
  [string]$Base64ClusterCA,
  [string]$DNSClusterIP,
  [string]$ServiceCIDR,

  [ValidateSet("docker","containerd")]
  [string]$ContainerRuntime
)