eksctl tag node root volumes properly

see https://aws.amazon.com/premiumsupport/knowledge-center/cloudformation-instance-tag-root-volume/ for a potential work around to this

Jul 17 '19 17:07 errordeveloper

As it took me a bit of time to figure out how to do that with eksctl, here is an example

First create an IAM policy to allow tag creation

Cust-Ec2Tags-Policy

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ec2:CreateTags"
            ],
            "Resource": [
                "*"
            ]
        }
    ]
}

Then in your config file update your nodeGroups section with :

nodeGroups:
  - name: mynodegroup
    iam:
      attachPolicyARNs:
        - arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly
        - arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy
        - arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy
        - arn:aws:iam::<aws account id>:policy/Cust-Ec2Tags-Policy
[...]
    preBootstrapCommands:
      - |
        cat <<EOF > /usr/local/sbin/tag_ebs_volume.sh
        #!/bin/bash
        AWS_AVAIL_ZONE=\$(curl -S http://169.254.169.254/latest/meta-data/placement/availability-zone)
        AWS_REGION="\`echo \"\$AWS_AVAIL_ZONE\" | sed 's/[a-z]$//'\`"
        AWS_INSTANCE_ID=\$(curl -S http://169.254.169.254/latest/meta-data/instance-id)
        ROOT_VOLUME_IDS=\$(aws ec2 describe-instances --region \$AWS_REGION --instance-id \$AWS_INSTANCE_ID --output text --query Reservations[0].Instances[0].BlockDeviceMappings[0].Ebs.VolumeId)
        while IFS=$'\t' read -r key val
        do
          aws ec2 create-tags --resources \$ROOT_VOLUME_IDS --region \$AWS_REGION --tags Key=\${key},Value="\${val}"
        done < <(aws ec2 describe-instances --region \$AWS_REGION --instance-id \${AWS_INSTANCE_ID} --query "Reservations[*].Instances[*].[Tags[*]]" --output text | grep ^prefix_)
        EOF
      - "chmod +x /usr/local/sbin/tag_ebs_volume.sh"
      - "/usr/local/sbin/tag_ebs_volume.sh"

Note that in this example it will propagate only tags with a specific prefix "prefix_"

Hope that may help someone

Apr 07 '20 10:04 JeremJR

Thank you @JeremJR , this helped a lot! 👍

Aug 21 '20 14:08 log2

Doing it from a script within the new instance won't be good enough for billing tags, as there is a short period where the volume exists, but is not tagged.

According to this https://forums.aws.amazon.com/thread.jspa?threadID=122354&start=25&tstart=0, There is a way to tag the volumes within an Auto Scaling Group via Launch Templates.

Apparently eksctl already uses launch templates, maybe all we need is a way to populate them with the tags.

Oct 05 '20 11:10 MartinEmrich

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

Jan 25 '21 02:01 github-actions[bot]

Just to confirm: It's the year 2021 and we have to manually add tags to EBS volumes mounted on to EC2 instances launched from EKS manually?

Like using that bash script above.

NOT COMPLAINING. just verifying :)

Jan 27 '21 20:01 bitva77

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

Feb 28 '21 02:02 github-actions[bot]

oh I ain't letting this die just out of spite now

Feb 28 '21 05:02 bitva77

The fact that this feature is still open and not prioritized is fascinating.

Mar 17 '21 05:03 vodwood

Will anyone take a look at this? We have some EBS volume enforce tagging policy. So we can't use user data to add tags later but only when invoking EC2 RunInstances API.

Mar 23 '21 05:03 liangxibing

Just want to piggyback on this; how is it in 2021 AWS doesn't allow a OU level SCP for tagging actually work because of major services that need to negate it? EBS volumes is one, and Fargate is another as from console it creates a CloudFormation stack that we require tags on , and it doesn't tag it. Come on AWS, stop creating 5000 new services per week and work on the EXISTING ONES. thanks in advance!

Apr 22 '21 15:04 mcburn13

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

May 23 '21 02:05 github-actions[bot]

This issue was closed because it has been stalled for 5 days with no activity.

May 28 '21 02:05 github-actions[bot]

Can this be re-opened?

Jan 07 '22 00:01 mike-stewart

Apologies, the reason this wasn't prioritised is that, apparently, there was a workaround, however bad, and there were other things with higher priority awaiting fixes and features all around. It's difficult to balance these things properly. Eventually, we'll get to things though, it just takes time.

Other than that, we always accept contributions. :) It's possible that this can be fixed via the launch template but it still requires time and effort and testing and verifying that the proposed solution actually works, even after the volume was rotated out.

That said, we'll take a look at this and see if the launch template is a viable solution or not.

Jan 07 '22 07:01 Skarlso

Actually, we already tag volumes with a specific tag through launch templates:

		}, gfnec2.LaunchTemplate_TagSpecification{
			ResourceType: gfnt.NewString("volume"),
			Tags:         cfnTags,

where cfnTags are

	cfnTags := []cloudformation.Tag{
		{
			Key:   gfnt.NewString("Name"),
			Value: gfnt.NewString(generateNodeName(ng, meta)),
		},
	}

I'm guessing you would like to insert custom tags in here?

Jan 07 '22 08:01 Skarlso

Actually, actually reading more into this. All tags, set through Tags spec in the nodegroup are applied to the LaunchTemplate.

	for k, v := range ng.Tags {
		cfnTags = append(cfnTags, cloudformation.Tag{
			Key:   gfnt.NewString(k),
			Value: gfnt.NewString(v),
		})
	}

Which is then applied as the above. ng.Tags is coming from:

	// Applied to the Autoscaling Group and to the EC2 instances (unmanaged),
	// Applied to the EKS Nodegroup resource and to the EC2 instances (managed)
	// +optional
	Tags map[string]string `json:"tags,omitempty"`

Configuration:

managedNodeGroups:
  - name: managed-ng-1
    minSize: 2
    desiredCapacity: 2
    maxSize: 4
    labels:
      role: worker
    tags:
      whatever: billing

The whatever: billing will be applied to

	launchTemplateTagSpecs = append(launchTemplateTagSpecs,
		gfnec2.LaunchTemplate_TagSpecification{
			ResourceType: gfnt.NewString("instance"),
			Tags:         cfnTags,
		}, gfnec2.LaunchTemplate_TagSpecification{
			ResourceType: gfnt.NewString("volume"),
			Tags:         cfnTags,
		}, gfnec2.LaunchTemplate_TagSpecification{
			ResourceType: gfnt.NewString("network-interface"),
			Tags:         cfnTags,
		})

The instance, the volume and the network-interface. Please verify if this is satisfactory.

Jan 07 '22 08:01 Skarlso

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

Feb 07 '22 01:02 github-actions[bot]

@Skarlso IDK about the original author of this issue, but for me, the tags are propagated to managed nodes and their volume, but not to the network interfaces.

Feb 07 '22 07:02 MartinEmrich

@MartinEmrich Yep, tried it. I can see that it doesn't propagate. We'll take a look.

Feb 07 '22 08:02 Skarlso

I can confirm that we are giving it the right value:

            {
              "ResourceType": "network-interface",
              "Tags": [
                {
                  "Key": "Name",
                  "Value": "test-tags-cluster-2-managed-ng-1-Node"
                },
                {
                  "Key": "whatever",
                  "Value": "billing"
                },
                {
                  "Key": "alpha.eksctl.io/nodegroup-name",
                  "Value": "managed-ng-1"
                },
                {
                  "Key": "alpha.eksctl.io/nodegroup-type",
                  "Value": "managed"
                }
              ]
            }

This might be an AWS issue. But I'll investigate further.

Feb 07 '22 08:02 Skarlso

Oh freck.

The type of resource to tag. Currently, the resource types that support tagging on creation are instance and volume. To tag a resource after it has been created, see CreateTags.

Well.... :/

Feb 07 '22 08:02 Skarlso

So this gets into the area now about tagging things after creation and out of the scope of the cloudformation stack which is that... Everything we do after creation will have the effect that it's temporary. eksctl will not monitor tags and won't reapply them. And since these resources are controlled by the ASG, or the autoscaler, new instances will not have this tag, since the launchtemplate doesn't support them.

Thus, if an instance is rotated out, that's it. The tags are lost. And we can't do anything about that I'm afraid until there is AWS support for it through the LaunchTemplate.

Feb 07 '22 08:02 Skarlso

I raised this issue internally, let's see if there is any future regarding tagging ENIs at least.

Feb 07 '22 08:02 Skarlso

I came across this issue because I'm trying to solve the challenge of parsing the Cost & Usage Report and figuring out which volume went with which EC2 instance.

Doing it from a script within the new instance won't be good enough for billing tags, as there is a short period where the volume exists, but is not tagged.

According to this https://forums.aws.amazon.com/thread.jspa?threadID=122354&start=25&tstart=0, There is a way to tag the volumes within an Auto Scaling Group via Launch Templates.

Apparently eksctl already uses launch templates, maybe all we need is a way to populate them with the tags.

Unfortunately that forum link doesn't work any more so I don't know what it said. Looking at other blog posts and the like, I can see that tags and their corresponding values can be set in the launch template.

But because they are fixed values, I don't see any way of setting a dynamic value like the EC2 instance ID.

Is there a solution to this?

Sep 08 '22 14:09 pcolmer

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

Oct 29 '22 02:10 github-actions[bot]

This issue was closed because it has been stalled for 5 days with no activity.

Nov 04 '22 02:11 github-actions[bot]