cloudformation-coverage-roadmap icon indicating copy to clipboard operation
cloudformation-coverage-roadmap copied to clipboard

AWS::EC2::VPC - False-positive drift on EnableDnsHostnames and EnableDnsSupport when using Git Sync

Open gautaz opened this issue 1 year ago • 13 comments

Name of the resource

AWS::EC2::VPC

Resource Name

No response

Issue Description

EDIT (2024-09-28): The issue only occurs if Git Sync is used.

AWS is detecting a drift on a newly created VPC with EnableDnsHostnames and EnableDnsSupport set to true. The "Actual" output is missing both values.

Expected Behavior

No drift should be detected.

Observed Behavior

Here is the drift report:

image

This is inconsistent with what is displayed in the VPC details:

image

Test Cases

EDIT (2024-09-28): The following template only reproduces the issue if the deployment is done with Git Sync.

Here is an extract of the provided template:

AWSTemplateFormatVersion: '2010-09-09'
Description: stack template
Parameters:
  VpcCidrBlock:
    Type: String
Resources:
  VPC:
    Properties:
      CidrBlock: !Ref 'VpcCidrBlock'
      EnableDnsHostnames: true
      EnableDnsSupport: true
    Type: AWS::EC2::VPC

Other Details

No response

gautaz avatar Sep 24 '24 09:09 gautaz

My local test does not replicate this issue. Can you provide a region or stackid?

wangcaro3793 avatar Sep 27 '24 15:09 wangcaro3793

Yes, the region is eu-west-3 and here is the stack id: arn:aws:cloudformation:eu-west-3:432287323027:stack/DevelopmentStack/9f88fb50-7c05-11ef-8392-0e9fe9bec705

gautaz avatar Sep 27 '24 16:09 gautaz

if you try with the extracted sample template, do you still see the same issue? If so, can you share the stack id?

wangcaro3793 avatar Sep 27 '24 18:09 wangcaro3793

Max quota for VPC per region reached on our account. I am asking for a quota increase in order to test. The request is still pending.

gautaz avatar Sep 27 '24 19:09 gautaz

I have just deployed a new stack with the following id: arn:aws:cloudformation:eu-west-3:432287323027:stack/issue-2138/2b504a60-7d07-11ef-8cec-0eae10b2d427

The template is:

AWSTemplateFormatVersion: '2010-09-09'
Description: stack template
Resources:
  VPC:
    Properties:
      CidrBlock: '172.50.0.0/16'
      EnableDnsHostnames: true
      EnableDnsSupport: true
    Type: AWS::EC2::VPC

No drift is detected in this case so this means other resources probably need to be deployed to reproduce the issue.

If you need anymore testing, do not hesitate to ask.

EDIT: The full template reproducing the issue is quite large but I can provide it if needed.

gautaz avatar Sep 27 '24 19:09 gautaz

We also have a smaller stack reproducing the issue: arn:aws:cloudformation:eu-west-3:432287323027:stack/HubStack/9afb6ed0-7c03-11ef-8098-0ac092c2d633

Apart from the VPC, this stack also deploys private subnets with route tables and also some VPC endpoints.

Here is the template
AWSTemplateFormatVersion: '2010-09-09'
Outputs:
  EcrApiVpcEndpointRegionDnsEntry:
    Export:
      Name: ServicesHubEcrApiVpcEndpointRegionDnsEntry
    Value: !Select
      - 0
      - !GetAtt 'EcrApiVpcEndpoint.DnsEntries'
  EcrDkrVpcEndpointRegionDnsEntry:
    Export:
      Name: ServicesHubEcrDkrVpcEndpointRegionDnsEntry
    Value: !Select
      - 0
      - !GetAtt 'EcrDkrVpcEndpoint.DnsEntries'
  EcsAgentVpcEndpointRegionDnsEntry:
    Export:
      Name: ServicesHubEcsAgentVpcEndpointRegionDnsEntry
    Value: !Select
      - 0
      - !GetAtt 'EcsAgentVpcEndpoint.DnsEntries'
  EcsTelemetryVpcEndpointRegionDnsEntry:
    Export:
      Name: ServicesHubEcsTelemetryVpcEndpointRegionDnsEntry
    Value: !Select
      - 0
      - !GetAtt 'EcsTelemetryVpcEndpoint.DnsEntries'
  EcsVpcEndpointRegionDnsEntry:
    Export:
      Name: ServicesHubEcsVpcEndpointRegionDnsEntry
    Value: !Select
      - 0
      - !GetAtt 'EcsVpcEndpoint.DnsEntries'
  LogsVpcEndpointRegionDnsEntry:
    Export:
      Name: ServicesHubLogsVpcEndpointRegionDnsEntry
    Value: !Select
      - 0
      - !GetAtt 'LogsVpcEndpoint.DnsEntries'
  PrivateSubnet0CidrBlock:
    Export:
      Name: ServicesHubPrivateSubnet0CidrBlock
    Value: !GetAtt 'PrivateSubnet0.CidrBlock'
  PrivateSubnet0RouteTableId:
    Export:
      Name: ServicesHubPrivateSubnet0RouteTableId
    Value: !Ref 'PrivateSubnet0RouteTable'
  PrivateSubnet1CidrBlock:
    Export:
      Name: ServicesHubPrivateSubnet1CidrBlock
    Value: !GetAtt 'PrivateSubnet1.CidrBlock'
  PrivateSubnet1RouteTableId:
    Export:
      Name: ServicesHubPrivateSubnet1RouteTableId
    Value: !Ref 'PrivateSubnet1RouteTable'
  PrivateSubnet2CidrBlock:
    Export:
      Name: ServicesHubPrivateSubnet2CidrBlock
    Value: !GetAtt 'PrivateSubnet2.CidrBlock'
  PrivateSubnet2RouteTableId:
    Export:
      Name: ServicesHubPrivateSubnet2RouteTableId
    Value: !Ref 'PrivateSubnet2RouteTable'
  SqsVpcEndpointRegionDnsEntry:
    Export:
      Name: ServicesHubSqsVpcEndpointRegionDnsEntry
    Value: !Select
      - 0
      - !GetAtt 'SqsVpcEndpoint.DnsEntries'
  VpcId:
    Export:
      Name: ServicesHubVpcId
    Value: !GetAtt 'VPC.VpcId'
Parameters:
  PrivateSubnet0CidrBlock:
    Type: String
  PrivateSubnet1CidrBlock:
    Type: String
  PrivateSubnet2CidrBlock:
    Type: String
  VpcCidrBlock:
    Type: String
Resources:
  EcrApiVpcEndpoint:
    Properties:
      SecurityGroupIds:
        - !Ref 'VpcEndpointsSecurityGroup'
      ServiceName: !Sub 'com.amazonaws.${AWS::Region}.ecr.api'
      SubnetIds:
        - !Ref 'PrivateSubnet0'
        - !Ref 'PrivateSubnet1'
        - !Ref 'PrivateSubnet2'
      VpcEndpointType: Interface
      VpcId: !Ref 'VPC'
    Type: AWS::EC2::VPCEndpoint
  EcrDkrVpcEndpoint:
    Properties:
      SecurityGroupIds:
        - !Ref 'VpcEndpointsSecurityGroup'
      ServiceName: !Sub 'com.amazonaws.${AWS::Region}.ecr.dkr'
      SubnetIds:
        - !Ref 'PrivateSubnet0'
        - !Ref 'PrivateSubnet1'
        - !Ref 'PrivateSubnet2'
      VpcEndpointType: Interface
      VpcId: !Ref 'VPC'
    Type: AWS::EC2::VPCEndpoint
  EcsAgentVpcEndpoint:
    Properties:
      SecurityGroupIds:
        - !Ref 'VpcEndpointsSecurityGroup'
      ServiceName: !Sub 'com.amazonaws.${AWS::Region}.ecs-agent'
      SubnetIds:
        - !Ref 'PrivateSubnet0'
        - !Ref 'PrivateSubnet1'
        - !Ref 'PrivateSubnet2'
      VpcEndpointType: Interface
      VpcId: !Ref 'VPC'
    Type: AWS::EC2::VPCEndpoint
  EcsTelemetryVpcEndpoint:
    Properties:
      SecurityGroupIds:
        - !Ref 'VpcEndpointsSecurityGroup'
      ServiceName: !Sub 'com.amazonaws.${AWS::Region}.ecs-telemetry'
      SubnetIds:
        - !Ref 'PrivateSubnet0'
        - !Ref 'PrivateSubnet1'
        - !Ref 'PrivateSubnet2'
      VpcEndpointType: Interface
      VpcId: !Ref 'VPC'
    Type: AWS::EC2::VPCEndpoint
  EcsVpcEndpoint:
    Properties:
      SecurityGroupIds:
        - !Ref 'VpcEndpointsSecurityGroup'
      ServiceName: !Sub 'com.amazonaws.${AWS::Region}.ecs'
      SubnetIds:
        - !Ref 'PrivateSubnet0'
        - !Ref 'PrivateSubnet1'
        - !Ref 'PrivateSubnet2'
      VpcEndpointType: Interface
      VpcId: !Ref 'VPC'
    Type: AWS::EC2::VPCEndpoint
  LogsVpcEndpoint:
    Properties:
      SecurityGroupIds:
        - !Ref 'VpcEndpointsSecurityGroup'
      ServiceName: !Sub 'com.amazonaws.${AWS::Region}.logs'
      SubnetIds:
        - !Ref 'PrivateSubnet0'
        - !Ref 'PrivateSubnet1'
        - !Ref 'PrivateSubnet2'
      VpcEndpointType: Interface
      VpcId: !Ref 'VPC'
    Type: AWS::EC2::VPCEndpoint
  PrivateSubnet0:
    Properties:
      AvailabilityZone: !Select
        - 0
        - !GetAZs
          Ref: AWS::Region
      CidrBlock: !Ref 'PrivateSubnet0CidrBlock'
      VpcId: !Ref 'VPC'
    Type: AWS::EC2::Subnet
  PrivateSubnet0RouteTable:
    Properties:
      VpcId: !Ref 'VPC'
    Type: AWS::EC2::RouteTable
  PrivateSubnet0RouteTableAssociation:
    Properties:
      RouteTableId: !Ref 'PrivateSubnet0RouteTable'
      SubnetId: !Ref 'PrivateSubnet0'
    Type: AWS::EC2::SubnetRouteTableAssociation
  PrivateSubnet1:
    Properties:
      AvailabilityZone: !Select
        - 1
        - !GetAZs
          Ref: AWS::Region
      CidrBlock: !Ref 'PrivateSubnet1CidrBlock'
      VpcId: !Ref 'VPC'
    Type: AWS::EC2::Subnet
  PrivateSubnet1RouteTable:
    Properties:
      VpcId: !Ref 'VPC'
    Type: AWS::EC2::RouteTable
  PrivateSubnet1RouteTableAssociation:
    Properties:
      RouteTableId: !Ref 'PrivateSubnet1RouteTable'
      SubnetId: !Ref 'PrivateSubnet1'
    Type: AWS::EC2::SubnetRouteTableAssociation
  PrivateSubnet2:
    Properties:
      AvailabilityZone: !Select
        - 2
        - !GetAZs
          Ref: AWS::Region
      CidrBlock: !Ref 'PrivateSubnet2CidrBlock'
      VpcId: !Ref 'VPC'
    Type: AWS::EC2::Subnet
  PrivateSubnet2RouteTable:
    Properties:
      VpcId: !Ref 'VPC'
    Type: AWS::EC2::RouteTable
  PrivateSubnet2RouteTableAssociation:
    Properties:
      RouteTableId: !Ref 'PrivateSubnet2RouteTable'
      SubnetId: !Ref 'PrivateSubnet2'
    Type: AWS::EC2::SubnetRouteTableAssociation
  SqsVpcEndpoint:
    Properties:
      SecurityGroupIds:
        - !Ref 'VpcEndpointsSecurityGroup'
      ServiceName: !Sub 'com.amazonaws.${AWS::Region}.sqs'
      SubnetIds:
        - !Ref 'PrivateSubnet0'
        - !Ref 'PrivateSubnet1'
        - !Ref 'PrivateSubnet2'
      VpcEndpointType: Interface
      VpcId: !Ref 'VPC'
    Type: AWS::EC2::VPCEndpoint
  VPC:
    Properties:
      CidrBlock: !Ref 'VpcCidrBlock'
      EnableDnsHostnames: true
      EnableDnsSupport: true
    Type: AWS::EC2::VPC
  VpcEndpointsSecurityGroup:
    Properties:
      GroupDescription: VPC Endpoints firewall rules
      SecurityGroupIngress:
        - CidrIp: '0.0.0.0/0'
          Description: HTTPS access to AWS APIs
          FromPort: 443
          IpProtocol: tcp
          ToPort: 443
      VpcId: !Ref 'VPC'
    Type: AWS::EC2::SecurityGroup

Hope this helps.

gautaz avatar Sep 27 '24 19:09 gautaz

Hello @wangcaro3793,

I have reproduced the issue with a minimal example. The issue occurs only when using Git Sync.

Here are the YAML files I did use on my Git repository:

stack/cfn-issue-2138/deployment.yaml:

template-file-path: 'stack/cfn-issue-2138/template.yaml'

stack/cfn-issue-2138/template.yaml:

AWSTemplateFormatVersion: '2010-09-09'
Description: cfn issue 2138 stack template
Resources:
  VPC:
    Properties:
      CidrBlock: '172.50.0.0/16'
      EnableDnsHostnames: true
      EnableDnsSupport: true
    Type: AWS::EC2::VPC

These have been used to deploy the following stack with Git Sync: arn:aws:cloudformation:eu-west-3:432287323027:stack/CfnIssue2138/bbbefb50-7d61-11ef-9733-0655824fe323

And this stack also reproduces the drift issue on its VPC.

gautaz avatar Sep 28 '24 06:09 gautaz

Hello @wangcaro3793, did you manage to reproduce the issue with Git Sync?

gautaz avatar Oct 03 '24 14:10 gautaz

hi @gautaz I don't have git sync setup so I'm not sure. But thanks for identifying that this is an issue with Git Sync. I'll have to loop in different support.

wangcaro3793 avatar Oct 04 '24 15:10 wangcaro3793

Hello @wangcaro3793,

Do I need to keep the stacks which helped demonstrating the issue or is the issue now reproduced on AWS side? There is no problem in keeping them for a while, I just do not want to keep unused resources lingering without being ever used.

gautaz avatar Oct 15 '24 13:10 gautaz

no need to keep the stack you can tear it down @gautaz

wangcaro3793 avatar Oct 16 '24 19:10 wangcaro3793

Hi @gautaz, I tried to reproduce this issue using git sync but also was not able to. Are you still experiencing this issue?

Here's my reproduction method:

Created a repository with deployment.yaml containing:

template-file-path: 'template.yaml'

and template.yaml containing:

AWSTemplateFormatVersion: '2010-09-09'
Description: stack template
Resources:
  VPC:
    Properties:
      CidrBlock: 10.0.0.0/16
      EnableDnsHostnames: true
      EnableDnsSupport: true
    Type: AWS::EC2::VPC

Created a CloudFormation stack synced from this repository:

Image

Detected drift on the stack and saw none:

Image

Created a commit to the repository to change template.yaml to:

AWSTemplateFormatVersion: '2010-09-09'
Description: stack template
Resources:
  VPC:
    Properties:
      CidrBlock: 10.0.0.0/16
      EnableDnsHostnames: 'true'
      EnableDnsSupport: 'true'
      Tags:
       - Key: stack
         Value: production
    Type: AWS::EC2::VPC

and waited until the commit deployed the change to CloudFormation. Then detected drift and again saw none:

Image

Verified that the state of the VPC is as desired WRT to EnableDnsHostnames / EnableDnsSupport, as shown in your screenshot:

Image

Modified the DNS settings manually via the VPC console:

Image

Then detected drift and saw that it did show the drift as expected:

Image

Modified the DNS settings manually back to the CFN state and updated the drift, and confirmed that there is again none:

Image

So my experience is that even with Git Sync, the drift detection behavior is working as expected

gravitylow avatar Dec 03 '25 17:12 gravitylow

Hello @gravitylow,

The issue is still reproduced on our stacks, even for stacks deployed a few days ago (i.e. 2025-12-02):

Image

Here is the relevant part in the template.yaml file that was used to deploy this stack (with git sync):

  VPC:
    Properties:
      CidrBlock: !Ref 'VpcCidrBlock'
      EnableDnsHostnames: true
      EnableDnsSupport: true
    Type: AWS::EC2::VPC

EDIT: I will try to reproduce again the issue with a simpler template because all our stacks' templates are currently quite large. I also just saw that your comment dates back to 2025-12-03, is there any chance that the fix also dates back to 2025-12-03 and that it does not resolve the issue retroactively on older stacks? All our stacks are also deployed on eu-west-3, does the fix apply to this region?

gautaz avatar Dec 10 '25 13:12 gautaz

Hello @gravitylow,

I have just redeployed my original minimal example reproducing the issue. I can confirm that the issue is still reproduced:

Image

Here is the ARN of the stack: arn:aws:cloudformation:eu-west-3:432287323027:stack/cfn-issue-2138/66beb530-d9ce-11f0-b1ab-068fb0765d6d

gautaz avatar Dec 15 '25 16:12 gautaz

Hi @gautaz , thanks for providing the stack ARN for the issue, I was able to figure out why this is happening for you with that information.

As it turns out, the IAM role which you are using does not have permission to invoke the ec2:DescribeVpcAttribute action, which is what CloudFormation uses to determine the state of EnableDnsSupport and EnableDnsHostnames. If you check your account's CloudTrail logs, you should see the authorization failure which results. Since these attributes are not present in the ec2:DescribeVpcs response output, access to ec2:DescribeVpcAttribute is required in order to determine the current value. By adding access to this action in your role's IAM policy, CloudFormation should be able to determine the current value and confirm that the resource has not drifted.

gravitylow avatar Dec 18 '25 18:12 gravitylow

Hello @gravitylow, thanks a lot for your support, adding ec2:DescribeVpcAttribute fixed the drift status. I also found the errors in CloudTrail.

Before closing the issue, I am just wondering if CloudFormation shouldn't tell that it cannot make the resource comparison and the reason why rather than raising a false drift status. Is this an already known issue?

gautaz avatar Dec 19 '25 14:12 gautaz