Transform Support for Cloudformation Stacksets

[akshayvvr]

Title -> SAM support for Cloudformation Stacksets Scope of request -> CloudFormation Stacksets should be able to support templates Expected behavior -> It should allow the definition of a transform in the stackset template or convert it to normal Cloudformation stack in the backend.

[benbridts]

I've been thinking about this today, some notes:

• Supporting any transform would be great, but can be challenging
• The transform would need to either run in the StackSet account or the StackInstance account. For most SAM resources this will not make it difference.
• caveat/exception: AWS::Serverless::Application, where the tranform needs permission to see the application and the StackInstance needs to be able to read the template from S3.
• caveat/exception: every upload to an S3 bucket needs to be readable by all the StackInstances. This should be solvable by using the organization id in a bucket policy, but might lead to errors that are not immediately obvious when the user forgets to do this.

[erkwish]

Now with the new Organization support this is something that I would really appreciate.

[temyers]

There is a current workaround supporting this use-case, by using nested stacks

Example https://s3.amazonaws.com/my-s3-bucket/stackset-spike/serverless-transform-stack.yml:

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: Create pipeline-dashboard application.
Resources:
  mechanicalrockpipelinedashboard:
    Type: AWS::Serverless::Application
    Properties:
      Location:
        ApplicationId: arn:aws:serverlessrepo:us-east-1:611781478414:applications/mechanicalrock-pipeline-dashboard
        SemanticVersion: 1.0.3
      Parameters:
        PipelinePattern: '*'

Deploy your stack set template:

Resources:
  serverlessTemplate:
    Type: AWS::CloudFormation::Stack
    Properties: 
      TemplateURL: "https://s3.amazonaws.com/my-s3-bucket/stackset-spike/serverless-transform-stack.yml"

Things of note:

• Target accounts need access to the s3 bucket containing your nested stack reference.
• The stack set deployment results in multiple nested stacks (as expected) meaning there's a possibility of confusion/maintenance burden
• Complicates your CICD flow as you need to package/distribute/generate your indirection template

[jaidisido]

There is a current workaround supporting this use-case, by using nested stacks

Example https://s3.amazonaws.com/my-s3-bucket/stackset-spike/serverless-transform-stack.yml:

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: Create pipeline-dashboard application.
Resources:
  mechanicalrockpipelinedashboard:
    Type: AWS::Serverless::Application
    Properties:
      Location:
        ApplicationId: arn:aws:serverlessrepo:us-east-1:611781478414:applications/mechanicalrock-pipeline-dashboard
        SemanticVersion: 1.0.3
      Parameters:
        PipelinePattern: '*'

Deploy your stack set template:

Resources:
  serverlessTemplate:
    Type: AWS::CloudFormation::Stack
    Properties: 
      TemplateURL: "https://s3.amazonaws.com/my-s3-bucket/stackset-spike/serverless-transform-stack.yml"

Things of note:

• Target accounts need access to the s3 bucket containing your nested stack reference.
• The stack set deployment results in multiple nested stacks (as expected) meaning there's a possibility of confusion/maintenance burden
• Complicates your CICD flow as you need to package/distribute/generate your indirection template

@temyers could you please specify how you enabled cross-account s3 access to the bucket? I have created a role in the parent account providing the child account ID s3:* on the bucket

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::ChildAccountId:root"
      },
      "Action": "sts:AssumeRole",
      "Condition": {}
    }
  ]
}

but still get an Access Denied

[rhanson32]

@jaidisido did you allow cross account access on the bucket itself? You need to enable cross account access via bucket policy. The bucket itself is not going to support cross account access natively.

[temyers]

For the purposes of the demonstration above, the bucket was publicly readable.

For real world usage, a bucket policy is required grant cross-account access

[PatMyron]

https://aws.amazon.com/about-aws/whats-new/2021/04/you-can-now-use-macros-and-transforms-in-cloudformation-templates-to-create-aws-cloudformation-stacksets/

[shnccg]

Any update on this? Still cannot find an easy way to deploy stacksets with transforms across multiple accounts/regions all integrated with Codepipeline.

[jyoungs]

I assume this is still open because it doesn't support the SERVICE_MANAGED permission model ☹️

[rhanson32]

That's my assumption as well. Would really like to use the service managed permission model with stacksets including SAM resources.

[jyoungs]

As a workaround, is there any way to run the transforms manually?

My template at hand only has one AWS::Serverless::Function which shouldn't be difficult to convert to an AWS::Lambda::Function (thus eliminating all transforms and side-stepping this issue). The thing is that I still want to use SAM to package it, so I want to use AWS::Serverless::Function initially, but I should be able to "transform" locally after the CodeURI is materialized. (I'm sure I could do it myself, but that seems like something I shouldn't build)

[tbenbrahim]

This is really needed for developers working on teams, where we need separate SAM templates to prevent overwriting each other, yet an organized way to deploy all of templates that comprise the application. So for example, create a DynamoDB table in one stack and export it, and two lambdas in two separate stacks making using of the imported table, with a single command to deploy/update the stack set when migrating to another account. Right now the single command is an AWS CLI script. Why a single SAM template does not work: I deploy my template to test a lambda. My colleague who is working on a different lambda, loses all of the changes, and his lambda no longer works, his new table and policy and role disappears, etc.... When he notices, he deploys his SAM template, and I lose my changes. Test over.

[Manouchehri]

@jyoungs Did you ever find a good workaround?

One idea I had was to use the CDK to roll a set of Stacks instead of StackSets, not sure if what I'd be losing out by ditching StackSets though. https://dev.to/aws-builders/build-serverless-applications-using-cdk-and-sam-4oig

[reidca]

This is another paper cut when using CloudFormation. For years I was frustrated that I could not use macros in StackSets and had to orchestrate my own stacks for each member account needing to use the template. Yesterday I searched to see if this was fixed and was pleased to see the post that support was there. Except now I find I cannot use it in Service Managed stacks which is really the only way Stacksets are used in my organization. Very frustrating and will cause me to waste more time to create yet more work-arounds.