cfn submit fails with opaque `Internal Failure`

[bowlofarugula]

rpdk.log is attached.

I'm registering a custom resource type in multiple AWS accounts using the typescript plugin. I've had success in most accounts and am already using the custom resource in a few stacks. But in a particular account, cfn submit always fails while waiting for registration to complete:

Successfully submitted type. Waiting for registration with token 'c372ede9-e066-4d70-9cdd-6894c7a1e715' to complete.
Failed to register the type with registration token 'c372ede9-e066-4d70-9cdd-6894c7a1e715'.
Please see response for additional information: '{'ProgressStatus': 'FAILED', 'Description': 'Deployment is currently in DEPLOY_STAGE of status FAILED\nDeployment failed with error. Error message: [c372ede9-e066-4d70-9cdd-6894c7a1e715] Internal Failure', 'TypeArn': 'arn:aws:cloudformation:us-east-1:135761614524:type/resource/Extend-Confluent-RoleBinding', 'TypeVersionArn': 'arn:aws:cloudformation:us-east-1:135761614524:type/resource/Extend-Confluent-RoleBinding/00000001', 'ResponseMetadata': {'RequestId': '38daa4e7-90bd-4192-b5b9-a569cf402b70', 'HTTPStatusCode': 200, 'HTTPHeaders': {'x-amzn-requestid': '38daa4e7-90bd-4192-b5b9-a569cf402b70', 'content-type': 'text/xml', 'content-length': '789', 'date': 'Thu, 31 Mar 2022 00:32:16 GMT'}, 'RetryAttempts': 0}}'
=== Caught downstream error ===
Waiter TypeRegistrationComplete failed: Waiter encountered a terminal failure state: For expression "ProgressStatus" we matched expected path: "FAILED"

I've retried multiple times, tried deleting and recreating CloudFormationManagedUploadInfrastructure, with no luck. I've confirmed that I can submit other resource types to this account without issue, and I can submit this resource type to other accounts without issue. The error message is always the same, aside from a different registration token each time.

[bowlofarugula]

My current hunch is there might be some bad state for this resource type in an AWS-owned place that I'm not able to manage.

[bowlofarugula]

I poked a bit and found describe-type-registration shows this.

$ aws --no-cli-pager cloudformation describe-type-registration --registration-token c372ede9-e066-4d70-9cdd-6894c7a1e715
{
    "ProgressStatus": "FAILED",
    "Description": "Deployment is currently in DEPLOY_STAGE of status FAILED\nDeployment failed with error. Error message: [c372ede9-e066-4d70-9cdd-6894c7a1e715] Internal Failure",
    "TypeArn": "arn:aws:cloudformation:us-east-1:135761614524:type/resource/Extend-Confluent-RoleBinding",
    "TypeVersionArn": "arn:aws:cloudformation:us-east-1:135761614524:type/resource/Extend-Confluent-RoleBinding/00000001"
}

And deregister-type can't find the type.

$ aws cloudformation deregister-type --arn 'arn:aws:cloudformation:us-east-1:135761614524:type/resource/Extend-Confluent-RoleBinding'

An error occurred (TypeNotFoundException) when calling the DeregisterType operation: The type 'null' cannot be found.

$ aws cloudformation deregister-type --type-name 'Extend::Confluent::RoleBinding' --type 'RESOURCE'

An error occurred (TypeNotFoundException) when calling the DeregisterType operation: The type 'Extend::Confluent::RoleBinding' cannot be found.

This has now happened with a few resource types in two accounts. Any type registration that gets stuck in this state continues to fail in the same way. All instances of the failing types can register without issue in other accounts.

[NickDarvey]

We're seeing the same thing.

Repro

# first install the execution role
aws cloudformation create-stack \
  --template-url https://community-resource-provider-catalog.s3.amazonaws.com/community-servicequotas-s3-resource-role-0.1.0.yml \
  --stack-name community-servicequotas-s3-resource-role \
  --capabilities CAPABILITY_IAM

aws cloudformation wait stack-create-complete \
  --stack-name community-servicequotas-s3-resource-role

# get the value of the ExecutionRoleArn Output
aws cloudformation describe-stacks \
  --stack-name community-servicequotas-s3-resource-role

# register the cloudformation type
aws cloudformation register-type \
  --type-name Community::ServiceQuotas::S3 \
  --type RESOURCE \
  --schema-handler-package s3://community-resource-provider-catalog/community-servicequotas-s3-0.1.0.zip \
  --execution-role <execution-role-arn>

Actual

❯ aws cloudformation describe-type-registration --registration-token  "f9f2a953-75bf-4fb1-a78f-55091e83f2b4"  --profile Me --region us-east-1
{
    "ProgressStatus": "FAILED",
    "Description": "Deployment is currently in DEPLOY_STAGE of status FAILED\nDeployment failed with error. Error message: [f9f2a953-75bf-4fb1-a78f-55091e83f2b4] Internal Failure",
    "TypeArn": "arn:aws:cloudformation:us-east-1:1234:type/resource/Community-ServiceQuotas-S3",
    "TypeVersionArn": "arn:aws:cloudformation:us-east-1:1234:type/resource/Community-ServiceQuotas-S3/00000001"
}

[NickDarvey]

@IanMExtend you should give it another go now. It looks AWS resolved the issue.

[sdoppalsm]

I am facing similar issue, did anyone get around this ?

[yeejus]

I am facing this issue locally as well. Did anyone have any resolution?

[filatushkin]

I've got the same problem when attempted to run an operation with a wrong --region. Once --region was set, it started to work