copilot-cli icon indicating copy to clipboard operation
copilot-cli copied to clipboard
verified publisher icon aws

Bug with SSM secrets?

Open afgallo opened this issue 3 years ago • 19 comments

No matter how hard I try, I can't really get passed this error:

Invalid request provided: Create T
askDefinition: The Systems Manager parameter name specified for secret
JWT_KEY is invalid. The parameter name can be up to 2048 characters a
nd include the following letters and symbols: a-zA-Z0-9_.-

JWT_KEY seems to be perfectly fine as a param name.

I've reviewed this page a dozen times and still can't find what I am missing https://aws.github.io/copilot-cli/docs/commands/secret-init/

Any help is really appreciated

UPDATE: I'm running copilot v1.16.0 This is a Load Balanced Web Service

afgallo avatar Apr 10 '22 21:04 afgallo

Hello, @afgallo! Can you tell me a bit more? I have tried to replicate this error with your param name, but didn't get the same error. Did secret init work for you? Did you put the secret name into your Load Balanced Web Service manifest? Did you get the error when you ran svc deploy? Thanks!

huanjani avatar Apr 11 '22 16:04 huanjani

Hi @huanjani yes, I basically ran secret init and followed the prompts as per below:

image

Then I added the following to my manifest file which is a load balanced service:

image

I got the error when I ran svc deploy

I wonder if that was an issue with the AWS api?

afgallo avatar Apr 12 '22 07:04 afgallo

I'm still unable to replicate this perplexing error 🤔 .... I'll keep looking into it, but as a workaround to get you going, can you manually create JWT_KEY through the SSM Parameter Store, being sure to tag it with copilot-application and copilot-environment? (https://aws.github.io/copilot-cli/docs/developing/secrets/)

huanjani avatar Apr 12 '22 20:04 huanjani

Okay, I'm finally getting that error! Stay tuned...

huanjani avatar Apr 12 '22 22:04 huanjani

Thanks for not giving up on this, @huanjani! I was getting concerned you would not be able to reproduce it as it somehow worked for me today 🤔

afgallo avatar Apr 13 '22 11:04 afgallo

Oh, really! That's good to know! At least you're unblocked. I was going to suggest deleting and trying again, but that wasn't a solution that got to the bottom of this. 🤣

The way I got it was atypical, though, and now it's working! 😤

huanjani avatar Apr 13 '22 15:04 huanjani

👋 I'm also experiencing this issue when setting an IP address as a secret with copilot secret init. Important to note, that this was working previously, but I needed to add a domain and HTTPS endpoint for our app and so after destroying and recreating the app and attempting to deploy a service I receive:

    Resource handler returned message: "Invalid request provided: Create T                                   
    askDefinition: The Systems Manager parameter name specified for secret                                   
     MY_SECRET_HERE is invalid. The parameter name can be up to 2048 character                                   
    s and include the following letters and symbols: a-zA-Z0-9_.-, (Servic                                   
    e: AmazonECS; Status Code: 400; Error Code: ClientException; Request I                                   
    D: 967b0aac-3f81-4910-8e05-52601c8897eb; Proxy: null)" (RequestToken:                                    
    5d80ebd9-b11a-02b2-27da-43040b7fc2a9, HandlerErrorCode: InvalidRequest                                   
    )

I have also tried deleting and manually recreating the secret in SSM parameter store, but no luck.

Sidenote: I just updated to 1.19 today from 1.18 and am running a "Load Balanced Web Service"

UPDATE: In a bizarre twist of events, I renamed my secrets, which all began with MONGO_ (for MongoDB) to begin with just DB_ instead, "hand typed" the secrets instead of pasting them from a .env in vsCode and it started working 🤔 Now, I'm wondering if something with copy/paste was inserting some incorrect characters into the cli command OR if renaming had any bearing.

aflansburg avatar Jul 01 '22 15:07 aflansburg

Hi @aflansburg ! That's so strange 🤔 I wonder if the hypothesis can be validated by running copilot svc package and seeing if the secret names in the TaskDefinition contain any weird characters 💭

But I'm glad you got it working!

efekarakus avatar Jul 01 '22 18:07 efekarakus

Hi @efekarakus , @huanjani I got the same problem! When Codepipeline running trigger by code merged, at the last phase image

checked on Cloudformation: image

Try to delete a parameter store, and recreate by copilot secret init but still error.

I tried by create a change set from Cloudformation by the template I uploaded, It's updated successful without any errors. But it's just a workaround solution...I need the pipeline working well...

mvn-bachhuynh-dn avatar Jul 05 '22 10:07 mvn-bachhuynh-dn

image Sometime is another param

mvn-bachhuynh-dn avatar Jul 05 '22 11:07 mvn-bachhuynh-dn

Hi @mvn-bachhuynh-dn ! thank you for reaching out to us, would you mind copy pasting the SSM parameter names in secrets if there is nothing confidential? The whole secrets block would be super helpful for us to try to reproduce the issue

efekarakus avatar Jul 05 '22 20:07 efekarakus

I ended up having this issue again, but determined it was a copy and paste error. When receiving the guidance from the CLI to update the manifest file with the secrets I was copying output like this:

MY_SECRET: /copilot/${COPILOT_APPLICATION_NAME}/${COPILOT_ENVIRONMENT_NAME}/secrets/MY_SECRET`

and leaving the character ` in the manifest file........ removing that fixed the issue 🤦

aflansburg avatar Jul 05 '22 20:07 aflansburg

omg!

ok it sounds like if we print with code blocks ``` instead of just the ` character this bug scenario would go away.

We should replace https://github.com/aws/copilot-cli/blob/a28a4682b4bf53e616c7840077c30e6224170498/internal/pkg/cli/secret_init.go#L419 with color.HighlightCodeBlock

efekarakus avatar Jul 05 '22 20:07 efekarakus

Hey all! I did some digging and was able to reproduce the issue (thanks @aflansburg!) and find something that I think makes this a hard bug to track down:

To reproduce:

  1. create secret TEST_SECRET_ONE (i copy-pasted the same value in for each of my env’s)
  2. update manifest with secret correctly (no trailing `, key and value copy&pasted from terminal.app directly into vscode)
  3. Run copilot svc deploy - it works fine
  4. add ` to the end of secret value in manifest
  5. copilot svc deploy, doesn’t work (relevant cfn in copilot svc package:
Secrets:
  - Name: TEST_SECRET_ONE
    ValueFrom: /copilot/playground/beta/secrets/TEST_SECRET_ONE`

Error message:

Resource handler returned message: "Invalid request provided: Create T                                       
    askDefinition: The Systems Manager parameter name specified for secret                                       
     TEST_SECRET_ONE is invalid. The parameter name can be up to 2048 char                                       
    acters and include the following letters and symbols: a-zA-Z0-9_.-, (S                                       
    ervice: AmazonECS; Status Code: 400; Error Code: ClientException; Requ                                       
    est ID: 44b16b74-452a-4e55-ae73-10c7ae5f611e; Proxy: null)" (RequestTo                                       
    ken: ba1b9a59-caf9-de68-b843-c28b531ccc8e, HandlerErrorCode: InvalidRe                                       
    quest)                                                                                                       
✘ deploy service iss-3446 to environment beta: deploy service: stack playground-beta-iss-3446 did not complete successfully and exited with status UPDATE_ROLLBACK_COMPLETE
  1. Remove ` from end of secret in copilot manifest
  2. copilot svc deploy, still fails (relevant cfn from copilot svc package:
Secrets:
  - Name: TEST_SECRET_ONE
    ValueFrom: /copilot/playground/beta/secrets/TEST_SECRET_ONE

(so the CFN template generated by Copilot appears to be getting updated - the ` is gone)

Error message:

✘ deploy service iss-3446 to environment beta: deploy service: change set with name copilot-14edaf98-e209-4ce3-913c-45b8c4fbcd2d for stack playground-beta-iss-3446 has no changes: Resource handler returned message: "Invalid request provided: Create TaskDefinition: The Systems Manager parameter name specified for secret TEST_SECRET_ONE is invalid. The parameter name can be up to 2048 characters and include the following letters and symbols: a-zA-Z0-9_.-, (Service: AmazonECS; Status Code: 400; Error Code: ClientException; Request ID: 44b16b74-452a-4e55-ae73-10c7ae5f611e; Proxy: null)" (RequestToken: ba1b9a59-caf9-de68-b843-c28b531ccc8e, HandlerErrorCode: InvalidRequest)

Based on this last error message, it appears that CloudFormation doesn't generate a change set when ValueFrom changes in a Task Definition. If you make a different change in the manifest (like change count), CloudFormation will pick up the secret change as well and deploy a new Task Definition that works.

Next steps:

  • Make the change suggested by @efekarakus, should reduce the potential of getting into this situtation
  • Test changing ValueFrom outside of Copilot and follow up with relevant the internal teams to make sure changes to ValueFrom get detected

dannyrandall avatar Jul 05 '22 21:07 dannyrandall

Hi all and @efekarakus , here my secrets block on my manifest.yml. No any "`" at the end on any line.

secrets:
    APP_NAME: /copilot/${COPILOT_APPLICATION_NAME}/${COPILOT_ENVIRONMENT_NAME}/secrets/APP_NAME
    APP_ENV: /copilot/${COPILOT_APPLICATION_NAME}/${COPILOT_ENVIRONMENT_NAME}/secrets/APP_ENV
    APP_KEY: /copilot/${COPILOT_APPLICATION_NAME}/${COPILOT_ENVIRONMENT_NAME}/secrets/APP_KEY
    APP_DEBUG: /copilot/${COPILOT_APPLICATION_NAME}/${COPILOT_ENVIRONMENT_NAME}/secrets/APP_DEBUG
    APP_URL: /copilot/${COPILOT_APPLICATION_NAME}/${COPILOT_ENVIRONMENT_NAME}/secrets/APP_URL
    ASSET_URL: /copilot/${COPILOT_APPLICATION_NAME}/${COPILOT_ENVIRONMENT_NAME}/secrets/ASSET_URL
    LOG_CHANNEL: /copilot/${COPILOT_APPLICATION_NAME}/${COPILOT_ENVIRONMENT_NAME}/secrets/LOG_CHANNEL
    LOG_LEVEL: /copilot/${COPILOT_APPLICATION_NAME}/${COPILOT_ENVIRONMENT_NAME}/secrets/LOG_LEVEL
    DB_CONNECTION: /copilot/${COPILOT_APPLICATION_NAME}/${COPILOT_ENVIRONMENT_NAME}/secrets/DB_CONNECTION
    DB_HOST: /copilot/${COPILOT_APPLICATION_NAME}/${COPILOT_ENVIRONMENT_NAME}/secrets/DB_HOST
    DB_PORT: /copilot/${COPILOT_APPLICATION_NAME}/${COPILOT_ENVIRONMENT_NAME}/secrets/DB_PORT
    DB_DATABASE: /copilot/${COPILOT_APPLICATION_NAME}/${COPILOT_ENVIRONMENT_NAME}/secrets/DB_DATABASE
    DB_USERNAME: /copilot/${COPILOT_APPLICATION_NAME}/${COPILOT_ENVIRONMENT_NAME}/secrets/DB_USERNAME
    DB_PASSWORD: /copilot/${COPILOT_APPLICATION_NAME}/${COPILOT_ENVIRONMENT_NAME}/secrets/DB_PASSWORD
    BROADCAST_DRIVER: /copilot/${COPILOT_APPLICATION_NAME}/${COPILOT_ENVIRONMENT_NAME}/secrets/BROADCAST_DRIVER
    CACHE_DRIVER: /copilot/${COPILOT_APPLICATION_NAME}/${COPILOT_ENVIRONMENT_NAME}/secrets/CACHE_DRIVER
    QUEUE_CONNECTION: /copilot/${COPILOT_APPLICATION_NAME}/${COPILOT_ENVIRONMENT_NAME}/secrets/QUEUE_CONNECTION
    SESSION_DRIVER: /copilot/${COPILOT_APPLICATION_NAME}/${COPILOT_ENVIRONMENT_NAME}/secrets/SESSION_DRIVER
    SESSION_LIFETIME: /copilot/${COPILOT_APPLICATION_NAME}/${COPILOT_ENVIRONMENT_NAME}/secrets/SESSION_LIFETIME
    REDIS_HOST: /copilot/${COPILOT_APPLICATION_NAME}/${COPILOT_ENVIRONMENT_NAME}/secrets/REDIS_HOST
    REDIS_PASSWORD: /copilot/${COPILOT_APPLICATION_NAME}/${COPILOT_ENVIRONMENT_NAME}/secrets/REDIS_PASSWORD
    REDIS_PORT: /copilot/${COPILOT_APPLICATION_NAME}/${COPILOT_ENVIRONMENT_NAME}/secrets/REDIS_PORT
    REDIS_CLIENT: /copilot/${COPILOT_APPLICATION_NAME}/${COPILOT_ENVIRONMENT_NAME}/secrets/REDIS_CLIENT
    MAIL_MAILER: /copilot/${COPILOT_APPLICATION_NAME}/${COPILOT_ENVIRONMENT_NAME}/secrets/MAIL_MAILER
    MAIL_HOST: /copilot/${COPILOT_APPLICATION_NAME}/${COPILOT_ENVIRONMENT_NAME}/secrets/MAIL_HOST
    MAIL_PORT: /copilot/${COPILOT_APPLICATION_NAME}/${COPILOT_ENVIRONMENT_NAME}/secrets/MAIL_PORT
    MAIL_USERNAME: /copilot/${COPILOT_APPLICATION_NAME}/${COPILOT_ENVIRONMENT_NAME}/secrets/MAIL_USERNAME
    MAIL_PASSWORD: /copilot/${COPILOT_APPLICATION_NAME}/${COPILOT_ENVIRONMENT_NAME}/secrets/MAIL_PASSWORD
    MAIL_ENCRYPTION: /copilot/${COPILOT_APPLICATION_NAME}/${COPILOT_ENVIRONMENT_NAME}/secrets/MAIL_ENCRYPTION
    MAIL_FROM_ADDRESS: /copilot/${COPILOT_APPLICATION_NAME}/${COPILOT_ENVIRONMENT_NAME}/secrets/MAIL_FROM_ADDRESS
    MAIL_FROM_NAME: /copilot/${COPILOT_APPLICATION_NAME}/${COPILOT_ENVIRONMENT_NAME}/secrets/MAIL_FROM_NAME
    PUSHER_APP_ID: /copilot/${COPILOT_APPLICATION_NAME}/${COPILOT_ENVIRONMENT_NAME}/secrets/PUSHER_APP_ID
    PUSHER_APP_KEY: /copilot/${COPILOT_APPLICATION_NAME}/${COPILOT_ENVIRONMENT_NAME}/secrets/PUSHER_APP_KEY
    PUSHER_APP_SECRET: /copilot/${COPILOT_APPLICATION_NAME}/${COPILOT_ENVIRONMENT_NAME}/secrets/PUSHER_APP_SECRET
    PUSHER_APP_CLUSTER: /copilot/${COPILOT_APPLICATION_NAME}/${COPILOT_ENVIRONMENT_NAME}/secrets/PUSHER_APP_CLUSTER
    JWT_SECRET: /copilot/${COPILOT_APPLICATION_NAME}/${COPILOT_ENVIRONMENT_NAME}/secrets/JWT_SECRET
    AWS_BUCKET: /copilot/${COPILOT_APPLICATION_NAME}/${COPILOT_ENVIRONMENT_NAME}/secrets/AWS_BUCKET
    AWS_URL: /copilot/${COPILOT_APPLICATION_NAME}/${COPILOT_ENVIRONMENT_NAME}/secrets/AWS_URL
    AWS_ENDPOINT: /copilot/${COPILOT_APPLICATION_NAME}/${COPILOT_ENVIRONMENT_NAME}/secrets/AWS_ENDPOINT
    AWS_DEFAULT_REGION: /copilot/${COPILOT_APPLICATION_NAME}/${COPILOT_ENVIRONMENT_NAME}/secrets/AWS_DEFAULT_REGION
    FE_URL: /copilot/${COPILOT_APPLICATION_NAME}/${COPILOT_ENVIRONMENT_NAME}/secrets/FE_URL
    LOG_SLACK_WEBHOOK_URL: /copilot/${COPILOT_APPLICATION_NAME}/${COPILOT_ENVIRONMENT_NAME}/secrets/LOG_SLACK_WEBHOOK_URL
    GATSBY_API_URL: /copilot/${COPILOT_APPLICATION_NAME}/${COPILOT_ENVIRONMENT_NAME}/secrets/GATSBY_API_URL
    REACT_APP_API_URI: /copilot/${COPILOT_APPLICATION_NAME}/${COPILOT_ENVIRONMENT_NAME}/secrets/REACT_APP_API_URI
    FE_ADMIN_URL: /copilot/${COPILOT_APPLICATION_NAME}/${COPILOT_ENVIRONMENT_NAME}/secrets/FE_ADMIN_URL
    HAI2MAIL_API_PATH: /copilot/${COPILOT_APPLICATION_NAME}/${COPILOT_ENVIRONMENT_NAME}/secrets/HAI2MAIL_API_PATH
    HAI2MAIL_API_TOKEN: /copilot/${COPILOT_APPLICATION_NAME}/${COPILOT_ENVIRONMENT_NAME}/secrets/HAI2MAIL_API_TOKEN

mvn-bachhuynh-dn avatar Jul 06 '22 01:07 mvn-bachhuynh-dn

I wonder the Codebuild step have a problem, in the step, it will generate Cloudformation stack template and the Deploy step will use the generated CloudFormation stack template to create change set and excute. So please take a look again in the Codebuild phase.

When I use current CloudFormation stack template, try to add new secret manual, use it to create change set and excute...It works normally.

mvn-bachhuynh-dn avatar Jul 06 '22 01:07 mvn-bachhuynh-dn

Hi All, I resolved it!

Rootcause:

buildspec.yml was too old for copilot old version. and It does not support ${COPILOT_APPLICATION_NAME}/${COPILOT_ENVIRONMENT_NAME}/ in the SSM name

Resolve:

  • Delete the old pipeline and recreate the new one.
copilot pipeline delete
copilot pipeline init
copilot pipeline deploy

mvn-bachhuynh-dn avatar Jul 06 '22 12:07 mvn-bachhuynh-dn

Hey @mvn-bachhuynh-dn, glad you were able to resolve it! Do you know what version of Copilot you were using?

dannyrandall avatar Jul 06 '22 14:07 dannyrandall

@dannyrandall Hi Danny, The old version I used: v1.8 Now, I updated it to v1.17

mvn-bachhuynh-dn avatar Jul 07 '22 02:07 mvn-bachhuynh-dn