dependabot
Unable to get dependabot working with private CodeArtifact registry
Manually configured a test token as a dependabot secret and updated using
aws codeartifact get-authorization-token --domain DOMAIN --domain-owner OWNER --query authorizationToken --output text
dependabot.yml:
version: 2
registries:
npm-codeartifact:
type: npm-registry
url: https://domain-owner.d.codeartifact.us-east-1.amazonaws.com/npm/repositoryname/
username: aws
password: "${{secrets.TEST_TOKEN}}"
# Also used token: "${{secrets.TEST_TOKEN}}" and that gave the same results
updates:
- package-ecosystem: "npm"
allow:
- dependency-name: "@privatedomain/myprivatepackage"
directory: "/"
registries:
- npm-codeartifact
schedule:
interval: "daily"
Repo contains a packages.json which defines the package and a yarn.lock Package contains no further dependencies/sub dependencies (its very simple)
When dependabot runs, and no update needed, it succeeds (no errors generated) When an update is needed it sees the update and does the following:
updater | INFO <job_job_id> Starting job processing
updater | INFO <job_job_id> Starting update job for myorg/dependabot.test
updater | INFO <job_job_id> Checking if @privatedomain/myprivatepackage 1.0.24-beta needs updating
proxy | 2022/07/28 20:23:56 [028] GET https://domain-owner.d.codeartifact.us-east-1.amazonaws.com:443/npm/repositoryname/@privatedomain%2Fmyprivatepackage
proxy | 2022/07/28 20:23:56 [028] * authenticating npm registry request (host: domain-owner.d.codeartifact.us-east-1.amazonaws.com, basic auth)
proxy | 2022/07/28 20:23:57 [028] 200 https://domain-owner.d.codeartifact.us-east-1.amazonaws.com:443/npm/repositoryname/@privatedomain%2Fmyprivatepackage
proxy | 2022/07/28 20:23:57 [030] GET https://domain-owner.d.codeartifact.us-east-1.amazonaws.com:443/npm/repositoryname/@privatedomain%2Fmyprivatepackage/1.0.26
proxy | 2022/07/28 20:23:57 [030] * authenticating npm registry request (host: domain-owner.d.codeartifact.us-east-1.amazonaws.com, basic auth)
proxy | 2022/07/28 20:23:57 [030] 200 https://domain-owner.d.codeartifact.us-east-1.amazonaws.com:443/npm/repositoryname/@privatedomain%2Fmyprivatepackage/1.0.26
updater | INFO <job_job_id> Latest version is 1.0.26
proxy | 2022/07/28 20:24:00 [032] GET https://registry.yarnpkg.com:443/@privatedomain%2fmyprivatepackage
proxy | 2022/07/28 20:24:00 [032] 404 https://registry.yarnpkg.com:443/@privatedomain%2fmyprivatepackage
updater | INFO <job_job_id> Requirements to unlock own
updater | INFO <job_job_id> Requirements update strategy bump_versions
updater | INFO <job_job_id> Updating @privatedomain/myprivatepackage from 1.0.24-beta to 1.0.26
proxy | 2022/07/28 20:24:02 [034] GET https://registry.yarnpkg.com:443/@privatedomain%2fmyprivatepackage
proxy | 2022/07/28 20:24:02 [034] 404 https://registry.yarnpkg.com:443/@privatedomain%2fmyprivatepackage
updater | INFO <job_job_id> Handled error whilst updating @privatedomain/myprivatepackage: private_source_authentication_failure {:source=>"domain-owner.d.codeartifact.us-east-1.amazonaws.com:443/npm/repositoryname"}
updater | INFO <job_job_id> Finished job processing
updater | INFO Results:
updater | Dependabot encountered '1' error(s) during execution, please check the logs for more details.
updater | time="2022-07-28T20:24:04Z" level=info msg="task complete" container_id=job-job_id-updater exit_code=0 job_id=job_id step=updater
If i try without the /npm/repositoryname/ in the url which supposedly is the correct way - (url: https://domain-owner.d.codeartifact.us-east-1.amazonaws.com) It will try https://domain-owner.d.codeartifact.us-east-1.amazonaws.com:443/@privatedomain%2Fmyprivatepackage instead and 403 "missing Authentication Token" It will not even try the url it found the updated version on, just directly to that url only and fails:
updater | INFO <job_job_id> Updating @privatedomain/myprivatepackage from 1.0.24-beta to 1.0.26
proxy | 2022/07/28 16:04:01 [034] GET https://domain-owner.d.codeartifact.us-east-1.amazonaws.com:443/@privatedomain%2fmyprivatepackage
proxy | 2022/07/28 16:04:01 [034] * authenticating npm registry request (host: domain-owner.d.codeartifact.us-east-1.amazonaws.com, token auth)
proxy | 2022/07/28 16:04:01 [034] 403 https://domain-owner.d.codeartifact.us-east-1.amazonaws.com:443/@privatedomain%2fmyprivatepackage
updater | ERROR <job_job_id> Error processing @privatedomain/myprivatepackage (Dependabot::SharedHelpers::HelperSubprocessFailed)
updater | ERROR <job_job_id> https://domain-owner.d.codeartifact.us-east-1.amazonaws.com/@privatedomain%2fmyprivatepackage: Missing Authentication Token
Is there something not right? This is not an authentication issue as the token is fresh and clearly seeing the updated version with a 200
Expected results?
- dependabot will see the updated version, and open a PR automatically
Manually configured a test token as a dependabot secret and updated using
aws codeartifact get-authorization-token --domain DOMAIN --domain-owner OWNER --query authorizationToken --output textdependabot.yml:
version: 2 registries: npm-codeartifact: type: npm-registry url: https://domain-owner.d.codeartifact.us-east-1.amazonaws.com/npm/repositoryname/ username: aws password: "${{secrets.TEST_TOKEN}}" # Also used token: "${{secrets.TEST_TOKEN}}" and that gave the same results updates: - package-ecosystem: "npm" allow: - dependency-name: "@privatedomain/myprivatepackage" directory: "/" registries: - npm-codeartifact schedule: interval: "daily"Repo contains a packages.json which defines the package and a yarn.lock Package contains no further dependencies/sub dependencies (its very simple)
When dependabot runs, and no update needed, it succeeds (no errors generated) When an update is needed it sees the update and does the following:
updater | INFO <job_job_id> Starting job processing updater | INFO <job_job_id> Starting update job for myorg/dependabot.test updater | INFO <job_job_id> Checking if @privatedomain/myprivatepackage 1.0.24-beta needs updating proxy | 2022/07/28 20:23:56 [028] GET https://domain-owner.d.codeartifact.us-east-1.amazonaws.com:443/npm/repositoryname/@privatedomain%2Fmyprivatepackage proxy | 2022/07/28 20:23:56 [028] * authenticating npm registry request (host: domain-owner.d.codeartifact.us-east-1.amazonaws.com, basic auth) proxy | 2022/07/28 20:23:57 [028] 200 https://domain-owner.d.codeartifact.us-east-1.amazonaws.com:443/npm/repositoryname/@privatedomain%2Fmyprivatepackage proxy | 2022/07/28 20:23:57 [030] GET https://domain-owner.d.codeartifact.us-east-1.amazonaws.com:443/npm/repositoryname/@privatedomain%2Fmyprivatepackage/1.0.26 proxy | 2022/07/28 20:23:57 [030] * authenticating npm registry request (host: domain-owner.d.codeartifact.us-east-1.amazonaws.com, basic auth) proxy | 2022/07/28 20:23:57 [030] 200 https://domain-owner.d.codeartifact.us-east-1.amazonaws.com:443/npm/repositoryname/@privatedomain%2Fmyprivatepackage/1.0.26 updater | INFO <job_job_id> Latest version is 1.0.26 proxy | 2022/07/28 20:24:00 [032] GET https://registry.yarnpkg.com:443/@privatedomain%2fmyprivatepackage proxy | 2022/07/28 20:24:00 [032] 404 https://registry.yarnpkg.com:443/@privatedomain%2fmyprivatepackage updater | INFO <job_job_id> Requirements to unlock own updater | INFO <job_job_id> Requirements update strategy bump_versions updater | INFO <job_job_id> Updating @privatedomain/myprivatepackage from 1.0.24-beta to 1.0.26 proxy | 2022/07/28 20:24:02 [034] GET https://registry.yarnpkg.com:443/@privatedomain%2fmyprivatepackage proxy | 2022/07/28 20:24:02 [034] 404 https://registry.yarnpkg.com:443/@privatedomain%2fmyprivatepackage updater | INFO <job_job_id> Handled error whilst updating @privatedomain/myprivatepackage: private_source_authentication_failure {:source=>"domain-owner.d.codeartifact.us-east-1.amazonaws.com:443/npm/repositoryname"} updater | INFO <job_job_id> Finished job processing updater | INFO Results: updater | Dependabot encountered '1' error(s) during execution, please check the logs for more details. updater | time="2022-07-28T20:24:04Z" level=info msg="task complete" container_id=job-job_id-updater exit_code=0 job_id=job_id step=updaterIf i try without the /npm/repositoryname/ in the url which supposedly is the correct way - (url: https://domain-owner.d.codeartifact.us-east-1.amazonaws.com) It will try https://domain-owner.d.codeartifact.us-east-1.amazonaws.com:443/@privatedomain%2Fmyprivatepackage instead and 403 "missing Authentication Token" It will not even try the url it found the updated version on, just directly to that url only and fails:
updater | INFO <job_job_id> Updating @privatedomain/myprivatepackage from 1.0.24-beta to 1.0.26 proxy | 2022/07/28 16:04:01 [034] GET https://domain-owner.d.codeartifact.us-east-1.amazonaws.com:443/@privatedomain%2fmyprivatepackage proxy | 2022/07/28 16:04:01 [034] * authenticating npm registry request (host: domain-owner.d.codeartifact.us-east-1.amazonaws.com, token auth) proxy | 2022/07/28 16:04:01 [034] 403 https://domain-owner.d.codeartifact.us-east-1.amazonaws.com:443/@privatedomain%2fmyprivatepackage updater | ERROR <job_job_id> Error processing @privatedomain/myprivatepackage (Dependabot::SharedHelpers::HelperSubprocessFailed) updater | ERROR <job_job_id> https://domain-owner.d.codeartifact.us-east-1.amazonaws.com/@privatedomain%2fmyprivatepackage: Missing Authentication TokenIs there something not right? This is not an authentication issue as the token is fresh and clearly seeing the updated version with a 200
Expected results?
- dependabot will see the updated version, and open a PR automatically
See X
Doesn't the token is valid up for 12 hours max? (+ AFAIK if taken from an assumed role is limited to role token duration) https://docs.aws.amazon.com/codeartifact/latest/ug/tokens-authentication.html
Is there a fix for this? I have the same issue
I have the packages on codeartifact private registry.
I need to generate the CODEARTIFACT_TOKEN from the CLI as I need to run aws codeartifact get-authorization-token
This means that I need to generate the token, add it as a secret manually, the next day I need to do it all over again.
If dependabot.yaml worked like a Github Action we could use the run command.
Any workaround for this?
I believe dependabot should support AWS Access/Secret key (even better assume role) to generate short time living token for accessing private pypi in the background.
We need something. Token generation is possible with Github Actions, so it shouldnt be hard to do the same with dependabot
Hey everyone, thanks for the increased traction on this issue however i'd like to repeat one very important detail here:
This is not an authentication issue as the token is fresh and clearly seeing the updated version with a 200
Feel free to submit your own github issue for all of these token generation issues/problems. Keeping the token in sync is possible by running a cron/fargate/lambda which refreshes the token, encrypts and pushes into github via API call
Just rechecked and this is still an issue.
To recap:
- Token is good - this is not an authentication issue.
- dependabot sees the correct package and notes it needs to update
- upon updating, it uses a completely DIFFERENT path and the dependabot operation fails.
Package lives at:
https://domain-owner.d.codeartifact.us-east-1.amazonaws.com:443/npm/repositoryname/@privatedomain%2Fmyprivatepackage/1.0.26
Dependabot attempts the update at: (registry configured as: https://domain-owner.d.codeartifact.us-east-1.amazonaws.com/npm/repositoryname/)
https://registry.yarnpkg.com:443/@privatedomain%2fmyprivatepackage
Dependabot attempts the update at: (registry configured as: https://domain-owner.d.codeartifact.us-east-1.amazonaws.com/)
https://domain-owner.d.codeartifact.us-east-1.amazonaws.com:443/@privatedomain%2fmyprivatepackage
We are facing a similar issue - but not with CodeArtifact but with the Github Packages Registry.
We do have two repos which are depending on the same package. One repo is doing just fine and is finding the package from our private Github Packages Registry. The other repo is behaving exactly as you have described.
Both are using the same organization secret.
Not working repo:
updater | INFO <job_505559082> Checking if @gp-9000/logging 1.1.7 needs updating
proxy | 2022/11/09 07:00:37 [270] GET https://npm.pkg.github.com:443/@gp-9000%2Flogging
proxy | 2022/11/09 07:00:37 [270] * authenticating npm registry request (host: npm.pkg.github.com, token auth)
proxy | 2022/11/09 07:00:37 [270] 200 https://npm.pkg.github.com:443/@gp-9000%2Flogging
proxy | 2022/11/09 07:00:37 [272] GET https://npm.pkg.github.com:443/@gp-9000%2Flogging/1.1.10
proxy | 2022/11/09 07:00:37 [272] * authenticating npm registry request (host: npm.pkg.github.com, token auth)
proxy | 2022/11/09 07:00:37 [272] 405 https://npm.pkg.github.com:443/@gp-9000%2Flogging/1.1.10
updater | INFO <job_505559082> Latest version is 1.1.10
proxy | 2022/11/09 07:00:41 [274] GET https://registry.npmjs.org:443/@gp-9000%2flogging
proxy | 2022/11/09 07:00:41 [274] 404 https://registry.npmjs.org:443/@gp-9000%2flogging
proxy | 2022/11/09 07:00:41 [276] GET https://registry.npmjs.org:443/@gp-9000%2flogging
proxy | 2022/11/09 07:00:41 [276] 404 https://registry.npmjs.org:443/@gp-9000%2flogging
updater | INFO <job_505559082> Requirements to unlock own
updater | INFO <job_505559082> Requirements update strategy bump_versions_if_necessary
updater | INFO <job_505559082> Updating @gp-9000/logging from 1.1.7 to 1.1.10
proxy | 2022/11/09 07:00:45 [278] GET https://registry.npmjs.org:443/@gp-9000%2flogging
proxy | 2022/11/09 07:00:46 [278] 404 https://registry.npmjs.org:443/@gp-9000%2flogging
proxy | 2022/11/09 07:00:46 [280] GET https://registry.npmjs.org:443/@gp-9000%2flogging
proxy | 2022/11/09 07:00:46 [280] 404 https://registry.npmjs.org:443/@gp-9000%2flogging
updater | INFO <job_505559082> Handled error whilst updating @gp-9000/logging: private_source_authentication_failure {:source=>"npm.pkg.github.com"}
working repo:
updater | INFO <job_505559583> Checking if @gp-9000/logging 1.1.9 needs updating
proxy | 2022/11/09 07:00:47 [106] GET https://npm.pkg.github.com:443/@gp-9000%2Flogging
proxy | 2022/11/09 07:00:47 [106] * authenticating npm registry request (host: npm.pkg.github.com, token auth)
proxy | 2022/11/09 07:00:48 [106] 200 https://npm.pkg.github.com:443/@gp-9000%2Flogging
proxy | 2022/11/09 07:00:48 [108] GET https://npm.pkg.github.com:443/@gp-9000%2Flogging/1.1.10
proxy | 2022/11/09 07:00:48 [108] * authenticating npm registry request (host: npm.pkg.github.com, token auth)
proxy | 2022/11/09 07:00:48 [108] 405 https://npm.pkg.github.com:443/@gp-9000%2Flogging/1.1.10
updater | INFO <job_505559583> Latest version is 1.1.10
proxy | 2022/11/09 07:00:52 [110] GET https://npm.pkg.github.com:443/@gp-9000%2flogging
proxy | 2022/11/09 07:00:52 [110] * authenticating npm registry request (host: npm.pkg.github.com, token auth)
proxy | 2022/11/09 07:00:53 [110] 200 https://npm.pkg.github.com:443/@gp-9000%2flogging
updater | INFO <job_505559583> Requirements to unlock own
updater | INFO <job_505559583> Requirements update strategy bump_versions_if_necessary
updater | INFO <job_505559583> Updating @gp-9000/logging from 1.1.9 to 1.1.10
proxy | 2022/11/09 07:00:57 [112] GET https://npm.pkg.github.com:443/@gp-9000%2flogging
proxy | 2022/11/09 07:00:57 [112] * authenticating npm registry request (host: npm.pkg.github.com, token auth)
proxy | 2022/11/09 07:00:58 [112] 200 https://npm.pkg.github.com:443/@gp-9000%2flogging
proxy | 2022/11/09 07:00:59 [114] GET https://api.github.com:443/repos/GP-9000/webend/commits?per_page=100
proxy | 2022/11/09 07:00:59 [114] * authenticating github api request
proxy | 2022/11/09 07:00:59 [114] 200 https://api.github.com:443/repos/GP-9000/webend/commits?per_page=100
proxy | 2022/11/09 07:00:59 [116] GET https://npm.pkg.github.com:443/@gp-9000%2Flogging/latest
proxy | 2022/11/09 07:00:59 [116] * authenticating npm registry request (host: npm.pkg.github.com, token auth)
proxy | 2022/11/09 07:00:59 [116] 405 https://npm.pkg.github.com:443/@gp-9000%2Flogging/latest
proxy | 2022/11/09 07:00:59 [118] GET https://npm.pkg.github.com:443/@gp-9000%2Flogging
proxy | 2022/11/09 07:00:59 [118] * authenticating npm registry request (host: npm.pkg.github.com, token auth)
proxy | 2022/11/09 07:01:00 [118] 200 https://npm.pkg.github.com:443/@gp-9000%2Flogging
proxy | 2022/11/09 07:01:00 [120] GET https://api.github.com:443/repos/GP-9000/logging/releases?per_page=100
proxy | 2022/11/09 07:01:00 [120] * authenticating github api request
proxy | 2022/11/09 07:01:00 [120] 200 https://api.github.com:443/repos/GP-9000/logging/releases?per_page=100
proxy | 2022/11/09 07:01:00 [122] GET https://api.github.com:443/repos/GP-9000/logging/contents/
proxy | 2022/11/09 07:01:00 [122] * authenticating github api request
proxy | 2022/11/09 07:01:00 [122] 200 https://api.github.com:443/repos/GP-9000/logging/contents/
proxy | 2022/11/09 07:01:00 [124] GET https://github.com:443/GP-9000/logging.git/info/refs?service=git-upload-pack
proxy | 2022/11/09 07:01:00 [124] * authenticating git server request (host: github.com)
proxy | 2022/11/09 07:01:00 [124] 200 https://github.com:443/GP-9000/logging.git/info/refs?service=git-upload-pack
proxy | 2022/11/09 07:01:00 [126] GET https://api.github.com:443/repos/GP-9000/logging/contents/?ref=v1.1.10
proxy | 2022/11/09 07:01:00 [126] * authenticating github api request
proxy | 2022/11/09 07:01:00 [126] 200 https://api.github.com:443/repos/GP-9000/logging/contents/?ref=v1.1.10
proxy | 2022/11/09 07:01:00 [128] GET https://github.com:443/GP-9000/logging.git/info/refs?service=git-upload-pack
proxy | 2022/11/09 07:01:00 [128] * authenticating git server request (host: github.com)
proxy | 2022/11/09 07:01:01 [128] 200 https://github.com:443/GP-9000/logging.git/info/refs?service=git-upload-pack
proxy | 2022/11/09 07:01:01 [130] GET https://api.github.com:443/repos/GP-9000/logging/commits?sha=v1.1.9
proxy | 2022/11/09 07:01:01 [130] * authenticating github api request
proxy | 2022/11/09 07:01:01 [130] 200 https://api.github.com:443/repos/GP-9000/logging/commits?sha=v1.1.9
proxy | 2022/11/09 07:01:01 [132] GET https://api.github.com:443/repos/GP-9000/logging/commits?sha=v1.1.10
proxy | 2022/11/09 07:01:01 [132] * authenticating github api request
proxy | 2022/11/09 07:01:01 [132] 200 https://api.github.com:443/repos/GP-9000/logging/commits?sha=v1.1.10
proxy | 2022/11/09 07:01:01 [134] GET https://api.github.com:443/repos/GP-9000/logging/commits?sha=v1.1.9
proxy | 2022/11/09 07:01:01 [134] * authenticating github api request
proxy | 2022/11/09 07:01:02 [134] 200 https://api.github.com:443/repos/GP-9000/logging/commits?sha=v1.1.9
proxy | 2022/11/09 07:01:02 [136] GET https://api.github.com:443/repos/GP-9000/logging/commits?sha=v1.1.10
proxy | 2022/11/09 07:01:02 [136] * authenticating github api request
proxy | 2022/11/09 07:01:02 [136] 200 https://api.github.com:443/repos/GP-9000/logging/commits?sha=v1.1.10
proxy | 2022/11/09 07:01:02 [138] GET https://api.github.com:443/repos/GP-9000/logging/commits?sha=v1.1.9
proxy | 2022/11/09 07:01:02 [138] * authenticating github api request
proxy | 2022/11/09 07:01:02 [138] 200 https://api.github.com:443/repos/GP-9000/logging/commits?sha=v1.1.9
proxy | 2022/11/09 07:01:02 [140] GET https://api.github.com:443/repos/GP-9000/logging/commits?sha=v1.1.10
proxy | 2022/11/09 07:01:02 [140] * authenticating github api request
proxy | 2022/11/09 07:01:02 [140] 200 https://api.github.com:443/repos/GP-9000/logging/commits?sha=v1.1.10
updater | INFO <job_505559583> Submitting @gp-9000/logging pull request for creation
i noticed if your package-lock.json has the 443 in the resolved url like https://domain-owner.d.codeartifact.us-east-1.amazonaws.com:443/npm/repositoryname/@privatedomain%2Fmyprivatepackage then if i set the dependabot registry url to https://domain-owner.d.codeartifact.us-east-1.amazonaws.com:443/npm/repositoryname/ it worked. tried this after reading https://github.com/dependabot/dependabot-core/issues/4669#issuecomment-1024693173
For anyone stumbling on this issue, my final solution:
- Add the
replaces-base: truein dependabot.yml - Create .npmrc in the app's directory with the following:
@privateregistry:registry=https://domain-owner.d.codeartifact.us-east-1.amazonaws.com/npm/repositoryname/
Thanks for circling back to let us know. What you did is generally what we recommend.
I am unclear though--did it turn out the :443 wasn't necessary if you configured the .npmrc as above?
One question for everyone hitting this:
Is there a mismatch in the URLs between the registry credentials that you give Dependabot and what you're putting in your .npmrc file?
More specifically, do you only see this if the 443 port is present in one and absent in the other?
If so, it may mean our security proxy isn't matching up the URLs when applying the credentials. Although in that case we should probably just strip out the 443 port... But before I file an internal bug and start digging into this, can someone who's hitting this confirm whether or not that's the behavior you're encountering?
@jeffwidman removing the :443 in the lock got it to work initially, but the generated PR (for the update) wanted to put it right back in.. I tried adding :443 to the url in dependabot.yml, that didnt work.. It was adding the configuration in an .npmrc that was the final solution for me. (.yml config no :443) Still has the :443 in the lockfile but now dependabot is ok with it. Initially, there was no .npmrc at all and i think that was the source of my issue.
For the .npmrc, i didnt need to add any creds, just the reference/pointer. I'm using a token in the dependabot.yml for the creds thats passed in as a dependabot secret from github