docker
docker compose up from ecs context, run from command line gives a "unexpected status code [manifests latest]: 403 Forbidden"
Description
Trying to upload a multi-container setup to was ecs. I previously could successfully deploy using up and end using down. But recently without changing anything in the docker compose files I am constantly getting this error unexpected status code [manifests latest]: 403 Forbidden.
Describe the results you received: unexpected status code [manifests latest]: 403 Forbidden.
Describe the results you expected: Deployed container with 26 steps(CloudFormation, tasks, load balancer etc)
Additional information you deem important (e.g. issue happens only occasionally): quite recent, after updating docker desktop to v3.1
Output of docker version:
Client: Docker Engine - Community
Cloud integration: 1.0.7
Version: 20.10.2
API version: 1.41
Go version: go1.13.15
Git commit: 2291f61
Built: Mon Dec 28 16:12:42 2020
OS/Arch: darwin/amd64
Context: default
Experimental: true
Server: Docker Engine - Community
Engine:
Version: 20.10.2
API version: 1.41 (minimum version 1.12)
Go version: go1.13.15
Git commit: 8891c58
Built: Mon Dec 28 16:15:28 2020
OS/Arch: linux/amd64
Experimental: true
containerd:
Version: 1.4.3
GitCommit: 269548fa27e0089a8b8278fc4fc781d7f65a939b
runc:
Version: 1.0.0-rc92
GitCommit: ff819c7e9184c13b7c2607fe6c30ae19403a7aff
docker-init:
Version: 0.19.0
GitCommit: de40ad0
Output of docker context show:
You can also run docker context inspect context-name to give us more details but don't forget to remove sensitive content.
(paste your output here)
Output of docker info:
(paste your output here)
Additional environment details (AWS ECS, Azure ACI, local, etc.): ecs context, macOS Big Sur.
please run command with debug log level to help diagnose this error:
docker --debug compose up
Same here, it was working previously, now it does not. I simplified the compose file to only mysql:
❯ docker --debug compose up --file test-stack.yaml
DEBU[0000] deploying on AWS with region="eu-west-3"
DEBU[0000] resolving host=registry-1.docker.io
DEBU[0000] do request host=registry-1.docker.io request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=containerd/1.4.3+unknown request.method=HEAD url="https://registry-1.docker.io/v2/library/mysql/manifests/5.7.31"
DEBU[0000] fetch response received host=registry-1.docker.io response.header.content-length=156 response.header.content-type=application/json response.header.date="Fri, 22 Jan 2021 15:07:59 GMT" response.header.docker-distribution-api-version=registry/2.0 response.header.strict-transport-security="max-age=31536000" response.header.www-authenticate="Bearer realm=\"https://auth.docker.io/token\",service=\"registry.docker.io\",scope=\"repository:library/mysql:pull\"" response.status="401 Unauthorized" url="https://registry-1.docker.io/v2/library/mysql/manifests/5.7.31"
DEBU[0000] Unauthorized header="Bearer realm=\"https://auth.docker.io/token\",service=\"registry.docker.io\",scope=\"repository:library/mysql:pull\"" host=registry-1.docker.io
DEBU[0001] do request host=registry-1.docker.io request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=containerd/1.4.3+unknown request.method=HEAD url="https://registry-1.docker.io/v2/library/mysql/manifests/5.7.31"
DEBU[0002] fetch response received host=registry-1.docker.io response.header.content-length=320 response.header.content-type=application/vnd.docker.distribution.manifest.list.v2+json response.header.date="Fri, 22 Jan 2021 15:08:01 GMT" response.header.docker-content-digest="sha256:b3dc8d10307ab7b9ca1a7981b1601a67e176408be618fc4216d137be37dae10b" response.header.docker-distribution-api-version=registry/2.0 response.header.etag="\"sha256:b3dc8d10307ab7b9ca1a7981b1601a67e176408be618fc4216d137be37dae10b\"" response.header.strict-transport-security="max-age=31536000" response.status="200 OK" url="https://registry-1.docker.io/v2/library/mysql/manifests/5.7.31"
DEBU[0002] resolved desc.digest="sha256:b3dc8d10307ab7b9ca1a7981b1601a67e176408be618fc4216d137be37dae10b" host=registry-1.docker.io
mysql:5.7.31 resolved to docker.io/library/mysql:5.7.31@sha256:b3dc8d10307ab7b9ca1a7981b1601a67e176408be618fc4216d137be37dae10b
DEBU[0002] Retrieve default VPC
DEBU[0002] Retrieve SubNets
UnauthorizedOperation: You are not authorized to perform this operation.
status code: 403, request id: 32e23217-41ca-495d-b9c9-09832816cd66
If you put these permissions at the user, it will work:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "docker",
"Effect": "Allow",
"Action": [
"cloudformation:*",
"servicediscovery:*",
"elasticloadbalancing:*",
"ecs:ListAccountSettings",
"ecs:DescribeClusters",
"ecs:DescribeServices",
"ecs:ListTasks",
"ecs:DescribeTasks",
"ecs:RegisterTaskDefinition",
"ecs:DeregisterTaskDefinition",
"ecs:CreateCluster",
"ecs:DeleteCluster",
"ecs:CreateService",
"ecs:DeleteService",
"ec2:DescribeVpcs",
"ec2:DescribeSubnets",
"ec2:DescribeSecurityGroups",
"ec2:Describe*",
"ec2:CreateSecurityGroup",
"ec2:DeleteSecurityGroup",
"ec2:CreateTags",
"ec2:DeleteTags",
"ec2:AuthorizeSecurityGroupIngress",
"ec2:RevokeSecurityGroupIngress",
"iam:CreateRole",
"iam:DeleteRole",
"iam:PassRole",
"iam:AttachRolePolicy",
"iam:DetachRolePolicy",
"logs:DescribeLogGroups",
"logs:FilterLogEvents",
"logs:CreateLogGroup",
"logs:DeleteLogGroup",
"route53:ListHostedZonesByName",
"route53:GetHealthCheck",
"route53:GetHostedZone",
"route53:CreateHostedZone",
"route53:DeleteHostedZone"
],
"Resource": "*"
}
]
}
I would like to know which ec2:Describe permissions are really missing though (one of the permission is ec2:Describe*, which includes all the Describe).
Same problem here, updating to the latest version causes the unexpected status code [manifests latest]: 401 Unauthorized error. Worked ok on the previous version.
DEBU[0000] deploying on AWS with region="us-east-1"
DEBU[0001] resolving host=669336776822.dkr.ecr.ap-southeast-1.amazonaws.com
DEBU[0001] do request host=669336776822.dkr.ecr.ap-southeast-1.amazonaws.com request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=containerd/1.4.3+unknown request.method=HEAD url="https://669336776822.dkr.ecr.ap-southeast-1.amazonaws.com/v2/grantit-server-v2/manifests/latest"
DEBU[0001] fetch response received host=669336776822.dkr.ecr.ap-southeast-1.amazonaws.com response.header.content-length=15 response.header.content-type="text/plain; charset=utf-8" response.header.date="Tue, 26 Jan 2021 15:15:44 GMT" response.header.docker-distribution-api-version=registry/2.0 response.header.www-authenticate="Basic realm=\"https://669336776822.dkr.ecr.ap-southeast-1.amazonaws.com/\",service=\"ecr.amazonaws.com\"" response.status="401 Unauthorized" url="https://669336776822.dkr.ecr.ap-southeast-1.amazonaws.com/v2/grantit-server-v2/manifests/latest"
DEBU[0001] Unauthorized header="Basic realm=\"https://669336776822.dkr.ecr.ap-southeast-1.amazonaws.com/\",service=\"ecr.amazonaws.com\"" host=669336776822.dkr.ecr.ap-southeast-1.amazonaws.com
unexpected status code [manifests latest]: 401 Unauthorized
Using an administrator account
I was also seeing this 403 error message unexpectedly and it turned out my authorization token to my private ECR registry had expired. Re-authenticating with aws ecr get-login-password | docker login --username AWS --password-stdin <user-id>.dkr.ecr.eu-central-1.amazonaws.com fixed it.
Howdy 👋 .
I've just ran into this problem, and @tpatja's solution fixed my issue (thanks!).
Seeing that as of now there are 16 other 👍 on that comment, I'm guessing is one of the most common sources of this issue. Would it be possible for docker to validate/suggest checking the ecr login when this error occurs?
Thanks y'all, and keep up the awesome work -- this tooling is definitely the simplest and easiest to get going on ECS.
I was also seeing this 403 error message unexpectedly and it turned out my authorization token to my private ECR registry had expired. Re-authenticating with
aws ecr get-login-password | docker login --username AWS --password-stdin <user-id>.dkr.ecr.eu-central-1.amazonaws.comfixed it.
Saw this error as well and for me it was the ecr login expired.
unfortunately the aws ecr get-login-password didn't work for me. I am running from docker context myecscontext and I still get the same [manifests latest]: 403 Forbidden error
If you used an existing AWS profile during docker context creation, ensure that the aws ecr get-login-password command, @tpatja has shared, uses the same AWS profile:
aws ecr --profile <profile> get-login-password | docker login --username AWS --password-stdin <user-id>.dkr.ecr.eu-central-1.amazonaws.com
If you put these permissions at the user, it will work:
{ "Version": "2012-10-17", "Statement": [ { "Sid": "docker", "Effect": "Allow", "Action": [ "cloudformation:*", "servicediscovery:*", "elasticloadbalancing:*", "ecs:ListAccountSettings", "ecs:DescribeClusters", "ecs:DescribeServices", "ecs:ListTasks", "ecs:DescribeTasks", "ecs:RegisterTaskDefinition", "ecs:DeregisterTaskDefinition", "ecs:CreateCluster", "ecs:DeleteCluster", "ecs:CreateService", "ecs:DeleteService", "ec2:DescribeVpcs", "ec2:DescribeSubnets", "ec2:DescribeSecurityGroups", "ec2:Describe*", "ec2:CreateSecurityGroup", "ec2:DeleteSecurityGroup", "ec2:CreateTags", "ec2:DeleteTags", "ec2:AuthorizeSecurityGroupIngress", "ec2:RevokeSecurityGroupIngress", "iam:CreateRole", "iam:DeleteRole", "iam:PassRole", "iam:AttachRolePolicy", "iam:DetachRolePolicy", "logs:DescribeLogGroups", "logs:FilterLogEvents", "logs:CreateLogGroup", "logs:DeleteLogGroup", "route53:ListHostedZonesByName", "route53:GetHealthCheck", "route53:GetHostedZone", "route53:CreateHostedZone", "route53:DeleteHostedZone" ], "Resource": "*" } ] }I would like to know which
ec2:Describepermissions are really missing though (one of the permission isec2:Describe*, which includes all the Describe).
It worked like a charm for me!
@em-nani same issue on my end.. ecr get-login-password fails because I'm in the ECS/AWS context. I reverted back to the default context first so I can perform ecr get-login-password and switch back to ECS/AWS context. Need to make sure that the login completes successfully.
The error 403 Forbidden should go away on the next compose up :)
