Add option to use gzip with S3 client for GCS compatibility

[goodspark]

I tried to use GCS' S3 interoperability to use GCS as a remote registry cache for Docker builds. The idea here is that with minimal changes, you should be able to use GCS in place of S3.

This failed with 403s (even though I tested the GCS HMAC key with Python/boto3 successfully). After digging into the code more and using the AWS Go SDK's logging, it looks like Google always adds gzip(gfe) to the Accept-Encoding header when computing the canonical request hash, whereas the the S3 client disables gzip and only uses identity. So the computed hashes for the canonical requests are different. It looks like this was intentional in https://github.com/aws/aws-sdk-go-v2/pull/748, but a way to add it back was provided in the SetHeaderValue middleware.

Here's the output from the S3 client when sending the request:

---[ CANONICAL STRING  ]-----------------------------
PUT
/manifests/MY_IMAGE_NAME
x-id=PutObject
accept-encoding:identity
amz-sdk-invocation-id:35119a7f-4e72-4f6d-a924-5b73363f12d6
amz-sdk-request:attempt=1; max=3
content-length:183
content-type:application/octet-stream
host:MY_BUCKET_NAME.storage.googleapis.com
x-amz-content-sha256:UNSIGNED-PAYLOAD
x-amz-date:20230327T070514Z

accept-encoding;amz-sdk-invocation-id;amz-sdk-request;content-length;content-type;host;x-amz-content-sha256;x-amz-date
UNSIGNED-PAYLOAD

Here's an example response from GCS (with some newlines for easier reading):

<?xml version='1.0' encoding='UTF-8'?>
<Error>
<Code>SignatureDoesNotMatch</Code>
<Message>The request signature we calculated does not match the signature you provided. Check your Google secret key and signing method.</Message>
<StringToSign>AWS4-HMAC-SHA256
20230327T070514Z
20230327/auto/s3/aws4_request
63c93f54527f5afb265efb802ce4c44ac731d4c3ea9795d919dd1b506f2286a8</StringToSign>
<CanonicalRequest>PUT
/manifests/MY_IMAGE_NAME
x-id=PutObject
accept-encoding:identity,gzip(gfe)
amz-sdk-invocation-id:35119a7f-4e72-4f6d-a924-5b73363f12d6
amz-sdk-request:attempt=1; max=3
content-length:183
content-type:application/octet-stream
host:MY_BUCKET_NAME.storage.googleapis.com
x-amz-content-sha256:UNSIGNED-PAYLOAD
x-amz-date:20230327T070514Z

accept-encoding;amz-sdk-invocation-id;amz-sdk-request;content-length;content-type;host;x-amz-content-sha256;x-amz-date
UNSIGNED-PAYLOAD</CanonicalRequest>
</Error>

As a test, I commented out this line and returned nil. This made signature checks work. The request still failed for another reason (context deadlines with the solver), but that'll have to be filed in a separate issue.

https://github.com/moby/buildkit/blob/4b4a41f3f15359b18c4e5d3cc5c7a52ebc555894/vendor/github.com/aws/aws-sdk-go-v2/service/s3/api_client.go#L591-L593

I don't know if this will break compatibility with S3. So maybe the best course of action here is to add an option to the S3 client in Buildkit that conditionally adds gzip to the Accept-Encoding header?

Alternatively, add support for GCS. But that's a bigger task.

[zhilyaev]

The same

[sorenhansendk]

I have almost the same problem, when I try to upload to Google Cloud's S3 interface.

I have tried to add CHECKSUM_CALCULATION environment variables, but I don't think it's respected.

docker buildx create \
  --name s3-test \
  --driver docker-container \
  --driver-opt env.AWS_ACCESS_KEY_ID=XXX \
  --driver-opt env.AWS_SECRET_ACCESS_KEY=XXX \
  --driver-opt env.AWS_REQUEST_CHECKSUM_CALCULATION=when_required \
  --driver-opt env.AWS_RESPONSE_CHECKSUM_VALIDATION=when_required \
  --use

I get this error, when I try to use the cache:

ERROR: failed to solve: error writing manifest: myapp-cache: operation error S3: PutObject, https response error StatusCode: 403, RequestID: , HostID: , api error SignatureDoesNotMatch: Access denied.
7 v0.23.2 buildkitd --allow-insecure-entitlement=network.host
github.com/moby/buildkit/cache/remotecache/s3.(*exporter).Finalize
        /src/cache/remotecache/s3/s3.go:293
github.com/moby/buildkit/solver/llbsolver.runCacheExporters.func1.1
        /src/solver/llbsolver/solver.go:783
github.com/moby/buildkit/solver/llbsolver.inBuilderContext.func1
        /src/solver/llbsolver/solver.go:1153
github.com/moby/buildkit/solver.(*Job).InContext
        /src/solver/jobs.go:824
github.com/moby/buildkit/solver/llbsolver.inBuilderContext
        /src/solver/llbsolver/solver.go:1149
github.com/moby/buildkit/solver/llbsolver.runCacheExporters.func1
        /src/solver/llbsolver/solver.go:764
golang.org/x/sync/errgroup.(*Group).add.func1
        /src/vendor/golang.org/x/sync/errgroup/errgroup.go:130
runtime.goexit
        /usr/local/go/src/runtime/asm_arm64.s:1223