[flyteadmin] Add support for KMS SSE to S3 backend
Tracking issue
Part of a group:
- https://github.com/flyteorg/stow/pull/11
- https://github.com/flyteorg/flyte/pull/4897
- https://github.com/flyteorg/flytekit/pull/2193
Why are the changes needed?
S3 Stow implementation does not yet support setting ServerSideEncryption (SSE). We are particularly interested in the AWS Key Management Service (KMS) case.
What changes were proposed in this pull request?
Adds a new extra_args key (optional string field) to the stow config, which contains the keys/value like ServerSideEncryption: x, SSEKMSKeyId: x. It is passed through the storage.yaml section.
How was this patch tested?
See https://github.com/flyteorg/stow/pull/11
Tested on a Flyte deployment, against an S3 bucket with policy denying any request without "s3:x-amz-server-side-encryption": "aws:kms".
(Pdb) rsp.url
'https://...s3.us-west-2.amazonaws.com/...&X-Amz-SignedHeaders=content-md5%3Bhost%3B
x-amz-server-side-encryption%3Bx-amz-server-side-encryption-aws-kms-key-id
&x-amz-server-side-encryption=aws%3Akms&x-amz-server-side-encryption-aws-kms-key-id=...
&X-Amz-Signature=...'
(Pdb) rsp.status_code
200
(Pdb) rsp.headers
{'x-amz-id-2': '...', 'x-amz-request-id': '...', 'Date': 'Thu, 15 Feb 2024 23:28:48 GMT',
'x-amz-server-side-encryption': 'aws:kms', 'x-amz-server-side-encryption-aws-kms-key-id': ''...',
'Server': 'AmazonS3', ...}
Related PRs
- https://github.com/flyteorg/stow/pull/11
- https://github.com/flyteorg/flyte/pull/4897
- https://github.com/flyteorg/flytekit/pull/2193
Docs link
Thank you for opening this pull request! 🙌
These tips will help get your PR across the finish line:
- Most of the repos have a PR template; if not, fill it out to the best of your knowledge.
- Sign off your commits (Reference: DCO Guide).
Filed https://github.com/flyteorg/flyte/issues/4949 to discuss the ETags aren't MD5 issue
See https://github.com/flyteorg/stow/pull/11#issuecomment-1979815168