flyte icon indicating copy to clipboard operation
flyte copied to clipboard

[flyteadmin] Add support for KMS SSE to S3 backend

Open ddl-rliu opened this issue 1 year ago • 1 comments

Tracking issue

Part of a group:

  1. https://github.com/flyteorg/stow/pull/11
  2. https://github.com/flyteorg/flyte/pull/4897
  3. https://github.com/flyteorg/flytekit/pull/2193

Why are the changes needed?

S3 Stow implementation does not yet support setting ServerSideEncryption (SSE). We are particularly interested in the AWS Key Management Service (KMS) case.

What changes were proposed in this pull request?

Adds a new extra_args key (optional string field) to the stow config, which contains the keys/value like ServerSideEncryption: x, SSEKMSKeyId: x. It is passed through the storage.yaml section.

How was this patch tested?

See https://github.com/flyteorg/stow/pull/11

Tested on a Flyte deployment, against an S3 bucket with policy denying any request without "s3:x-amz-server-side-encryption": "aws:kms".

(Pdb) rsp.url
'https://...s3.us-west-2.amazonaws.com/...&X-Amz-SignedHeaders=content-md5%3Bhost%3B
x-amz-server-side-encryption%3Bx-amz-server-side-encryption-aws-kms-key-id
&x-amz-server-side-encryption=aws%3Akms&x-amz-server-side-encryption-aws-kms-key-id=...
&X-Amz-Signature=...'
(Pdb) rsp.status_code
200
(Pdb) rsp.headers
{'x-amz-id-2': '...', 'x-amz-request-id': '...', 'Date': 'Thu, 15 Feb 2024 23:28:48 GMT', 
'x-amz-server-side-encryption': 'aws:kms', 'x-amz-server-side-encryption-aws-kms-key-id': ''...', 
'Server': 'AmazonS3', ...}

Related PRs

  1. https://github.com/flyteorg/stow/pull/11
  2. https://github.com/flyteorg/flyte/pull/4897
  3. https://github.com/flyteorg/flytekit/pull/2193

Docs link

ddl-rliu avatar Feb 13 '24 22:02 ddl-rliu

Thank you for opening this pull request! 🙌

These tips will help get your PR across the finish line:

  • Most of the repos have a PR template; if not, fill it out to the best of your knowledge.
  • Sign off your commits (Reference: DCO Guide).

welcome[bot] avatar Feb 13 '24 22:02 welcome[bot]

Filed https://github.com/flyteorg/flyte/issues/4949 to discuss the ETags aren't MD5 issue

ddl-ebrown avatar Feb 25 '24 03:02 ddl-ebrown

See https://github.com/flyteorg/stow/pull/11#issuecomment-1979815168

ddl-rliu avatar Mar 05 '24 23:03 ddl-rliu