fluent-bit icon indicating copy to clipboard operation
fluent-bit copied to clipboard
verified publisher icon fluent

S3 Plugin: Implement Server-Side Encryption option

Open msolters opened this issue 1 year ago • 0 comments

  • Introduces a server_side_encryption parameter for the S3 [OUTPUT] plugin. Possible values are AES256 and aws:kms, as per AWS API documentation: https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObject.html#API_PutObject_ResponseSyntax
  • If either value is provided, the x-amz-server-side-encryption header will be included with S3 requests with the corresponding value set.

Enter [N/A] in the box, if an item is not applicable to your change.

Testing Before we can approve your change; please submit the following in a comment:

  • [x] Example configuration file for the change
  • [x] Debug log output from testing the change

Example config:

---
service:
  http_server: off
  http_listen: 0.0.0.0
  http_port: 2029
  flush: 1
  daemon: off
  log_level: info

pipeline:
  inputs:
    - name: tail
      read_from_head: true
      path: /etc/fluent/logs/*
      refresh_interval: 1
      ignore_older: 1m

  outputs:
    - name: stdout
      match: "*"
    - name: s3
      match: "*"
      bucket: "foobar"
      server_side_encryption: AES256

This shows the S3 output config loading correctly. Obviously the bucket isn't real in this config but the config load throws no error:

root@1969e5ffe6ff:/# fluent-bit/bin/fluent-bit -c /cfg/config.yaml
Fluent Bit v3.0.1
* Copyright (C) 2015-2024 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

___________.__                        __    __________.__  __          ________
\_   _____/|  |  __ __   ____   _____/  |_  \______   \__|/  |_  ___  _\_____  \
 |    __)  |  | |  |  \_/ __ \ /    \   __\  |    |  _/  \   __\ \  \/ / _(__  <
 |     \   |  |_|  |  /\  ___/|   |  \  |    |    |   \  ||  |    \   / /       \
 \___  /   |____/____/  \___  >___|  /__|    |______  /__||__|     \_/ /______  /
     \/                     \/     \/               \/                        \/

[2024/03/28 02:40:34] [ info] [fluent bit] version=3.0.1, commit=62b337ef78, pid=12
[2024/03/28 02:40:34] [ info] [storage] ver=1.5.1, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2024/03/28 02:40:34] [ info] [cmetrics] version=0.7.0
[2024/03/28 02:40:34] [ info] [ctraces ] version=0.4.0
[2024/03/28 02:40:34] [ info] [input:tail:tail.0] initializing
[2024/03/28 02:40:34] [ info] [input:tail:tail.0] storage_strategy='memory' (memory only)
[2024/03/28 02:40:34] [error] [input:tail:tail.0] read error, check permissions: /etc/fluent/logs/*
[2024/03/28 02:40:34] [ warn] [input:tail:tail.0] error scanning path: /etc/fluent/logs/*
[2024/03/28 02:40:34] [ info] [output:stdout:stdout.0] worker #0 started
[2024/03/28 02:40:34] [ info] [output:s3:s3.1] Using upload size 100000000 bytes
[2024/03/28 02:40:34] [error] [/src/fluent-bit/src/flb_http_client.c:1239 errno=111] Connection refused
[2024/03/28 02:40:34] [ warn] [imds] unable to evaluate IMDS version
...

When the bucket is real (and IAM permissions exist) this does correctly upload to SSE-enabled buckets.

A negative example, where server_side_encryption: wumbo, fails to load as expected:

root@1969e5ffe6ff:/# fluent-bit/bin/fluent-bit -c /cfg/config.yaml
Fluent Bit v3.0.1
* Copyright (C) 2015-2024 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

___________.__                        __    __________.__  __          ________
\_   _____/|  |  __ __   ____   _____/  |_  \______   \__|/  |_  ___  _\_____  \
 |    __)  |  | |  |  \_/ __ \ /    \   __\  |    |  _/  \   __\ \  \/ / _(__  <
 |     \   |  |_|  |  /\  ___/|   |  \  |    |    |   \  ||  |    \   / /       \
 \___  /   |____/____/  \___  >___|  /__|    |______  /__||__|     \_/ /______  /
     \/                     \/     \/               \/                        \/

[2024/03/28 02:42:31] [ info] [fluent bit] version=3.0.1, commit=62b337ef78, pid=27
[2024/03/28 02:42:31] [ info] [storage] ver=1.5.1, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2024/03/28 02:42:31] [ info] [cmetrics] version=0.7.0
[2024/03/28 02:42:31] [ info] [ctraces ] version=0.4.0
[2024/03/28 02:42:31] [ info] [input:tail:tail.0] initializing
[2024/03/28 02:42:31] [ info] [input:tail:tail.0] storage_strategy='memory' (memory only)
[2024/03/28 02:42:31] [error] [input:tail:tail.0] read error, check permissions: /etc/fluent/logs/*
[2024/03/28 02:42:31] [ warn] [input:tail:tail.0] error scanning path: /etc/fluent/logs/*
[2024/03/28 02:42:31] [ info] [output:stdout:stdout.0] worker #0 started
[2024/03/28 02:42:31] [ info] [output:s3:s3.1] Using upload size 100000000 bytes
[2024/03/28 02:42:31] [error] [aws_compress] unknown sse type: wumbo
[2024/03/28 02:42:31] [error] [output:s3:s3.1] unknown server-side encryption type: wumbo
[2024/03/28 02:42:31] [error] [output] failed to initialize 's3' plugin
[2024/03/28 02:42:31] [error] [engine] output initialization failed
[2024/03/28 02:42:32] [ info] [input] pausing tail.0
[2024/03/28 02:42:32] [ info] [output:stdout:stdout.0] thread worker #0 stopping...
[2024/03/28 02:42:32] [ info] [output:stdout:stdout.0] thread worker #0 stopped

Starting without server_side_encryption being defined at all also doesn't crash:

root@1969e5ffe6ff:/# fluent-bit/bin/fluent-bit -c /cfg/config.yaml
Fluent Bit v3.0.1
* Copyright (C) 2015-2024 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

___________.__                        __    __________.__  __          ________
\_   _____/|  |  __ __   ____   _____/  |_  \______   \__|/  |_  ___  _\_____  \
 |    __)  |  | |  |  \_/ __ \ /    \   __\  |    |  _/  \   __\ \  \/ / _(__  <
 |     \   |  |_|  |  /\  ___/|   |  \  |    |    |   \  ||  |    \   / /       \
 \___  /   |____/____/  \___  >___|  /__|    |______  /__||__|     \_/ /______  /
     \/                     \/     \/               \/                        \/

[2024/03/28 03:11:07] [ info] [fluent bit] version=3.0.1, commit=62b337ef78, pid=31
[2024/03/28 03:11:07] [ info] [storage] ver=1.5.1, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2024/03/28 03:11:07] [ info] [cmetrics] version=0.7.0
[2024/03/28 03:11:07] [ info] [ctraces ] version=0.4.0
[2024/03/28 03:11:07] [ info] [input:tail:tail.0] initializing
[2024/03/28 03:11:07] [ info] [input:tail:tail.0] storage_strategy='memory' (memory only)
[2024/03/28 03:11:07] [error] [input:tail:tail.0] read error, check permissions: /etc/fluent/logs/*
[2024/03/28 03:11:07] [ warn] [input:tail:tail.0] error scanning path: /etc/fluent/logs/*
[2024/03/28 03:11:07] [ info] [output:stdout:stdout.0] worker #0 started
[2024/03/28 03:11:07] [ info] [output:s3:s3.1] Using upload size 100000000 bytes
[2024/03/28 03:11:07] [error] [/src/fluent-bit/src/flb_http_client.c:1239 errno=111] Connection refused
[2024/03/28 03:11:07] [ warn] [imds] unable to evaluate IMDS version
  • [x] Attached Valgrind output that shows no leaks or memory corruption was found

Valgrind heap summary with server_side_encryption: AES256:

[2024/03/28 03:34:43] [ info] [output:s3:s3.1] thread worker #0 stopping...
[2024/03/28 03:34:43] [ info] [output:s3:s3.1] thread worker #0 stopped
==12==
==12== HEAP SUMMARY:
==12==     in use at exit: 0 bytes in 0 blocks
==12==   total heap usage: 8,566 allocs, 8,566 frees, 949,054 bytes allocated
==12==
==12== All heap blocks were freed -- no leaks are possible
==12==
==12== For lists of detected and suppressed errors, rerun with: -s
==12== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

  • [ ] Run local packaging test showing all targets (including any new ones) build.
  • [ ] Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

  • [x] Documentation required for this feature

Documentation PR here

Backporting

  • [ ] Backport to latest stable release.

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

msolters avatar Mar 28 '24 03:03 msolters