Support readonly container file system

[jhnlsn]

Opening this enhancement request to discuss the option proposed here: https://github.com/aws/amazon-cloudwatch-agent/pull/208

[supersime]

I absolutely would propose a solution to this issue - i.e. to allow the cloudwatch-agent container to run in read-only mode. In particular, when using AWS Security Hub, we get these HIGH severity FAILED alerts due to running cloudwatch-agent in a task as a sidecar to our application container:

This control checks if ECS containers are limited to read-only access to mounted root filesystems. This control fails if the ReadonlyRootFilesystem parameter in the container definition of ECS task definitions is set to ‘false’ For information on how to correct this issue, consult the AWS Security Hub controls documentation.: https://docs.aws.amazon.com/console/securityhub/ECS.5/remediation

We will not be able to resolve this HIGH severity alert until this container can be run as read-only.

In particular, when the container is run as read-only (i.e. with the ReadonlyRootFilesystem flag set to true in the ECS task definition > container definition (per the Security Hub remediation link above), it won't start. The errors in the ECS log are:

2023/07/25 06:42:35 E! Cannot translate JSON config into TOML, ERROR is exit status 1
2023/07/25 06:42:35 I! Return exit error: exit code=1
2023/07/25 06:42:35 Configuration validation first phase failed. Agent version: 1.0. Verify the JSON input is only using features supported by this version.
2023/07/25 06:42:35 E! Failed to create the configuration validation file. Reason: open /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.toml: read-only file system
2023/07/25 06:42:35 E! Failed to create the configuration validation file. Reason: open /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.toml: read-only file system
2023/07/25 06:42:35 I! Valid Json input schema.
2023/07/25 06:42:35 Reading json config from from environment variable CW_CONFIG_CONTENT.
2023/07/25 06:42:35 unable to scan config dir /etc/cwagentconfig with error: lstat /etc/cwagentconfig: no such file or directory
Cannot access /etc/cwagentconfig: lstat /etc/cwagentconfig: no such file or directory
/opt/aws/amazon-cloudwatch-agent/bin/default_linux_config.json does not exist or cannot read. Skipping it.
2023/07/25 06:42:35 Reading json config file path: /opt/aws/amazon-cloudwatch-agent/bin/default_linux_config.json ...
I! Detected the instance is ECS
2023/07/25 06:42:35 I! attempt to access ECS task metadata to determine whether I'm running in ECS.
E! [EC2] Cannot get EC2 Metadata from IMDS: EC2 metadata is not available.
2023/07/25 06:42:35 I! D! [EC2] Found active network interface