Apply log_ecs_reformatting to console/System.out appenders

[eyalkoren]

We currently restrict ecs-reformatting only to file appenders (hardcoded in the implementations of createAndStartEcsAppender() implementations).

We should apply SHADE and OVERRIDE options to standard output logging as well, especially since this is a common logging output in containerized environments.

[eyalkoren]

@fgabolde just referred me to my own words:

NOTE: while SHADE and REPLACE options are only relevant to file log appenders, the OVERRIDE option is also valid for other appenders, like System out and console

which referred me to my own code, which behaves differently when log_ecs_reformatting == OVERRIDE - creating and setting an ECS reformatter to any appender, as long as the formatter is allowed.

What's still missing is the ability to do the same when log_ecs_reformatting == SHADE. When prioritizing, we need to consider whether there's an option we'll want to support that in the containerized environments use case. For example - duplicate/shade the stdout log in k8s to a shared volume monitored by Elastic Agent, instead of overriding it.

[SylvainJuge]

My understanding is currently the following:

App logs --(SHADE)--> plaintext original output unaffected + ECS JSON output (separate log file) App logs --(REPLACE)--> original output discarded + ECS JSON output (separate log file) App Logs --(OVERRIDE)--> original output destination written in ECS JSON format (original location, just different format)

Which means that for file/console appenders, we should have the following:

SHADE duplicates to separate ECS JSON log stream.

• file appender: ECS JSON file created with a file next to original log file (or configured location)
• console appender: should we create a file to store the ECS JSON log ? --> what this issue seems to be about if I understand it correctly. (1)

REPLACE discards original log stream and route log events to ECS JSON format and a separate log files.

• file appender: log events not written to the original log file, but written to an ECS JSON formatted log created next to the original log file (or configured location)
• console appender: should we create a file to store the ECS JSON log (would be consistent with SHADE) ? --> seems also relevant, but not mentioned explicitly. (2)

OVERRIDE preserves log destination, but changes its format to ECS JSON

• file appender: log events are written to the original log file, but in ECS JSON format instead of plaintext
• console appender: log events are written to the console, but in ECS JSON format instead of plaintext

Thus I'll start with (1), then maybe also cover (2).

[SylvainJuge]

Closing this issue for now as the related PR was closed, the main reasons are the following:

• we don't have enough feedback on relevance of such feature in practice
• for container-based deployments, the most common way to send logs is through standard output and use filebeat auto-discover feature, thus most of the time the override reformatting mode will be enough to reformat log to ECS-JSON.
• we discovered a few complications during the implementation, for example the console appenders do not have a file-name, hence there might be cases where the agent would have to write to current working directory, which is not ideal

Of course, feel free to comment (or re-open) later if relevant.