LambdaEcsLayout seems to incorrectly serialize the service field

[RMSD]

What were you trying to accomplish?

Our Kibana instance couldn't parse the ECS layout provided by LambdaEcsLayout; we believe thanks to the service field.

Expected Behavior

Service should be a nested field: https://www.elastic.co/guide/en/ecs/current/ecs-service.html#_service_field_details Meaning ECS expects an object for service

Current Behavior

Looking at a sample prettied output:

{
      "samplingRate": "0.0", 
      "service": "service_undefined",
      "stack": "test", 
 }

This isn't what's expected by ECS layout.

Possible Solution

Ether provided a nested field

{
      "samplingRate": "0.0", 
      "service": {
         "name": "service_undefined"
       },
      "stack": "test", 
 }

or an implied nested field

{
      "samplingRate": "0.0", 
      "service.name": "service_undefined",
      "stack": "test", 
 }

Steps to Reproduce (for bugs)

My log4j2 config is pretty simple:

<?xml version="1.0" encoding="UTF-8"?>
<Configuration>
    <Properties>
        <Property name="environment">${env:Stack}</Property>
    </Properties>
    <Appenders>
        <Console name="JsonAppender" target="SYSTEM_OUT">
            <JsonTemplateLayout eventTemplateUri="classpath:LambdaEcsLayout.json">
                <EventTemplateAdditionalField key="labels.owner" value="myTeam"/>
                <EventTemplateAdditionalField key="labels.system_name" value="my-lambda"/>
                <EventTemplateAdditionalField key="labels.environment" value="${environment}"/>
            </JsonTemplateLayout>
        </Console>
    </Appenders>
    <Loggers>
        <Logger name="JsonLogger" level="INFO" additivity="false">
            <AppenderRef ref="JsonAppender"/>
        </Logger>
        <Root level="INFO">
            <AppenderRef ref="JsonAppender"/>
        </Root>
    </Loggers>
</Configuration>

the additional fields are some extra ECS fields that my company requires we add explicitly rather than implicitly.

Environment

• Powertools for AWS Lambda (Java) version used: 1.18.0
• Packaging format (Layers, Maven/Gradle): Maven
• AWS Lambda function runtime: Java 17
• Debugging logs

N/A

[jeromevdl]

Thanks @RMSD for reporting this issue.

That's not an easy change as the service is added as a custom key (see here) and used both in json layout and ecs layout as is. Fixing the ecs will also change the json and introduce a breaking change.

It is fixed in v2 (here) but not released yet.

What I can suggest as a quick fix is to do the LoggingUtils.appendKey("service.name", LambdaHandlerProcessor.serviceName()); in your Lambda function constructor.

[RMSD]

Thanks for the workaround! Glad to hear this is fixed in 2, I'd love to see the docs updated for the ECS layout in the until 2 comes out.

I'm also assuming I'd have to remove "service" similarly manually from LoggingUtils while adding this new field.

[jeromevdl]

Thanks for the workaround! Glad to hear this is fixed in 2, I'd love to see the docs updated for the ECS layout in the until 2 comes out.

I'll see if I can fix the bug when time permits.

I'm also assuming I'd have to remove "service" similarly manually from LoggingUtils while adding this new field.

Not sure. depends if it disturbs Kibana. If that's the case then yes. Please let me know if that's the case as it impacts my fix.

[RMSD]

Since this is a static value, I decided to set this in the log4j2 level so I don't have to add this to multiple handlers that my lambda has:

<?xml version="1.0" encoding="UTF-8"?>
<Configuration>
    <Properties>
        <Property name="environment">${env:Stack}</Property>
        <Property name="service_name">${env:POWERTOOLS_SERVICE_NAME}</Property>
    </Properties>
    <Appenders>
        <Console name="JsonAppender" target="SYSTEM_OUT">
            <JsonTemplateLayout eventTemplateUri="classpath:LambdaEcsLayout.json">
                <EventTemplateAdditionalField key="service.name" value="${service_name}"/>
                <EventTemplateAdditionalField key="labels.owner" value="myTeam"/>
                <EventTemplateAdditionalField key="labels.system_name" value="my-lambda"/>
                <EventTemplateAdditionalField key="labels.environment" value="${environment}"/>
            </JsonTemplateLayout>
        </Console>
    </Appenders>
    <Loggers>
        <Logger name="JsonLogger" level="INFO" additivity="false">
            <AppenderRef ref="JsonAppender"/>
        </Logger>
        <Root level="INFO">
            <AppenderRef ref="JsonAppender"/>
        </Root>
    </Loggers>
</Configuration>

This produced:

{
    "service": "my-lambda-handler",
    "stack": "test3",
    "xray_trace_id": "1-65d4f715-1c505d9e387e661d24a820cb",
    "service.name": "my-lambda handler",
}

So I'd say I have to hard remove it since this will still cause the parsing problems.

[RMSD]

I can confirm:

On top of what I had above, I added

    @Override
    @Logging
    public String handleRequest(DynamodbEvent dynamodbEvent, Context string) {
        LoggingUtils.removeKey("service");
    }

to my handler

and my appended output is now:

{
    "samplingRate": "0.0",
    "stack": "test",
    "xray_trace_id": "1-65d53c58-5c5fce1e0742264a383a4e36",
    "service.name": "service-name",
}

it seems to have been parsed correctly by our kibana after these changes.

[jeromevdl]

Thanks @RMSD

[jeromevdl]

We won't fix this in v1. Looks like it was not really used since this problem is quite important and was not reported until very recently (while the feature is there since a while).

The following snippet solves the problem:

LoggingUtils.appendKey("service.name", LambdaHandlerProcessor.serviceName());
LoggingUtils.removeKey("service");

v2 already fixes this (here for log4j and here for logback)