Log record "extra" fields are missing when using LogstashFormatter

[droriv]

When using LogstashFormatter the specific log record "extra" is missing from the formatted message

from loguru import logger
from logstash_async.handler import AsynchronousLogstashHandler
from logstash_async.formatter import LogstashFormatter

logstash_handler = AsynchronousLogstashHandler(
    remote_host,
    remote_port,
    database_path,
)

logstash_formatter = LogstashFormatter(
    extra={
        'default': 'extra',
    },
)

logstash_handler.setFormatter(logstash_formatter)
logger.add(
    logstash_handler,
    level=level,
    format='{message}',
    backtrace=False,
)

logger.info('some message', extra={'some': 'extra'})

formatted message won't include the extra={'some': 'extra'} but will include 'default': 'extra' This is because of the _get_extra_fields method which doesn't add the record.extra to the extra fields. solved it locally by overriding the _get_extra_fields locally

class MyFormatter(LogstashFormatter):
    def _get_extra_fields(self, record):
        extra_fields = super()._get_extra_fields(record=record, )

        if record.extra:
            extra_fields = record.extra | extra_fields
        return extra_fields

thought you might want to add this to the code as well

thanks!

[eht16]

Sorry for the late response.

I can't reproduce the described behavior with your example. It results in a message like (stripped unrelated fields for readability):

{
    "message": "some message.",
    "type": "python-logstash",
    "extra": {
        "some": "extra",
        "default": "extra",
    }
}

Which version of the library are you using?

[droriv]

I'm using python-logstash-async==2.3.0

[eht16]

I just tried with your code again, except that I replaced the logger with a standard Python logger instance and added the handler via logger.addHandler(logstash_handler) and got the expected result.

Might the loguru package related? I don't what it does.

For reference, here is a full message:

{
       "message" => "some message",
           "pid" => 551789,
         "extra" => {
                     "func_name" => "<module>",
           "interpreter_version" => "3.9.9",
                          "line" => 30,
        "logstash_async_version" => "2.3.0",
                  "process_name" => "MainProcess",
                          "some" => "extra",
                          "path" => "test_issue69.py",
                   "thread_name" => "MainThread",
                       "default" => "extra",
                   "interpreter" => "venv/bin/python",
                   "logger_name" => "root"
    },
          "host" => "redacted",
       "program" => "test_issue69.py",
      "@version" => "1",
    "@timestamp" => 2022-01-09T15:09:46.441Z,
          "port" => 37278,
          "type" => "python-logstash",
         "level" => "INFO",
     "logsource" => "redacted"
}

[droriv]

It might be related to loguru but as I mentioned I was able to track the problem, it's related to the _get_extra_fields method and after overriding it with the fix, the issue was resolved, is it ok if I'll open a pr for that fix?

[eht16]

Basically a PR is ok but I would like to understand what's the difference and why it works for me with your example and why not for you.

[eht16]

Feel free to open a PR if you are still interested in and re-open this one. Until then, I like to close this issue.