geoip2 always process logs, regardless if lookup_key exist

[helmut72]

Bug Report

Describe the bug

geoip2 always process logs, regardless if lookup_key exist or not. It always adds empty/nil geoip fields to log lines.

To Reproduce

Add geoip2 filter and send a log without 'lookup_key'. Example config:

[INPUT]
    Name   dummy
    Tag    dummytag
    Dummy  {"any_key": "any_value"}

[FILTER]
    Name geoip2
    Match dummytag
    Database   /geoip/GeoLite2-City.mmdb
    Lookup_key remote_addr
    Record country remote_addr %{country.names.en}
    Record isocode remote_addr %{country.iso_code}

[OUTPUT]
    Name   stdout
    Match  dummytag

Output:

[0] dummytag: [[1691872691.252361439, {}], {"any_key"=>"any_value", "country"=>nil, "isocode"=>nil}]

Expected behavior

Ignore the log line if lookup_key doesn't exist. Are there really reasons to process the log, if no lookup_key exist?

Expected output:

[0] dummytag: [[1691872691.252361439, {}], {"any_key"=>"any_value"}]

Your Environment

fluent-bit 2.1.8

Thank you.

[patrick-stephens]

Probably a grep filter first to drop it? https://docs.fluentbit.io/manual/pipeline/filters/grep

input --> filter (grep) --> filter (geoip) --> output

The grep filter prevents the record carrying on if a key is missing:https://docs.fluentbit.io/manual/pipeline/filters/grep#excluding-records-missing-invalid-fields`

[helmut72]

The grep filter prevents the record carrying on if a key is missing

But I want and need the other log lines as it is. Just without empty geoip keys, because it doesn't make sense for log lines that doesn't contain the key configured in lookup_key.

For example a mailserver (postfix) doesn't log in every line ip addresses. But all other log lines are required too. It also saves CPU cycles, if a log line isn't parsed by geoip2 filter, when the key configured in lookup_key doesn't exist.

Have also tested re-tagging log lines which contains a remote_addr key as a workaround. Only this new tag will be processed by geoip2 filter. This works, but then the output is unordered from input.

I think it's a bug, because geoip2 filter process a "ghost" key, that doesn't exist.

[dmasteller4]

Having this same issue with a similar configuration in version 2.1.10

[helmut72]

My workaround is checking later, if there is a remote_addr key at all. If not, geoip lookup must have failed and therefore it's save to delete all geo* keys for this log line.

# workaround: remove all geo_* tags if there wasn't remote_addr
[FILTER]
    name      modify
    match     app1
    condition Key_does_not_exist remote_addr
    remove    geo_isocode
    remove    geo_city
    remove    geo_latitude
    remove    geo_longitude

[github-actions[bot]]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[helmut72]

Still think this behavior is a bug.

[YevhenLodovyi]

Still think this behavior is a bug.

+1

[github-actions[bot]]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[helmut72]

Still an issue