Filter: UUID

[RobertRad]

Is your feature request related to a problem? Please describe. I want to identify a specific log line in the system to which fluent-bit pushed the logs. As I seem not able to do this with Loki, I would like to add a unique log id to every line.

I tried to create an UUID with lua - there are several examples online. But in the end for all, I didn't have any senseful input to math.randomseed(...). The best was os.time(), but this would create a clash for all log lines in created in the same second. I'm not sure if using only the nano parts of the timestamp is sufficient. Using both seems not be applicable, because the math.randomseed(...)` is using an integer.

Describe the solution you'd like A Filter which adds an UUID to a record under a specified key.

[github-actions[bot]]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[adrian-skybaker]

I'm trying to do fluentbit -> Kinesis -> Elastic/other systems, and this would be useful as a way to eliminate duplicates.

It's a pity that UUID support was added to the elasticsearch output only, rather than as a more general option.

[leonardo-albertovich]

If I understood @RobertRad correctly what he wants to do is track which fluent-bit instance pushed which log record using a UUID.

In that case I think either the modify or record modifier filters would do the trick.

What you need to do is either set the UUID in your configuration or an environment variable which sounds like the right way to do it in containers / kubernetes since you don't want these UUIDs to be ephemeral because that would defeat the purpose.

[RobertRad]

@leonardo-albertovich That is not exactly what I am trying to do.

To give you a more broad overview of my use-case: All my logs land in Loki in Grafana cloud and I want to be able to share a link to a specific log line with my colleges. Like "this is the line, which tells us, why our systems crashed" or something like this. So every single generated log line needs to have an own UUID. Since this seems not possible, with Grafana / Loki itself, so I would like to have this feature in fluentbit. And it seems to make sense for other use cases, like the use case of detecting duplicates, @adrian-skybaker just mentioned.

[leonardo-albertovich]

Thanks for clarifying that @RobertRad, in that case I think the easiest and more generic approach to this would be exposing flb_utils_uuid_v4_gen to lua in flb_luajit_create.

I'm not entirely sure what the posture is when it comes to exposing functionality to lua scripts since there is no precedent but it sounds much simpler than creating a dedicated filter for this.

Feel free to ping me in slack if you are willing to make a PR for this, I think the complexity level, line count and time required both for the core functionality and test case should be fairly low according to this reference.

I'd like to encourage you to give it a try, I can assure you I'll be around if you get stuck =)

[nokute78]

I sent a patch to add uuid to each record. https://github.com/fluent/fluent-bit/pull/6422

[RobertRad]

Solved through the patch