Fluent Bit Go Input interface doesn't support passing of context - seriously limiting its value

[mp3monster]

To use the Go custom plugin framework effectively we need to be able to access the context when the FLBPluginInputCallback is invoked. Without this, the callback can only perform hardwired tasks, and the configuration values can not be accessed. As a result, the input becomes extremely constrained on what it can do.

The corresponding output callback FLBPluginFlushCtx can access the context information. We also understand that the Calyptia version of the Fluent Bit Go layer has some resolution for input. It would be good to see this solution in the core implementation.

Describe the solution you'd like Given that the context is visible in the Fluent Bit goproxy ( https://github.com/fluent/fluent-bit/blob/689afa14ae8c06e8ae32c930eaeb7daa305705db/src/proxy/go/go.c#L241 ) we believe it should be possible to expose the context. Therefore following the pattern taken with the output plugin implementing a variant of the FLBPluginInputCallback callback e.g. FLBPluginInputCallbackCtx would suggest itself to be the ideal solution.

Describe alternatives you've considered The alternative would be that plugin instance Id is exposed to the plugin as we see from the outputs e.g. dummy.0, and the custom plugin takes responsibility for mapping the id to the relevant configuration values (i.e. manage its own context)

Additional context The background is that the Fluent Bit with Kubernetes book which is trying to incorporate the fact we can illustrate the development of a fully featured Go Input and Output plugins. Easing to the task of plugin development for those more comfortable with Go (and other languages that provide memory management and stronger typing than C)

The book is trying to illustrate this by implementing a plugin that goes beyond a simple variation of 'dummy' to being able to configure the plugin to query a database (the typical use cases for the plugin's application is to access the slow query table content held in a DB's system tables - as is the case, or access audit events that an application may write to the DB rather than to stdout or a log file).

Further investigation has shown people have built PoC and demo work on the output but not the input layer - probably because of this constraint

See section above

[agup006]

@pwhelan would you happen to know if there any workarounds for this?

[github-actions[bot]]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[github-actions[bot]]

This issue was closed because it has been stalled for 5 days with no activity.