numaproj
Configurable Logging for Dropped Messages in User-Defined Sinks
Summary
In user-defined sinks, when Message.Drop() is called, it would be helpful to have a configurable option to log the dropped messages for debugging purposes.
Use Cases
This feature would be especially useful when developing or debugging custom sinks, where it's important to understand why messages are being dropped.
Proposed Behavior:
- Introduce a configuration option that allows users to enable logging of messages passed to Message.Drop().
- By default, this option should be disabled to avoid unintentionally logging sensitive or high-volume data.
Message from the maintainers:
If you wish to see this enhancement implemented please add a 👍 reaction to this issue! We often sort issues this way to know what to prioritize.
What if the pipeline contains multiple sinks. That feels odd to log a dropped event twice. Wouldn't it be better to log the dropped event in the vertex container where it's dropped?
It will be logged only once. drop means, do not forward to any outgoing edge.
