[Feature Request]: Pass the entire Kafka Record to the kafka-to-bigquery template and thus the UDF

[darrenwjones]

Related Template(s)

kafka-to-bigquery/kafka-common

What feature(s) are you requesting?

I'd like to see the entire Kafka record passed to the UDF so that we can filter/enrich/transform/etc the message with all the available info in mind. Looking through the code I'm not 100% confident that the eventData passed to the UDF is solely the content of the Kafka record message but this code snippet makes me think so. If I am incorrect in this assumption it would be helpful to have the input spec filled out for this template here.

The most important of the metadata being the headers. The headers can include extra and vital information regarding the content of the message so that time is not wasted parsing the message to fetch said info.

This metadata/headers would also need to be added to the output for the dead letter table.

[github-actions[bot]]

This issue has been marked as stale due to 180 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the issue at any time. Thank you for your contributions.

[darrenwjones]

Bump