How to capture bad records while using from_avro in abris

Open Tarannump opened this issue 3 years ago • 3 comments

I have use case to capture the bad records and store it in a separate location for future reference. Is it possible to get the records which are not deserialised instead of dropping them?

Oct 27 '22 18:10 Tarannump

You can use exception handlers introduced in this PR: https://github.com/AbsaOSS/ABRiS/pull/290

You will have to implement your own handler that will store the records though.

Oct 31 '22 08:10 cerveada

Thanks @cerveada for your response. We are using Glue Streaming ETL job which contains spark 3.1.1. Based on the PR looks like exception handlers are added in abris 6.3.0 which has spark 3.2.1. Is it possible to get the same feature in abris 5 as well?

Oct 31 '22 23:10 Tarannump

Hi @Tarannump Unfortunately we don't have the capacity to backport the feature to ABRiS 5. Of course, you are welcome to fork the repo or submit a pull request.

Nov 01 '22 08:11 kevinwallimann