Kevin Wallimann

For example, `za.co.absa.abris.avro.sql.AvroDeserializer` is a copy of `org.apache.spark.sql.avro.AvroDeserializer` (from spark version 2.4) with a few changes. It would be better to somehow reuse the original implementation otherwise, ABRiS' AvroDeserializer will...

Hi @kpr9991 Thanks for your examples. I ran your code with Abris and consumed the records as expected. I don't see any errors in your provided code. What versions of...

Hi @kpr9991 I was able to run the example using `.read` and `.write` and could ingest the records as expected. I don't see that this is a problem in Abris....

Hi @akshayar In Spark, it's impossible to change the schema during a run. So, indeed you have to restart the streaming job to consume the new field.

Hi @akshayar This is a limitation of Spark, so it's not in the scope of this project.

`pull_request_target` doesn't work like we think. It "runs in the context of the base of the pull request", so it doesn't actually run with the changes from the pull request....

Hi @github-raphael-douyere Spot on. Theoretically, it may be possible to change the logical plan from one microbatch to the next one, but I don't know how this could be implemented...

Hi @Tarannump Unfortunately we don't have the capacity to backport the feature to ABRiS 5. Of course, you are welcome to fork the repo or submit a pull request.

Hi @jelmew Thanks for reporting the issue. Could you please double-check if you are really using v3.5.0, or maybe the latest master version? I'm looking at https://github.com/apache/spark/blob/master/connector/avro/src/main/scala/org/apache/spark/sql/avro/AvroDeserializer.scala#L59 and indeed I...

Hi @jelmew Unfortunately, I don't have access to Spark on Databricks Runtime, but I will reproduce and fix the error by building Spark locally using the latest commit on [branch-3.5](https://github.com/apache/spark/tree/branch-3.5)....