github-raphael-douyere comments

Repositories
Issues
Comments

Results 3 comments of


                                            github-raphael-douyere

Do I have to restart the Spark streaming job for new schema to take effect ?

What could be done is to run Spark's streaming "forEachBatch" and initialize the `AbrisConfig` inside it. But it is a really limiting approach. I'm a bit surprised that there is...

Do I have to restart the Spark streaming job for new schema to take effect ?

Thanks for the answer. I see the issue: Spark infers the DataFrame schema only once and it cannot changed. An alternative would be to be able to stop the app...

Timestamp conversion issue

Sure, the stack trace is as follow : ``` [2023-05-03 15:08:11,082] ERROR WorkerSinkTask{id=test-partitioned-table-sink-0} Task threw an uncaught and unrecoverable exception. Task is being killed and will not recover until manually...