Dr Mich Talebzadeh

Hi, I have a similar error writing to BigQuery This is message sent that I can get it from console ``` $KAFKA_HOME/bin/kafka-console-consumer.sh --bootstrap-server rhes75:9092,rhes75:9093,rhes75:9094,rhes564:9092,rhes564:9093,rhes564:9094,rhes76:9092,rhes76:9093,rhes76:9094 --from-beginning --topic md --property print.key=true ```...

Hi, I sorted this one out using foreachBatch function in writeStream. Basically streaming data rows (securities) through Kafka to Spark structured Streaming and saving data to BQ via Spark BigQuery...

Hi, I am using the latest build of Dataproc 2.0.x release versions that uses Apache spark 3.1.1-RC 2 (release candidate). 2.0.x release versions | Dataproc Documentation | Google Cloud In...

Thanks I added the details to Google Dataproc mailing list. Cheers, Mich

I believe with mr as engine, there is the issue of OutputCommitter not being called somehow, something probably to do with jar.

Hi After using the correct jar file added to Hive as below ```ADD JAR /hive-bigquery-storage-handler-1.0-shaded.jar;``` and also adding the jar file gcs-connector-hadoop3-2.2.5-shaded.jar (as suggested by Igor), to HADOOP_CLASSPATH, I can...

As I stated this seems to be an issue with mr as engine. There is now the issue of OutputCommitter not being called somehow, something probably to do with jar,...

When encountering mixed schema rows, the current error message "{actual} is not a valid external type for schema of {expected}" lacks sufficient detail to identify the problematic column. This ambiguity...