spark-avro
spark-avro copied to clipboard
databricks
Avro Data Source for Apache Spark
The below code reads the messages from Kafka and the messages are in Avro so how do I parse the message and put it into a dataframe in Spark 2.2.0?...
In response to issues #211 and #201
Hi The current version reads .avro files from HDFS (or) any FS file path. I am storing my avro file stream in Kafka. Do you have a utility to read...
Hello, I want to use spark-structured-streaming to process data fetched from kafka messages, and then store as rows cassandra database. I need one clarification. The message in kafka are serialized...
Avro doesn't support very big numbers directly. It supports it through logicalTypes where you can specify value as string type but send the actual data type of the field as...
Hi, The decimal logicalTypes are seen in DF as binary and values are in hexadecimal Ex. for "**3.12**" : ``` org.apache.spark.sql.DataFrame = [col1: binary] df.select("col1").show() +-------+ | col1| +-------+ |[01...
Would it be possible to add a notice at the beginning of the readme to warn that this datasource is merged into spark 2.4 and all users of spark 2.4+...
When I have a schema which has a reference to itself, it causes an infinite recursion and thus a StackOverflowError, SchemaConverters should have some sort of bail-out if it reads...
LM-SJC-11001988:~ mayangupta$ spark-shell --packages com.databricks:spark-avro_2.11:4.0.0 Ivy Default Cache set to: /Users/mayangupta/.ivy2/cache The jars for the packages stored in: /Users/mayangupta/.ivy2/jars :: loading settings :: url = jar:file:/Library/spark-2.2.1-bin-hadoop2.7/jars/ivy-2.4.0.jar!/org/apache/ivy/core/settings/ivysettings.xml com.databricks#spark-avro_2.11 added as a...
We are using Spark 1.6 in our clusters and want to use this library to read avro files. As part of reading the avro files, we want to able to...
