Christopher Bynum
Christopher Bynum
The purpose of this was to convert an existing RDD to a DataFrame. I don't understand how this could be achieved with a UDF. My use case was streaming (traditional,...
Actually I suppose if you converted the RDD to a DataFrame containing a Row of Avro objects, then a UDF could be applied, but I still don't understand the benefit.
I'm happy to do anything necessary to get a solution merged. But in the meantime it is pretty easy to use the code in this PR. All you need are...
You can do this with `DStream.foreachRDD { rdd => df = rdd.toDF ... }` using the code in https://github.com/databricks/spark-avro/pull/216.
I implemented it as an implicit on `RDD[GenericRecord]`. If you `import RddUtils.RddToDataFrame` then you can call `toDF` on the RDD as I posted above.
Would also love to see this
https://github.com/databricks/spark-avro/pull/216