Gleb Kanterov
Gleb Kanterov
I'm not sure there is a way to fix it. Internally Spark represents `Option[Option[Int]]` as a single nullable `Integer`, so there is no difference in encoding `Some(None)` and `None`. To...
Spark has exactly the same behaviour: ```scala scala> val xs: List[Option[Option[Int]]] = List(Some(None), None) xs: List[Option[Option[Int]]] = List(Some(None), None) scala> spark.createDataset(xs).collect() res3: Array[Option[Option[Int]]] = Array(Some(None), Some(None)) ```
Another example of things is `X1[Option[X1[Option[Int]]]`, it doesn't work because in Spark struct itself can't be nullable, only it's fields, that's why in this case we can't find the difference...
@tscholak hm... good question, probably the best is to add to the test suite, AFAIK should work if `A` is primitive
@pietern Thanks. I've updated the code, PR description, and title. Can you add it to the merge queue?
@pietern can you please take a look, since the last time: - `merge.Override` has changed, so now we implement `OverrideVisitor` here, I added tests for it as well - left...
@tnsetting Please be mindful of the naming conventions for proto fields. See https://developers.google.com/protocol-buffers/docs/style#message_and_field_names
@pietern I've addressed all comments, and also renamed "pre-init" to "load" as we discussed offline
@pietern thanks for the review, can you please add to the merge queue?
@denik can you PTAL since you suggested adding headers?