apache
Add example for writing a `FileFormat`
Is your feature request related to a problem or challenge?
Now that @devinjdangelo has added better support for user defined file formats (see https://github.com/apache/datafusion/pull/11060)
We have an example of a custom table provider: https://datafusion.apache.org/library-user-guide/custom-table-providers.html and example
It would be great to have an example of creating a file format
Such an example would make sure the APIs added in this PR can be used for the end to end case, as well as obviously made it easier for others to discover and use this feature
Describe the solution you'd like
Add an example in https://github.com/apache/datafusion/tree/main/datafusion-examples that does:
- Creates a new file format (something basic like use CSV / ROT13)
- reads / writes data using that format
Describe alternatives you've considered
No response
Additional context
No response
I'll take this as I want to try it out. I do this for one of my libraries via the parser plus user defined nodes, so I want to see what this looks like as an alternative.
I'll take this as I want to try it out. I do this for one of my libraries via the parser plus user defined nodes, so I want to see what this looks like as an alternative.
Thanks @tshauck -- note https://github.com/apache/datafusion/pull/11060 isn't merged yet, so you may have to start the example against that branch rather than main