hillview
hillview copied to clipboard
Duration column should be handled by ParquetFileWriter
Currently the ParquetFileWriter cannot handle table with parquet columns. Due to the following reasons
- Parquet has a
Intervallogical type which has the same meaning of HillviewDuration, but its format is too convoluted. It's essentially 3 int32 numbers representing months, days, and milliseconds. If we only use the milliseconds field there might be a precision loss. - If we save the
Durationcolumn asDouble. Then currently we cannot guarantee reading back a saved table yields the same format as the original table.
One possible solution might be save as Double but also save a schema file.