dsq icon indicating copy to clipboard operation
dsq copied to clipboard

deprecated INT96 timestamp not supported for older Parquet files

Open andyEllixson opened this issue 3 years ago • 0 comments

Describe the bug and expected behavior I have a large number of customer Paquet files which have the timestamp in INT96 format. Timestamp is being interpreted as STR rather than a timestamp type. Timestamp should display as ISO8601 format Reproduction steps file/metadata version: ############ file meta data ############ created_by: parquet-mr version 1.10.1-databricks6 (build bd2ebc87e42b3936ac673e1556fa10fb8358307a) num_columns: 3 num_rows: 376504 num_row_groups: 1 format_version: 1.0 serialized_size: 863

############ Columns ############ event_time key value

############ Column(event_time) ############ name: event_time path: event_time max_definition_level: 1 max_repetition_level: 0 physical_type: INT96 logical_type: None converted_type (legacy): NONE

command: dsq c:/tmp/easy-path.parquet "select event_time from {} limit 4"

Versions windows 10: cmd.exe

  • dsq version: [e.g. 0.20.1]

Additional context, screenshots c:\tmp>dsq --pretty c:/tmp/easy-path.parquet "select event_time from {} limit 2" +------------+ | Event_time | +------------+ ♣�% | | ��►►)6 �% | +------------+ (2 rows easy-path.parquet.txt )

note: change extension to ".txt" strictly for attaching to this issue. Remove .txt prior to testing with it.

andyEllixson avatar Jul 23 '22 11:07 andyEllixson