aepp icon indicating copy to clipboard operation
aepp copied to clipboard

Support ingesting small parquet files

Open cmenguy opened this issue 3 years ago • 1 comments

Currently the small file API support in ingestion seems incomplete, as in the method uploadSmallFile it expects data of type Union[list, dict] which works fine for JSON, but for Parquet format you would be passing in bytes, I couldn't figure out how to get this working with some Parquet data so ended up using JSON. But I think either the prototype of the function needs to change since the notion of multiline isn't really applicable to Parquet, or we need to actually change the code to fully support Parquet binary payload.

cmenguy avatar Feb 01 '23 22:02 cmenguy

@pitchmuc please assign this to me as well

skanjila avatar Feb 10 '23 23:02 skanjila