PySUS icon indicating copy to clipboard operation
PySUS copied to clipboard

Pandas does not concatenate the parquet files

Open esloch opened this issue 3 years ago • 0 comments

Unable to concatenate parquet files to dataframe.

disease dengue year 2020

https://github.com/AlertaDengue/PySUS/blob/master/pysus/tests/test_data/test_sinan.py#L43

48 DENGBR20.parquet/6d35585c41984b459cc3f72986f9aa6c-0.parquet
49 DENGBR20.parquet/f1d9168f79e1431a933faa433859c305-0.parquet
50 DENGBR20.parquet/5d8a203ccca04e87b12c5ef465a4d32d-0.parquet
Killed
pysus/tests/test_data/test_sinan.py::TestSINANDownload::test_chunked_df_size Killed

Reproduce the error:

fn = download(2020, "dengue", return_fname=True)
for i, f in enumerate(glob(f"{fn}/*.parquet")):
    if i == 0:
        df2 = pd.read_parquet(f)
    else:
        df2 = pd.concat([df2, pd.read_parquet(f)], ignore_index=True)

esloch avatar Sep 02 '22 20:09 esloch