cdsapi icon indicating copy to clipboard operation
cdsapi copied to clipboard

(short) loss of connection corrupts data

Open BSchilperoort opened this issue 2 years ago • 0 comments

When trying to download fapar LAI data from the CDS using this package, I encountered a (short) loss of internet connection.

This caused the following error:

2023-02-28 11:50:27,112 ERROR Download incomplete, downloaded 1337900919 byte(s) out of 15881569751 2023-02-28 11:50:27,113 WARNING Sleeping 10 seconds 2023-02-28 11:50:37,140 WARNING Resuming download at byte 1337900919

However, the resulting .zip file has contains one netcdf file with a size of 1 KB, while the other files are all ~400-500 MB. This corrupted netcdf file is not openable.

It would be better if the request would completely fail, rather than corrupting an (unknown) number of files, which is only visible once extracting the .zip file and looking at the file sizes.

Code of my request
    cds_client.retrieve(
        "satellite-lai-fapar",
        {
            "format": "zip",
            "variable": "lai",
            "satellite": [
                "proba", "spot",
            ],
            "sensor": "vgt",
            "horizontal_resolution": "1km",
            "product_version": "V3",
            "year": f"{year}",
            "month": [
                "01", "02", "03",
                "04", "05", "06",
                "07", "08", "09",
                "10", "11", "12",
            ],
            "nominal_day": [
                "10", "20", "28",
                "30", "31",
            ],
        },
        f"LAI_fapar_vgt_{year}.zip"
    )

BSchilperoort avatar Feb 28 '23 12:02 BSchilperoort