Download data (ERA5) error, OSX conda install

[CsillaVG]

Bug report 3/3 Info on conda install: https://aneto.pt/posts/tutorials/2019-05-29-using-qgis-from-conda/

Action :: The attempt to run Meteorological Data: Download data (ERA5) from the processing toolbox runs into error after receiving the files from Copernicus.

Environment :: macOS Ventura 13.2 running with M1 QGIS version: 3.40.1-Bratislava Python version: 3.11.11 UMEP 4.1.1 UMEP for processing 2.0.35 SuPy version: 2024.5.24.dev

Error message ::

Algorithm started at: 2024-12-12T08:14:21
Algorithm 'Meteorological Data: Download data (ERA5)' starting…
Input parameters:
{ 'CRS' : QgsCoordinateReferenceSystem('EPSG:4326'), 'DATEINIEND' : '2019-12-31T00:00:00', 'DATEINISTART' : '2019-01-01T00:00:00', 'DIAG_HEIGHT' : 2, 'INPUT_POINT' : '1.5,43.5 [EPSG:4326]', 'OUTPUT_DIR' : '/Users/vuk/Library/CloudStorage/Dropbox/eXchange/QGISprojects/Toulouse/_ERA5data' }

SuPy version: 2024.5.24.dev
INPUT PARAMETERS:
lat = 1.5
lon = 43.5
Start = 2019-01-01T00:00:00
End = 2019-12-31T00:00:00
Diagnostic height = 2.0
Output folder = /Users/vuk/Library/CloudStorage/Dropbox/eXchange/QGISprojects/Toulouse/_ERA5data
Traceback (most recent call last):
File "/Users/vuk/Library/Application Support/QGIS/QGIS3/profiles/default/python/plugins/processing_umep/preprocessor/copernicusera5_algorithm.py", line 182, in processAlgorithm
sp.util.gen_forcing_era5(lat, lon, startDate, endDate, hgt_agl_diag=diagHeight, dir_save=Path(outputDir))
File "/Users/vuk/.local/lib/python3.11/site-packages/supy/util/_era5.py", line 747, in gen_forcing_era5
ds_sfc = xr.open_mfdataset(
^^^^^^^^^^^^^^^^^^
File "/Users/vuk/.local/lib/python3.11/site-packages/xarray/backends/api.py", line 1579, in open_mfdataset
datasets = [open_(p, **open_kwargs) for p in paths1d]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/vuk/.local/lib/python3.11/site-packages/xarray/backends/api.py", line 1579, in <listcomp>
datasets = [open_(p, **open_kwargs) for p in paths1d]
^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/vuk/.local/lib/python3.11/site-packages/xarray/backends/api.py", line 651, in open_dataset
engine = plugins.guess_engine(filename_or_obj)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/vuk/.local/lib/python3.11/site-packages/xarray/backends/plugins.py", line 194, in guess_engine
raise ValueError(error_msg)
ValueError: did not find a match in any of xarray's currently installed IO backends ['netcdf4', 'scipy']. Consider explicitly selecting one of the installed engines via the ``engine`` parameter, or installing additional IO dependencies, see:
https://docs.xarray.dev/en/stable/getting-started-guide/installing.html
https://docs.xarray.dev/en/stable/user-guide/io.html

Execution failed after 6929.84 seconds (1 hour 55 minutes 30 seconds)

Loading resulting layers
Algorithm 'Meteorological Data: Download data (ERA5)' finished

[biglimp]

Did you try to download outside of QGIS/UMEP and if so, did it work? As stated in the manual, the cdsapi can be very slow.

[CsillaVG]

Hi Fredrik,

I have no problem downloading ERA5 data outside of QGIS/UMEP. As stated, I've received (downloaded) all the 12 ***sfc.nc files of the requested year. This configuration goes furthest with execution, before running into an error.

[biglimp]

Then it might be a supy issue. @sunt05 ?

[sunt05]

Hello @CsillaVG

This one seems to be related to xarray - did you install any netcdf backend as requested by xarray?

ValueError: did not find a match in any of xarray's currently installed IO backends ['netcdf4', 'scipy']. Consider explicitly selecting one of the installed engines via the ``engine`` parameter, or installing additional IO dependencies, see: https://docs.xarray.dev/en/stable/getting-started-guide/installing.html https://docs.xarray.dev/en/stable/user-guide/io.html

[CsillaVG]

I have double checked the dependencies and they all seem ok. I also did some digging, and it turns out that this value error message is also given when a file is not found or empty. I am no expert of .nc, so I downloaded Panoply and tried to open with it. It too can't open the file and gives me an error. This is true for this 1 month-long test that I did, and my initial attempt for a full year (above). I found a few old .nc files that I've downloaded with the ERA5 downloader in 2022. They work fine with Panoply. Attached is the one-month-long test data with opening issues for reference. test202001.zip

[sunt05]

Have you tried removing these files and downloading them again?

[CsillaVG]

The examples attached were from today, after checking and updating dependencies, but I can give another try.

[sunt05]

This appears to be an upstream issue with the CDS data distribution. The nc files in your archive seem to be in zip format. Rename them as zip files and unzip them to access the nc files.

These can then be properly handled by ncetcf packages.

[paolodep36]

I am having the same problem, is there any fix we can do in the _era5.py function? when downloading seems that the file is a fake nc, after renaming .zip and unzipping i will have the same 2 files -accum -instant. is there a workaround?