datatree
datatree copied to clipboard
Fix reading from fsspec s3
Xarray does not like reading the same file twice from fsspec. See https://github.com/pydata/xarray/issues/6813
The following currently fails:
import fsspec
import datatree as dt
fs = fsspec.filesystem('s3', anon=True)
fp = 'noaa-goes16/ABI-L1b-RadF/2019/079/14/OR_ABI-L1b-RadF-M3C03_G16_s20190791400366_e20190791411133_c20190791411180.nc'
data = fs.open(fp)
dt.open_datatree(data, engine='h5netcdf', chunks={})
with
Traceback (most recent call last):
File "//example.py", line 24, in <module>
run_test(dt.open_datatree, fs, fp)
File "//example.py", line 15, in run_test
func(data, engine='h5netcdf', chunks={})
File "/opt/conda/lib/python3.10/site-packages/datatree/io.py", line 58, in open_datatree
return _open_datatree_netcdf(filename_or_obj, engine=engine, **kwargs)
File "/opt/conda/lib/python3.10/site-packages/datatree/io.py", line 67, in _open_datatree_netcdf
ds = open_dataset(filename, **kwargs)
File "/opt/conda/lib/python3.10/site-packages/xarray/backends/api.py", line 531, in open_dataset
backend_ds = backend.open_dataset(
File "/opt/conda/lib/python3.10/site-packages/xarray/backends/h5netcdf_.py", line 389, in open_dataset
store = H5NetCDFStore.open(
File "/opt/conda/lib/python3.10/site-packages/xarray/backends/h5netcdf_.py", line 157, in open
magic_number = read_magic_number_from_file(filename)
File "/opt/conda/lib/python3.10/site-packages/xarray/core/utils.py", line 645, in read_magic_number_from_file
raise ValueError(
ValueError: cannot guess the engine, file-like object read/write pointer not at the start of the file, please close and reopen, or use a context manager
My change fixes that.