datatree icon indicating copy to clipboard operation
datatree copied to clipboard

Fix reading from fsspec s3

Open wroberts4 opened this issue 3 years ago • 0 comments

Xarray does not like reading the same file twice from fsspec. See https://github.com/pydata/xarray/issues/6813

The following currently fails:

import fsspec
import datatree as dt
fs = fsspec.filesystem('s3', anon=True)
fp = 'noaa-goes16/ABI-L1b-RadF/2019/079/14/OR_ABI-L1b-RadF-M3C03_G16_s20190791400366_e20190791411133_c20190791411180.nc'
data = fs.open(fp)
dt.open_datatree(data, engine='h5netcdf', chunks={})

with

Traceback (most recent call last):
  File "//example.py", line 24, in <module>
    run_test(dt.open_datatree, fs, fp)
  File "//example.py", line 15, in run_test
    func(data, engine='h5netcdf', chunks={})
  File "/opt/conda/lib/python3.10/site-packages/datatree/io.py", line 58, in open_datatree
    return _open_datatree_netcdf(filename_or_obj, engine=engine, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/datatree/io.py", line 67, in _open_datatree_netcdf
    ds = open_dataset(filename, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/xarray/backends/api.py", line 531, in open_dataset
    backend_ds = backend.open_dataset(
  File "/opt/conda/lib/python3.10/site-packages/xarray/backends/h5netcdf_.py", line 389, in open_dataset
    store = H5NetCDFStore.open(
  File "/opt/conda/lib/python3.10/site-packages/xarray/backends/h5netcdf_.py", line 157, in open
    magic_number = read_magic_number_from_file(filename)
  File "/opt/conda/lib/python3.10/site-packages/xarray/core/utils.py", line 645, in read_magic_number_from_file
    raise ValueError(
ValueError: cannot guess the engine, file-like object read/write pointer not at the start of the file, please close and reopen, or use a context manager

My change fixes that.

wroberts4 avatar Jul 19 '22 22:07 wroberts4