fsspec
added the reinflate api
Is there a better name for this function?
You mean "reinflate"? Maybe if you write a 2-sentence description, the name will present itself :)
@martindurant what should be the module name? The module should be included in a sub-package like kerchunk.grib.<module> or directly in the kerchunk.<module name>?
I suggested _grib_idx above. Making kerchunk.grib into a package sounds like a bad idea.
Comments on naming (not my strong suit)... the Camus team had suggested "outbasing" as the name for this general approach of ingesting an index of chunks to a database and constructing the dataset/tree on the fly.
The "reinflate" method in particular builds the refspec dictionary for a zarr/xarray tree representing a set of grib data. The inputs are the static metadata for the refspec (everything but the chunks for variables with a time dimension), a set of chunk indexes for specific variables (usually stored in a DB), the aggregation type, and the set of axes that define how to arrange those chunks.
I wanted to be general, supporting all the FMRC slices which are represented in the AggregationType enum. The api allows passing a list of axes such that you can express a set of runtimes, a set of valid times, a set of horizons, or the 'best available' as of a certain time.
I hope there is some simpler expression of these concepts that might use this method as a backend, but I suspect you would have to make assumptions about the backend database/schema to wrap it up nicely.
- This code should be in _grib_idx.py, as discussed
- There should be tests
- There needs to be far more documentation on how to use the code (again), else no one will ever touch these functions.
Thank you for getting the process started. I will plan to make progress this month, but it will be a nights and weekends effort. If that is not progressing by the end of October we can regroup.
@emfdavid I was trying to figure out where the 2024 GSoC work stands, and it looks like this is the blocker? Is that correct?
