First pass at duckdb data interface
Description
A first mocked up version of what a revised data input layer may look like using standalone (i.e. no sqlalchemy) duckdb. So far implemented:
- reader functions for the region, commodity definition and commodity demand tables that populate the their data into duckdb whilst also returning a numpy array representation.
-
calculate_global_commoditiesfunction that takes the data from the commodity definition table and transforms it into the xarray representation required by Muse internally. - tests for each function implemented plus additional tests addressing any constraints applied to database columns.
Mostly pushing this for visibility at this point as @tsmbland is planning to expand this to cover the remaining database tables and xarray data structures.
Fixes # (issue)
Type of change
Please add a line in the relevant section of CHANGELOG.md to document the change (include PR #) - note reverse order of PR #s.
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Optimization (non-breaking, back-end change that speeds up the code)
- [ ] Bug fix (non-breaking change which fixes an issue)
- [ ] Breaking change (whatever its nature)
Key checklist
- [ ] All tests pass:
$ python -m pytest - [ ] The documentation builds and looks OK:
$ python -m sphinx -b html docs docs/build
Further checks
- [ ] Code is commented, particularly in hard-to-understand areas
- [ ] Tests added that prove fix is effective or that feature works
Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:
Project coverage is 71.30%. Comparing base (
f5bec10) to head (ae61de8).
Additional details and impacted files
@@ Coverage Diff @@
## new-data-input #379 +/- ##
===============================================
Coverage 71.30% 71.30%
===============================================
Files 44 44
Lines 5890 5890
Branches 1155 1155
===============================================
Hits 4200 4200
Misses 1369 1369
Partials 321 321
:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.