MUSE_OS icon indicating copy to clipboard operation
MUSE_OS copied to clipboard

First pass at duckdb data interface

Open cc-a opened this issue 1 year ago • 1 comments

Description

A first mocked up version of what a revised data input layer may look like using standalone (i.e. no sqlalchemy) duckdb. So far implemented:

  • reader functions for the region, commodity definition and commodity demand tables that populate the their data into duckdb whilst also returning a numpy array representation.
  • calculate_global_commodities function that takes the data from the commodity definition table and transforms it into the xarray representation required by Muse internally.
  • tests for each function implemented plus additional tests addressing any constraints applied to database columns.

Mostly pushing this for visibility at this point as @tsmbland is planning to expand this to cover the remaining database tables and xarray data structures.

Fixes # (issue)

Type of change

Please add a line in the relevant section of CHANGELOG.md to document the change (include PR #) - note reverse order of PR #s.

  • [ ] New feature (non-breaking change which adds functionality)
  • [ ] Optimization (non-breaking, back-end change that speeds up the code)
  • [ ] Bug fix (non-breaking change which fixes an issue)
  • [ ] Breaking change (whatever its nature)

Key checklist

  • [ ] All tests pass: $ python -m pytest
  • [ ] The documentation builds and looks OK: $ python -m sphinx -b html docs docs/build

Further checks

  • [ ] Code is commented, particularly in hard-to-understand areas
  • [ ] Tests added that prove fix is effective or that feature works

cc-a avatar Jun 27 '24 16:06 cc-a

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 71.30%. Comparing base (f5bec10) to head (ae61de8).

Additional details and impacted files
@@               Coverage Diff               @@
##           new-data-input     #379   +/-   ##
===============================================
  Coverage           71.30%   71.30%           
===============================================
  Files                  44       44           
  Lines                5890     5890           
  Branches             1155     1155           
===============================================
  Hits                 4200     4200           
  Misses               1369     1369           
  Partials              321      321           

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

codecov[bot] avatar Jun 27 '24 18:06 codecov[bot]