objectiv-analytics icon indicating copy to clipboard operation
objectiv-analytics copied to clipboard

Support new datastore: Athena

Open thijs-obj opened this issue 3 years ago • 0 comments

Step one: Bach basics

  • [x] Test setup Bach:
    • Create test infrastructure
    • Add --athena option for running tests
    • Add mark to annotate tests that support new datastore
  • [x] Make first test work:
    • make DataFrame.from_pandas() - required by most functional tests to get an initial DataFrame with test data
    • Make super basic test work: test_df.py::test_basic()
  • [x] Make tests for new datastore run in CI automatically
  • [ ] Make all Series work with new datastore
    • [x] Int
    • [x] Float
    • [x] Boolean
    • [x] Uuid
    • [x] String
    • [x] Json
    • [x] Date
    • [x] DateTime
    • [x] Time
    • [x] TimeDelta
    • [ ] Possibly more
  • [ ] Check which tests work out of the box, and annotate them to always run for the new datastore

Step two: Modelhub and more Bach

  • [ ] Test setup Modelhub
  • [ ] Port DataPipeline to work with new datastore
    • [ ] Make production data available
    • [ ] Write platform specific DataPipeline parts
  • [ ] Port custom Modelhub types
  • [ ] Check which Modelhub tests work out of the box, and annotate them to always run for the new datastore
  • [ ] Check what works in Notebooks out of the box

Step three

Work on anything that doesn't yet work in Bach/Modelhub, prioritize based on what is needed to make Modelhub and notebooks work

thijs-obj avatar Aug 09 '22 15:08 thijs-obj