flytekit-dbt plugin

[ariefrahmansyah]

• Supports dbt run and dbt test tasks
• The plugin includes integration test that need local PostgreSQL database

Signed-off-by: ariefrahmansyah [email protected]

TL;DR

Add flytekit-dbt plugin. Fixes https://github.com/flyteorg/flyte/issues/2202

Type

• [ ] Bug Fix
• [ ] Feature
• [X] Plugin

Are all requirements met?

• [ ] Code completed
• [ ] Smoke tested
• [X] Unit tests added
• [ ] Code documentation added
• [ ] Any pending items have an associated Issue

Complete description

For integration testing, flytekit-dbt needs a running PostgreSQL database. To achieve that, it assumes that the machine running the test already have PostgreSQL installed. Then, the pytest.fixtures executes setup_db.sh to create dbttest database. This database is used for dbt tasks to execute the transformation of dbt's jaffle shop example.

Tracking Issue

https://github.com/flyteorg/flyte/issues/2202

Follow-up issue

NA

[welcome[bot]]

Thank you for opening this pull request! 🙌

These tips will help get your PR across the finish line:

• Most of the repos have a PR template; if not, fill it out to the best of your knowledge.
• Sign off your commits (Reference: DCO Guide).

[codecov[bot]]

Codecov Report

Merging #886 (745fbb4) into master (2946952) will not change coverage. The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #886   +/-   ##
=======================================
  Coverage   86.47%   86.47%           
=======================================
  Files         230      230           
  Lines       22078    22078           
  Branches     2459     2459           
=======================================
  Hits        19093    19093           
  Misses       2578     2578           
  Partials      407      407           

---

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 2946952...745fbb4. Read the comment docs.

[kumare3]

thank you so much!

[kumare3]

@wild-endeavor - this could have been implemented by shell, but have a built in plugin helps a lot

[kumare3]

cc @ariefrahmansyah thank you for the PR, do you need some help from anyone in the community? I think this feature would really help many folks

[kumare3]

@ariefrahmansyah hows it going with this?

[esadler-hbo]

@ariefrahmansyah this might be useful for a project that I am working in. I'm about to do a write a spike and if dbt is accepted then I could help with this plugin

[eapolinario]

Subsumed by https://github.com/flyteorg/flytekit/pull/1150.