databricks
[DAB] Add support for requirements libraries in Job Tasks
Changes
While experimenting with DAB I discovered that requirements libraries are being ignored.
One thing worth mentioning is that bundle validate runs successfully, but bundle deploy fails. This PR only covers the second part.
Tests
Added a unit test
@andrewnester Thanks for the feedback. I'll try to work on this in the coming days.
Hey @andrewnester I updated the PR (also rebased onto latest main), please check if this goes into the right direction.
I'm not very familiar with the codebase, so I used the existing source and test code as orientation. However, I'm not sure how to write a unit test for translate_paths_jobs.
@andrewnester Thanks for the extensive feedback. I adapted the PR accordingly. I was also able to add a test in translate_paths_test
Hey @andrewnester
Thank you very much! As mentioned, as I'm new to this codebase, I really appreciate your guidance! It's definitely a quite interesting task to get familiar with DAB. 😉
I reverted the changes regarding artifacts (as a requirements.txt is not seen as an artifact).
Please check if it now makes sense to you
Confirmed manually that requirements.txt is uploaded and used correctly on DBR 15.1+
