MLOpsPython icon indicating copy to clipboard operation
MLOpsPython copied to clipboard

Improve local developer experience when submitting pipelines

Open xinyi-joffre opened this issue 6 years ago • 1 comments

Since most developers using this repo will iterate on the scripts for the pipeline steps (to update their models or evaluation), it would be great to make the process of submitting those changes smoother.

Current experience:

In the current repo, these 4 steps need to be followed each time a small change in made in one of the steps (train, register, etc.). This is in part due to the requirement to have a unique build id for each pipeline.

  • Make changes in a file (such as train.py)
  • Update build id in .env
  • Run python ml_service/pipelines/build_train_pipeline.py
  • Run python ml_service/pipelines/run_train_pipeline.py

Desired experience:

Since there is no "build_id" when submitting locally, it would be best to allow the user to use a junk or default value.

  • Make changes in a file (such as train.py)
  • Run one script to build and submit the pipeline to run.

xinyi-joffre avatar Jan 17 '20 21:01 xinyi-joffre

Here's the bash command line I use

export BUILD_BUILDID=$(uuidgen); python ml_service/pipelines/build_train_pipeline.py && python ml_service/pipelines/run_train_pipeline.py

All in one line. Not user friendly, but it does 2-4 in this list in one go...

algattik avatar Jan 18 '20 21:01 algattik