dstack icon indicating copy to clipboard operation
dstack copied to clipboard

doc: clarify behavior when running workflow without tag on dependency

Open kengz opened this issue 3 years ago • 0 comments

The following behavior of workflow run needs to be illustrated in the doc.

For a workflow file (note the lack of tag :latest in the dependency:

workflows:
  - name: download
    provider: python
    requirements: requirements.txt
    script: mnist/prepare_data.py
    artifacts:
      - data

  - name: train
    provider: python
    requirements: requirements.txt
    script: mnist/train.py
    depends-on:
      - download
    artifacts:
      - model
    resources:
      gpu: 1

running the sequence of commands:

dstack run download
# wait for completion, then run
dstack run train

will yield this:

the first command creates a run warm-implala-1. The second command doesn't use warm-implala-1, instead it runs from the start. Screen Shot 2022-06-19 at 10 40 43 PM

expectation/improvement

  • first expectation was train would use whichever the latest download was, regardless of tag being specified.
  • the user flow to: run download -> wait for completion -> tag latest in UI -> come back and run train is quick a lot of manual steps and friction.
  • suggestion: when no tag is specified in dependency, do run from scratch, that's fine. But when a latest tag is specified, automatically use the latest one. This means that latest has to be a reserved tag that always points to the latest run of a workflow - this is also semantically consistent. If user wants to manually use special tag like foo_1 then specify it, they can follow the usual tag usage as the doc currently explains.

kengz avatar Jun 20 '22 02:06 kengz