[BUG] Eager workflows should not always use the latest pipeline version

[ai-rnatour]

Describe the bug

Currently, eager workflows always invoke the latest version of whatever task or workflow is being executed. This presents a number of problems:

• When multiple developers are iterating on a pipeline in the development stage, their eager workflows might invoke tasks or workflows from other developers
• Eager workflows prevent reproducibility guarantees for production pipelines

For our use case, pipeline version pinning is important and this prevent us from taking full advantage of eager workflows.

Expected behavior

Eager workflows should accept a version field in their decorator to allow users to pin specific pipeline versions. If no version is provided, the eager workflow should default to invoking the same pipeline version as the currently executing task. This could be accomplished by examining the FLYTE_INTERNAL_VERSION environment variable.

Additional context to reproduce

No response

Screenshots

No response

Are you sure this issue hasn't been raised already?

• [X] Yes

Have you read the Code of Conduct?

• [X] Yes

[runllm[bot]]

Eager workflows in Flyte currently always invoke the latest version of tasks or workflows, which can lead to issues with reproducibility and conflicts during development. According to the Flyte documentation, eager workflows execute the latest version of tasks and workflows registered under the same project and domain.

To address this, you propose adding a version field to the eager workflow decorator to pin specific pipeline versions. This would ensure that eager workflows invoke the same version as the currently executing task, potentially by using the FLYTE_INTERNAL_VERSION environment variable.

Would you like more detail on how to implement this or any other specific aspect?

Sources:

• Flyte documentation

---

This is an AI-generated response and your feedback is appreciated! Please leave a 👍 if this is helpful and 👎 if it is not.

Mention @runllm to ask follow-up questions. See runllm.com for more!

[github-actions[bot]]

"Hello 👋, this issue has been inactive for over 90 days. To help maintain a clean and focused backlog, we'll be marking this issue as stale and will close the issue if we detect no activity in the next 7 days. Thank you for your contribution and understanding! 🙏"

[github-actions[bot]]

Hello 👋, this issue has been inactive for over 90 days and hasn't received any updates since it was marked as stale. We'll be closing this issue for now, but if you believe this issue is still relevant, please feel free to reopen it. Thank you for your contribution and understanding! 🙏