flytekit Add support cache for dynamic spec

Tracking issue

Related to #5543

Why are the changes needed?

We need to pass execution mode into TaskTemplate Metadata for Flyte Propeller to tell whether the node is a Dynamic node.

What changes were proposed in this pull request?

Pass execution mode into TaskTemplate Metadata while registering workflows.

Check all the applicable boxes

[X] All new and existing tests passed.
[X] All commits are signed-off.

Related PRs

#6372

Summary by Bito

This PR adds execution_mode support to determine task behavior in dynamic caching scenarios, implementing critical improvements for caching in dynamic specifications. It enhances core modules with properties and conversion functions, ensures default values when execution_mode is not provided, and refines type logic to manage circular dependencies while aligning production code with tests for consolidated dynamic workflow support.

Unit tests added: True

Estimated effort to review (1-5, lower is better): 2

Mar 25 '25 05:03 popojk

Thank you for opening this pull request! 🙌

These tips will help get your PR across the finish line:

Most of the repos have a PR template; if not, fill it out to the best of your knowledge.
Sign off your commits (Reference: DCO Guide).

Mar 25 '25 05:03 welcome[bot]

Code Review Agent Run #0d112d

Actionable Suggestions - 4

tests/flytekit/common/parameterizers.py - 1
- Missing comma in product function parameters · Line 142-142
flytekit/core/base_task.py - 1
- Undefined name in type annotation · Line 143-143
flytekit/core/python_function_task.py - 1
- Missing fallback for unknown enum values · Line 144-144
flytekit/models/task.py - 1
- Missing property getter for execution_mode attribute · Line 228-231

Review Details

Files reviewed - 7 · Commit Range: 58ddff9..0bce2cf
- flytekit/core/base_task.py
- flytekit/core/python_function_task.py
- flytekit/core/task.py
- flytekit/models/task.py
- tests/flytekit/common/parameterizers.py
- tests/flytekit/unit/core/test_python_function_task.py
- tests/flytekit/unit/models/test_tasks.py
Files skipped - 0
Tools
- Whispers (Secret Scanner) - ✔︎ Successful
- Detect-secrets (Secret Scanner) - ✔︎ Successful
- MyPy (Static Code Analysis) - ✔︎ Successful
- Astral Ruff (Static Code Analysis) - ✔︎ Successful

Bito Usage Guide

Commands

Type the following command in the pull request comment and save the comment.

/review - Manually triggers a full AI review.

Refer to the documentation for additional commands.

Configuration

This repository uses code_review_bito You can customize the agent settings here or contact your Bito workspace admin at [email protected].

Documentation & Help

AI Code Review powered by

Mar 25 '25 05:03 flyte-bot

Changelist by Bito

This pull request implements the following key changes.

Key Change Files Impacted

Key Change	Files Impacted
Feature Improvement - Enhanced Task Execution Mode Handling	- `base_task.py` - Introduced 'execution_mode' parameter and default initialization with future annotations import. - `python_function_task.py` - Added conversion methods to map between Python execution behavior and flyteidl execution mode. - `task.py` - Passed execution_mode during task registration to support dynamic workflow behavior. - `task.py` - Integrated execution_mode into TaskMetadata with property, default setting, and conversion logic.
Testing - Updated Test Cases for Execution Mode	- `parameterizers.py` - Modified product iteration to include execution_mode and added related import. - `test_python_function_task.py` - Included tasks_pb2 import to support new execution mode conversion tests. - `test_tasks.py` - Adjusted tuple parameters and added assertions to verify the correct execution_mode behavior.

Feature Improvement - Enhanced Task Execution Mode Handling

- base_task.py - Introduced 'execution_mode' parameter and default initialization with future annotations import.

- python_function_task.py - Added conversion methods to map between Python execution behavior and flyteidl execution mode.

- task.py - Passed execution_mode during task registration to support dynamic workflow behavior.

- task.py - Integrated execution_mode into TaskMetadata with property, default setting, and conversion logic.

Testing - Updated Test Cases for Execution Mode

- parameterizers.py - Modified product iteration to include execution_mode and added related import.

- test_python_function_task.py - Included tasks_pb2 import to support new execution mode conversion tests.

- test_tasks.py - Adjusted tuple parameters and added assertions to verify the correct execution_mode behavior.

Mar 25 '25 05:03 flyte-bot

Codecov Report

Attention: Patch coverage is 45.00000% with 11 lines in your changes missing coverage. Please review.

Project coverage is 35.66%. Comparing base (5503ee5) to head (b206d99).

Files with missing lines	Patch %	Lines
flytekit/models/task.py	33.33%	6 Missing :warning:
flytekit/core/base_task.py	40.00%	3 Missing :warning:
flytekit/core/python_function_task.py	66.66%	2 Missing :warning:

:exclamation: There is a different number of reports uploaded between BASE (5503ee5) and HEAD (b206d99). Click for more details.

HEAD has 40 uploads less than BASE

Flag BASE (5503ee5) HEAD (b206d99)

42 2

Flag	BASE (5503ee5)	HEAD (b206d99)
	42	2

Additional details and impacted files

@@             Coverage Diff             @@
##           master    #3209       +/-   ##
===========================================
- Coverage   81.71%   35.66%   -46.06%     
===========================================
  Files         335      214      -121     
  Lines       27415    22368     -5047     
  Branches     2920     2922        +2     
===========================================
- Hits        22403     7978    -14425     
- Misses       4177    14271    +10094     
+ Partials      835      119      -716

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

:rocket: New features to boost your workflow:

:snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Mar 25 '25 08:03 codecov[bot]

Code Review Agent Run #032d6a

Actionable Suggestions - 0

Review Details

Files reviewed - 3 · Commit Range: 0bce2cf..b206d99
- flytekit/core/base_task.py
- flytekit/core/python_function_task.py
- tests/flytekit/unit/core/test_python_function_task.py
Files skipped - 0
Tools
- Whispers (Secret Scanner) - ✔︎ Successful
- Detect-secrets (Secret Scanner) - ✔︎ Successful
- MyPy (Static Code Analysis) - ✔︎ Successful
- Astral Ruff (Static Code Analysis) - ✔︎ Successful

Bito Usage Guide

Commands

Type the following command in the pull request comment and save the comment.

/review - Manually triggers a full AI review.

Refer to the documentation for additional commands.

Configuration

This repository uses code_review_bito You can customize the agent settings here or contact your Bito workspace admin at [email protected].

Documentation & Help

AI Code Review powered by

Mar 25 '25 14:03 flyte-bot