Enhance Workflow to Support Checkpoint Restart from Failure Points

[Woo0ood]

Self Checks

• [X] I have searched for existing issues search for existing issues, including closed ones.
• [X] I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
• [X] [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:）
• [X] Please do not modify this template :) and fill in all the required fields.

1. Is this request related to a challenge you're experiencing? Tell me about your story.

Currently, when a long-running workflow encounters an error and terminates unexpectedly, restarting the workflow requires re-executing all the steps from the beginning, even those that previously completed successfully. This results in a significant amount of time being spent on re-executing known results, which is inefficient and time-consuming.

2. Additional context or comments

We are thrilled with the recent improvements and optimizations Dify has made to handle errors during workflow execution. These enhancements have significantly improved the robustness and reliability of our workflows. However, there is still room for further optimization to enhance the user experience and efficiency.

Proposed Enhancement:

To address this issue, we propose implementing a checkpoint restart mechanism for workflows. This feature would allow workflows to resume execution from the point of failure, using the results of the previously completed steps as inputs and outputs. By doing so, the workflow can continue from the error node onward, saving time and resources.

Benefits:

• Reduced execution time by avoiding redundant processing of previously successful steps.
• Improved efficiency and resource utilization.
• Enhanced user experience by providing a more resilient and fault-tolerant workflow execution process.

This feature will greatly benefit users who rely on long-running workflows, such as video generation, image generation, and large document processing.

We plan to initiate this enhancement and ultimately contribute the code to the Dify community. We hope you will consider our proposal and look forward to your feedback on our future pull requests.🎉

3. Can you help us with this feature?

• [X] I am interested in contributing to this feature.

[Yevanchen]

We are considering designing this capability

[Woo0ood]

We are considering designing this capability

Could you please let me know where this feature is on the roadmap and its estimated implementation time? If this feature is currently a mid- or long-term goal on the official roadmap, we would like to start developing it ourselves and will share the final result with the community.😆

[crazywoola]

@Woo0ood If you are interested in it, please add my wechat: crazyphage.

[dosubot[bot]]

Hi, @Woo0ood. I'm Dosu, and I'm helping the Dify team manage their backlog. I'm marking this issue as stale.

Issue Summary:

• The issue proposes enhancing the workflow system to support checkpoint restarts.
• @Yevanchen mentioned that the team is considering this capability.
• You expressed interest in developing the feature if it's a mid- or long-term goal.
• @crazywoola invited you to connect via WeChat for further discussion.

Next Steps:

• Please let us know if this issue is still relevant to the latest version of the Dify repository by commenting here.
• If there is no further activity, this issue will be automatically closed in 15 days.

Thank you for your understanding and contribution!

[yongli-abc]

Is this being planned now? It becomes significant pain point when the scale of the workflow increases.