Proposal: Add in-memory repository to improve workflow api perf

[xinlmain]

Self Checks

• [x] I have searched for existing issues search for existing issues, including closed ones.
• [x] I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
• [x] [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:)
• [x] Please do not modify this template :) and fill in all the required fields.

1. Is this request related to a challenge you're experiencing? Tell me about your story.

Core Objective

The default workflow-run api is very slow on simple tasks, making it unacceptable for real time interactions. The api calls spend most of the time interacting with DB. In a scenario where the api is called by another service, there is not much sense to save each call into DB.

Approach to Achieve the Objective

Add an in-memory repository so the execution won't interact with DB too much.

Implementation Plan

1. Add a new WorkflowNodeExecutionRepository implementation with pure in-memory underneath. This is viable as workflows are each executed in 1 process only.
2. Add a toggle to switch the repositories.

Expected Outcomes

Improved performance: Workflows api can become faster, and thus enables deploying a dedicated api service for serving external requests.

Tradeoffs

Not able to see execution detail in application logs from the web page.

2. Additional context or comments

none.

3. Can you help us with this feature?

• [x] I am interested in contributing to this feature.

[jzymx50]

I found for each api call, Dify will update the last updated time for api token in db, making high concurrent requests very slow. Could you also help to improve that? The function name is validate_and_get_api_token @xinlmain

[laipz8200]

Hi @xinlmain.

Thank you for your patience. In #21458, I have made two repositories configurable. Could you please take a look and consider implementing this feature based on the new structure? I believe that migrating persistence operations to Celery tasks is a better approach than abandoning persistence. I would appreciate hearing your thoughts.