[Feature][all] We are ready to contribute multiple features

[liukunyuan]

Search before asking

• [X] I had searched in the issues and found no similar feature requirement.

Description

We have these feature within our company and hope to contribute them. What feature communities require?

1. Worker and master grayscale function based on traffic
2. Optimization of worker and master load protection strategies (optimizing remaining physical memory and CPU algorithms on the machine) to address the issue of worker downtime
3. Force skip dependency detection during complement
4. Heartbeat reporting from master and worker to detect disconnection issues
5. Write the Yarn appid into the task_instance table during task execution to facilitate worker fault tolerance (solving the problem of worker machine downtime and inability to tolerate faults)
6. Workflow migration project interface
7. Recommend users to set the cron for the workflow (query the completion time of all upstream workflows in the last 7 days and then recommend the start time of this workflow)
8. After the task timeout fails, it is necessary to kill the Yarn task
9. Change workflow instances and task instances to single table queries to solve the problem of slow queries
10. Provide a batch recovery failure interface to solve the problem of slow recovery of workflow instances caused by cluster or MySQL issues
11. Add feature control tables to MySQL. All important functions require parameter control to turn on and off

Are you willing to submit a PR?

• [X] Yes I am willing to submit a PR!

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

[wangxj3]

What causes the fourth problem? And I have some suggest: 1.The fifth problem may cause task loss. A data processing task may contain more then one yarn job. If it fails between yarn tasks, this may cause the fault-tolerant task to succeed but miss some yarn task submissions. 2.Can the eighth one be configured? The easy failure of some long tasks may lead to a waste of complement resources.

[ruanwenjun]

Please Create DSIP to describe why you want to do this change, it's hard to know what you want to do by a simple description, and some of these should already been supported.

[liukunyuan]

What causes the fourth problem? And I have some suggest: 1.The fifth problem may cause task loss. A data processing task may contain more then one yarn job. If it fails between yarn tasks, this may cause the fault-tolerant task to succeed but miss some yarn task submissions. 2.Can the eighth one be configured? The easy failure of some long tasks may lead to a waste of complement resources.

1. What causes the fourth problem? Mysql network exception for a few minutes
2. A data processing task may contain more then one yarn job。 It will write multiple appids to the task_instance table

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.

[github-actions[bot]]

This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future.