[Improvement][MasterWorker] Self-recovery when master or worker lost connection from registry center

[caishunfeng]

Search before asking

• [X] I had searched in the issues and found no similar feature requirement.

Background

Now if a master or worker lost zk connection, it doesn't disconver itself immediately until checking the dead server list. And when it knows that it was judged to dead serve, it will stop itself, without self-recovery.

Prosoal

When master lost zk connection:

1. update current server state to wait reconnect
2. send server lost connection alert
3. keep quartz working (it will ensure work normally by quartz and db)
4. stop accepting new request
5. stop handling commands and process instances, clear the local running process instances; (it will be take over by other master)
6. wait to reconnect
7. when reconnect successfully, send server recover alert, update server state to normal and recover working

When worker lost zk connection:

1. update current server state to wait reconnect
2. send server lost connection alert
3. kill the running task (it will be task over by master and rerun)
4. stop accepting new request
5. wait to reconnect within a certain time
6. if reconnect timeout, stop itself
7. when reconnect successfully, send server recover alert, update server state to normal and recover working;

Originally posted by @caishunfeng in https://github.com/apache/dolphinscheduler/discussions/6643#discussioncomment-1706255

Related issues

#7004

Are you willing to submit a PR?

• [ ] Yes I am willing to submit a PR!

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

[github-actions[bot]]

Hi:

• Thank you for your feedback, we have received your issue, Please wait patiently for a reply.
• In order for us to understand your request as soon as possible, please provide detailed information、version or pictures.
• If you haven't received a reply for a long time, you can subscribe to the developer's email，Mail subscription steps reference https://dolphinscheduler.apache.org/en-us/community/development/subscribe.html ,Then write the issue URL in the email content and send question to [email protected].

[shangeyao]

please assign to me

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.

[github-actions[bot]]

This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future.