dolphinscheduler icon indicating copy to clipboard operation
dolphinscheduler copied to clipboard

[DSIP-16][Task] Support stream task

Open caishunfeng opened this issue 3 years ago • 3 comments

Search before asking

  • [X] I had searched in the issues and found no similar feature requirement.

Purpose

At present, DS is the scheduling of offline tasks, but as the business requires more and more timely addition of task results, there are more and more scenarios of stream-batch integration, so it is necessary to support real-time tasks.

Use case

  • define stream task, such as flink-stream.
  • stream task action in ds, such as start, stop, savepoint, monitor etc.
  • stream task in DAG, we can do impact analysis based on the relationship between tasks in the DAG, which is a proactive approach.

Related items

  • [x] Support stream task definition and instance action #11223
  • [ ] Use flink SDK replace the shell cli, make more standard and easy to develop #11440
  • [ ] Add flink cluster manager
  • [ ] Support flink sql
  • [ ] Add flink metircs, such as checkpoints, received, sent etc.
  • [ ] Support other stream task.
  • [ ] Event-based scheduling.
  • [ ] checkpoint/savepoint mange, task continuing, failover

caishunfeng avatar Aug 08 '22 10:08 caishunfeng

Thank you for your feedback, we have received your issue, Please wait patiently for a reply.

  • In order for us to understand your request as soon as possible, please provide detailed information、version or pictures.
  • If you haven't received a reply for a long time, you can join our slack and send your question to channel #troubleshooting

github-actions[bot] avatar Aug 08 '22 10:08 github-actions[bot]