ksml
ksml copied to clipboard
Implement Job deployment mode for data generator
The standard deployment for KSML is as a Kubernetes StatefulSet. This provides horizontal scaling options with predictable naming and volume claims, which is great for pipeline processing. The Kubernetes cluster will restart completed applications to meet the replica requirement.
But this is unwanted if KSML is configured as a data generator with a fixed number of produce calls, or other ending condition. By using a Kubernetes Job the data generation will run only once, and keep the pod in its final state to allow deployment and log inspection.
This PR adds the deploymentMode configuration option to deploy the application as either a StatefulSet or a Job.
Known limitations:
- Job deployments can not be updated, but must be removed and recreated. This is a standard Kubernetes Job limitation
- Jobs can not scale horizontally