ksml icon indicating copy to clipboard operation
ksml copied to clipboard

Implement Job deployment mode for data generator

Open richard-axual opened this issue 11 months ago • 0 comments

The standard deployment for KSML is as a Kubernetes StatefulSet. This provides horizontal scaling options with predictable naming and volume claims, which is great for pipeline processing. The Kubernetes cluster will restart completed applications to meet the replica requirement.

But this is unwanted if KSML is configured as a data generator with a fixed number of produce calls, or other ending condition. By using a Kubernetes Job the data generation will run only once, and keep the pod in its final state to allow deployment and log inspection.

This PR adds the deploymentMode configuration option to deploy the application as either a StatefulSet or a Job.

Known limitations:

  • Job deployments can not be updated, but must be removed and recreated. This is a standard Kubernetes Job limitation
  • Jobs can not scale horizontally

richard-axual avatar May 06 '25 11:05 richard-axual