Pause a transform

[oleiman]

Introduce an Admin API endpoint for pausing and unpausing transforms

TODO:

• [ ] Probably split admin endpoint into two
• [ ] rpk experience

Closes CORE-1653

Backports Required

• [ ] none - not a bug fix
• [ ] none - this is a backport
• [ ] none - issue does not exist in previous branches
• [ ] none - papercut/not impactful enough to backport
• [ ] v24.1.x
• [ ] v23.3.x
• [ ] v23.2.x

Release Notes

Improvements

• Adds the ability to pause a deployed data transform w/o removing it from the system

[oleiman]

/dt

[rockwotj]

Probably split admin endpoint into two

An alternative is to have an endpoint like: PATCH /transform/{name} where you can patch the transform object. Parameters would include environment variables, is paused and maybe some others? @mo-redpanda this patch endpoint could support updating environment variables only for console based transforms.

[oleiman]

Probably split admin endpoint into two

An alternative is to have an endpoint like: PATCH /transform/{name} where you can patch the transform object. Parameters would include environment variables, is paused and maybe some others? @mo-redpanda this patch endpoint could support updating environment variables only for console based transforms.

I like that idea. Unfortunately our seastar http mess doesn't support PATCH, so maybe PUT /transform/{name}/meta or something like that?

[oleiman]

(or finally go in and see how much work it would be to add PATCH support!)

[rockwotj]

I am not a REST purest so fine with PUT

[oleiman]

force push contents:

• patch metadata instead of pause outright (allows update of transform env)
• comments, minor CR changes

[oleiman]

force push fix swagger

[oleiman]

force push contents:

• rpk transform pause
• rpk transform start

[oleiman]

force push contents:

• various comments/route descriptions
• pause/start -> suspend/resume (tentative, pending input from devex & product)

[oleiman]

force push - rebase dev

[oleiman]

force push contents:

• address CR comments (use new seastar path param function, add a patch env test case)
• switch over from suspend/resume to pause/resume per discussion on slack and RFC

[vbotbuildovich]

new failures in https://buildkite.com/redpanda/redpanda/builds/48799#018f555d-0381-4daa-93bd-76655c05d6ef:

"rptest.tests.e2e_shadow_indexing_test.EndToEndThrottlingTest.test_throttling.cloud_storage_type=CloudStorageType.ABS"

new failures in https://buildkite.com/redpanda/redpanda/builds/48799#018f555d-0384-467b-93b7-3d71dde0472e:

"rptest.tests.e2e_shadow_indexing_test.EndToEndThrottlingTest.test_throttling.cloud_storage_type=CloudStorageType.S3"

new failures in https://buildkite.com/redpanda/redpanda/builds/48799#018f5561-efd2-4710-9ad5-98e2f456386f:

"rptest.tests.e2e_shadow_indexing_test.EndToEndThrottlingTest.test_throttling.cloud_storage_type=CloudStorageType.S3"

new failures in https://buildkite.com/redpanda/redpanda/builds/48799#018f5561-efcf-4409-92ec-9f75ddb4909b:

"rptest.tests.e2e_shadow_indexing_test.EndToEndThrottlingTest.test_throttling.cloud_storage_type=CloudStorageType.ABS"

new failures in https://buildkite.com/redpanda/redpanda/builds/48950#018f64a1-ace1-431c-94a4-482c1330ebe4:

"rptest.tests.data_transforms_test.DataTransformsTest.test_patch.use_rpk=True"

[vbotbuildovich]

ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/48799#018f555d-037c-4bb8-ae77-616030013190

ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/49138#018f7add-5c02-4a11-a430-f039f0ec9056

[oleiman]

force push long form rpk descriptions

[oleiman]

CI Failure:

• test_archival_service_rpfixture, which I believe is a known issue, unrelated to this change

[oleiman]

/ci-repeat 1

[oleiman]

force push contents:

• fix typo in rpk.py and harden DataTransformsTest.test_patch
• fix swagger formatting, commit history

[oleiman]

CI Failure is test_raft_rpunit

[oleiman]

force push adds json::validator for patch request