[CORE-2736] Adds output batch compression for Data Transforms

[oleiman]

This PR adds the ability to configure output batch compression on WASM transforms, either at deploy time or post facto, via metadata patch request. Includes rpk experience for same

TODO:

• [x] rpk bits and tests for deploying w/ compression turned on

Backports Required

• [x] none - not a bug fix
• [ ] none - this is a backport
• [ ] none - issue does not exist in previous branches
• [ ] none - papercut/not impactful enough to backport
• [ ] v24.1.x
• [ ] v23.3.x
• [ ] v23.2.x

Release Notes

Improvements

• Add output batch compression for Data Transforms (configurable per deployed transform)

[oleiman]

/ci-repeat 1

[oleiman]

/ci-repeat 1 release skip-units skip-redpanda-build

[vbotbuildovich]

new failures in https://buildkite.com/redpanda/redpanda/builds/49248#018f82f5-d2fc-4bd5-b204-a1774afd79bf:

"rptest.tests.connection_virtualizing_test.TestVirtualConnections.test_handling_invalid_ids"

new failures in https://buildkite.com/redpanda/redpanda/builds/49248#018f82f5-d2ff-46d2-8840-d9e063a709e2:

"rptest.tests.connection_virtualizing_test.TestVirtualConnections.test_no_head_of_line_blocking.different_clusters=False.different_connections=False"

new failures in https://buildkite.com/redpanda/redpanda/builds/50134#019009db-8de9-47d5-a29d-ba3e1ac43ce8:

"rptest.tests.topic_creation_test.TopicRecreateTest.test_topic_recreation_while_producing.workload=ACKS_1.cleanup_policy=compact"
"rptest.tests.e2e_shadow_indexing_test.EndToEndThrottlingTest.test_throttling.cloud_storage_type=CloudStorageType.ABS"

[oleiman]

force push to sync w/ dev after a couple weeks of inactivity

[oleiman]

force push contents:

• remove set_compression command (compression set only at deploy time to avoid confusion)
• fix up some tests to account for above
• remove compression column from list transforms output table (still present in JSON representation)

[oleiman]

force push remove vestigial commit

[oleiman]

PR labeler failure under investigation: https://redpandadata.slack.com/archives/C06MCRY75U7/p1718120121998649

[oleiman]

/ci-repeat 1

[vbotbuildovich]

ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/50134#019009db-3e9a-476b-9cc0-c729c1fc734e

[oleiman]

CI Failures (both previously known and unrelated) :

• https://github.com/redpanda-data/redpanda/issues/16036
• https://github.com/redpanda-data/redpanda/issues/14225

[oleiman]

/ci-repeat 1

[oleiman]

force push rebase dev to fix labeler action

[oleiman]

/ci-repeat 1

[oleiman]

force push contents:

• marginally improve comments on transform_metadata_patch
• improve string_switch handling for compression mode

[oleiman]

CI Failure is some pandatriage issue

[oleiman]

force push rebase dev to fix merge conflict

[oleiman]

CI Failure:

• Looks like https://github.com/redpanda-data/redpanda/issues/19953 (unrelated)