kafka Back-port KAFKA-16230 to 3.7 branch

Back-porting changes for KAFKA-16230 to 3.7. Original change here: #15328.

The Python VerifiableConsumer now passes in the --group-protocol and --group-remote-assignor command line arguments to VerifiableConsumer if the node is running 3.7.0+ and using the new consumer group.protocol.

Reviewers: Andrew Schofield [email protected], Lucas Brutschy [email protected]

Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

Aug 21 '24 18:08 kirktrue

@chia7712 @lucasbru @cadonna—would one of you be willing to review this cherry-pick? Thanks!

Aug 23 '24 18:08 kirktrue

Note: there are lots of test failures for PlaintextConsumerTest’s testPerPartitionLagMetricsCleanUpWithSubscribe and testPerPartitionLeadMetricsCleanUpWithSubscribe. This PR shouldn't affect those tests, but I'll investigate more.

cc @lianetm @philipnee

Aug 23 '24 21:08 kirktrue

@lianetm—can you take a look at this fix for system tests? Thanks!

Aug 29 '24 20:08 kirktrue

Hey @kirktrue , took a first look and overall it looks good. Is there a run of the system tests with this change? (agree that failures in PlainTextConsumer are unrelated to this PR). Thanks!

Aug 30 '24 15:08 lianetm

Is there a run of the system tests with this change?

Yes, there's we ran this internally a few times and the tests were able to run successfully.

Sep 03 '24 17:09 kirktrue

Thanks @lianetm!

Sep 03 '24 20:09 kirktrue