Update Kafka client version and change partition assignor

[tomncooper]

NOTE: Since this PR was posted the confluent-kafka version has already been updated, but the config change to use cooperative rebalancing in the consumer is still an important change.

💸 TL;DR

This PR updates the confluent kafka client version, which brings bug fixes and support for newer Kafka versions and features. It also updates the default partition assignor used by the Kafka consumers to use the "cooperative sticky" assigner, which is the recommended, faster option.

📜 Details

While Kafka servers are backwards compatible with older client versions, new features are limited to newer client versions so it is important to keep you clients up to date. Also, newer clients are backwards compatible with older server so there should not be any compatibility concerns. This, along with the obvious advantages of bug and performance fixes (including Mac M1 support) that come with new versions, makes this upgrade worthwhile.

The new client version also allows new configuration options. The newer consumer clients can use the "cooperative sticky" assignor (see KIP-726 for more details) which dramatically speeds up consumer group rebalances (by only changing partitions of consumers that have stopped rather that potentially shuffling the partitions of every consumer). This should make baseplate.py kafka consumer applications more stable.

✅ Checks

• [x] CI tests (if present) are passing
• [x] Adheres to code style for repo

[tomncooper]

@KTAtkinson Since this PR was posted the confluent-kafka version has already been updated, but the config change to use cooperative rebalancing in the consumer is still an important change.

What do we need to do to get this merged?