Kafka Pump does not auto-reconnect when partition leader fails/changes

[Milisha-Gupta]

Description:

While testing iDRAC Telemetry in a multi-broker Kafka HA setup, we observed that the Kafka Pump does not automatically reconnect to a new leader when the current partition leader fails/changes even though the pump is configured with a headless service.

Steps to Reproduce:

• Configure Kafka with multiple brokers and replication factor = 3.
• Start the Kafka Pump and have it write to a specific partition (e.g., partition 0).
• Kill the leader broker for that partition.
• Observe that the Kafka Pump stops sending data.

Expected Behavior: The Kafka Pump should detect the partition leader change and automatically reconnect to the new leader broker, resuming data ingestion without manual intervention.

Actual Behavior: The Kafka Pump still tries to push data to the dead broker remains disconnected from the topic/partition until it is restarted or manually redirected.

Impact:

Breaks continuous telemetry ingestion in HA scenarios. Requires manual intervention, reducing system reliability.