HIVE-27407: INSERT INTO external Kafka table fails with NoSuchFieldException

[zabetak]

What changes were proposed in this pull request?

The topicPartitionBookkeeper field was removed from TransactionManager in the latest Kafka version and was replaced by txnPartitionMap.

Below the commit which applied the respective refactoring in Kafka: https://github.com/apache/kafka/commit/3ea7b418fb3d7e9fc74c27751c1b02b04877f197

Renable the kafka_storage_handler.q test since it has now passed sucessfully from the flaky-checker: http://ci.hive.apache.org/job/hive-flaky-check/694/

Why are the changes needed?

Resolve NoSuchFieldException when running INSERT queries in Kafka tables

Does this PR introduce any user-facing change?

No

How was this patch tested?

mvn -pl itests/qtest -Pitests test -Dtest=TestMiniHiveKafkaCliDriver -Dqfile=kafka_storage_handler.q

[akshat0395]

Hi @zabetak will this change break any the backwards compatibility with older kafka version?

[zabetak]

Hi @zabetak will this change break any the backwards compatibility with older kafka version?

@akshat0395 Any code that relies on reflection and internal classes is unlike to be backward compatible in the long term. This change does not break backward compatibility; this already happened in HIVE-27105 where Kafka was upgraded from 2.5.0 to 3.4.0.

Hive does not guarantee that it will be compatible with multiple Kafka, Hadoop, Tez, etc. versions. We should just ensure that the one used in the pom.xml is supported/working. In other words, I don't think we should worry to much about previous Kafka versions but just make sure that things work for the one we actually use i.e., 3.4.0.

[sonarqubecloud[bot]]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
19 Code Smells

No Coverage information
No Duplication information

[akshat0395]

@zabetak any changes required in this post downgrade?

[github-actions[bot]]

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Feel free to reach out on the [email protected] list if the patch is in need of reviews.

[github-actions[bot]]

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Feel free to reach out on the [email protected] list if the patch is in need of reviews.