pinot icon indicating copy to clipboard operation
pinot copied to clipboard
verified publisher icon apache

Flaky Tests: TPCHQueryIntegrationTest and pinot-kafka-3.0

Open ankitsultana opened this issue 1 year ago • 3 comments

Saw these failures as part of #14064.

TPCH

Error:    TPCHQueryIntegrationTest.testTPCHQueries:111->testQueriesSucceed:115 » PinotClient java.util.concurrent.TimeoutException

pinot-kafka-3.0

21:49:15.240 ERROR [KafkaServer] [main] Fatal error during KafkaServer startup. Prepare to shutdown
kafka.zookeeper.ZooKeeperClientTimeoutException: Timed out waiting for connection while in state: CONNECTING
	at kafka.zookeeper.ZooKeeperClient.$anonfun$waitUntilConnected$3(ZooKeeperClient.scala:257) ~[kafka_2.12-3.8.0.jar:?]
	at kafka.zookeeper.ZooKeeperClient.waitUntilConnected(ZooKeeperClient.scala:253) ~[kafka_2.12-3.8.0.jar:?]
	at kafka.zookeeper.ZooKeeperClient.<init>(ZooKeeperClient.scala:115) ~[kafka_2.12-3.8.0.jar:?]
	at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:2317) ~[kafka_2.12-3.8.0.jar:?]
	at kafka.zk.KafkaZkClient$.createZkClient(KafkaZkClient.scala:2407) ~[kafka_2.12-3.8.0.jar:?]
	at kafka.server.KafkaServer.initZkClient(KafkaServer.scala:742) ~[kafka_2.12-3.8.0.jar:?]
	at kafka.server.KafkaServer.startup(KafkaServer.scala:230) ~[kafka_2.12-3.8.0.jar:?]
	at org.apache.pinot.plugin.stream.kafka30.utils.MiniKafkaCluster.start(MiniKafkaCluster.java:94) ~[test-classes/:?]
	at org.apache.pinot.plugin.stream.kafka30.KafkaPartitionLevelConsumerTest.setUp(KafkaPartitionLevelConsumerTest.java:69) ~[test-classes/:?]
...
Error:  Tests run: 8, Failures: 1, Errors: 0, Skipped: 7, Time elapsed: 39.93 s <<< FAILURE! -- in org.apache.pinot.plugin.stream.kafka30.KafkaPartitionLevelConsumerBackwardCompatibilityTest
Error:  org.apache.pinot.plugin.stream.kafka30.KafkaPartitionLevelConsumerBackwardCompatibilityTest.setUp -- Time elapsed: 39.72 s <<< FAILURE!
kafka.zookeeper.ZooKeeperClientTimeoutException: Timed out waiting for connection while in state: CONNECTING
	at kafka.zookeeper.ZooKeeperClient.$anonfun$waitUntilConnected$3(ZooKeeperClient.scala:257)
	at kafka.zookeeper.ZooKeeperClient.waitUntilConnected(ZooKeeperClient.scala:253)
	at kafka.zookeeper.ZooKeeperClient.<init>(ZooKeeperClient.scala:115)
	at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:2317)
	at kafka.zk.KafkaZkClient$.createZkClient(KafkaZkClient.scala:2407)
	at kafka.server.KafkaServer.initZkClient(KafkaServer.scala:742)
	at kafka.server.KafkaServer.startup(KafkaServer.scala:230)
	at org.apache.pinot.plugin.stream.kafka30.utils.MiniKafkaCluster.start(MiniKafkaCluster.java:94)
	at org.apache.pinot.plugin.stream.kafka30.KafkaPartitionLevelConsumerTest.setUp(KafkaPartitionLevelConsumerTest.java:69)

pinot-kafka-2.0

2024-09-27T11:41:19.469289580Z main ERROR Reconfiguration failed: No configuration found for '4e0e2f2a' at 'null' in 'null'
11:41:19.961 WARN [KafkaStreamMetadataProvider] [main] initial offset type is timestamp and its value evaluates to null hence proceeding with offset 1000 for topic foo partition 0
11:41:20.120 WARN [KafkaStreamMetadataProvider] [main] initial offset type is timestamp and its value evaluates to null hence proceeding with offset 1000 for topic bar partition 0
11:41:20.175 WARN [KafkaStreamMetadataProvider] [main] initial offset type is timestamp and its value evaluates to null hence proceeding with offset 1000 for topic bar partition 1
11:41:21.468 ERROR [Util] [main] Last transaction was partial.
11:41:21.489 ERROR [Util] [main] Last transaction was partial.
Error:  Tests run: 7, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 60.79 s <<< FAILURE! -- in org.apache.pinot.plugin.stream.kafka20.KafkaPartitionLevelConsumerTest
Error:  org.apache.pinot.plugin.stream.kafka20.KafkaPartitionLevelConsumerTest.testBuildConsumer -- Time elapsed: 20.50 s
Error:  org.apache.pinot.plugin.stream.kafka20.KafkaPartitionLevelConsumerTest.testConsumer -- Time elapsed: 0.691 s
Error:  org.apache.pinot.plugin.stream.kafka20.KafkaPartitionLevelConsumerTest.testFetchMessages -- Time elapsed: 10.02 s
Error:  org.apache.pinot.plugin.stream.kafka20.KafkaPartitionLevelConsumerTest.testFetchOffsets -- Time elapsed: 0.574 s
Error:  org.apache.pinot.plugin.stream.kafka20.KafkaPartitionLevelConsumerTest.testGetPartitionCount -- Time elapsed: 0.079 s
Error:  org.apache.pinot.plugin.stream.kafka20.KafkaPartitionLevelConsumerTest.testOffsetsExpired -- Time elapsed: 0.118 s
Error:  org.apache.pinot.plugin.stream.kafka20.KafkaPartitionLevelConsumerTest.tearDown -- Time elapsed: 2.822 s <<< FAILURE!
org.apache.commons.io.IOExceptionList: 2 exception(s): [org.apache.commons.io.IOIndexedException: IOException #0: Cannot delete file: /tmp/EmbeddedZooKeeper/log, org.apache.commons.io.IOIndexedException: IOException #1: Cannot delete file: /tmp/EmbeddedZooKeeper/data]
	at org.apache.commons.io.IOExceptionList.checkEmpty(IOExceptionList.java:50)
	at org.apache.commons.io.function.IOStream.forAll(IOStream.java:357)
	at org.apache.commons.io.function.IOStreams.forAll(IOStreams.java:42)
	at org.apache.commons.io.function.IOStreams.forAll(IOStreams.java:36)
	at org.apache.commons.io.function.IOConsumer.forAll(IOConsumer.java:80)
	at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:367)
	at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1242)
	at org.apache.pinot.plugin.stream.kafka.utils.EmbeddedZooKeeper.close(EmbeddedZooKeeper.java:54)
	at org.apache.pinot.plugin.stream.kafka20.utils.MiniKafkaCluster.close(MiniKafkaCluster.java:102)
	at org.apache.pinot.plugin.stream.kafka20.KafkaPartitionLevelConsumerTest.tearDown(KafkaPartitionLevelConsumerTest.java:107)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)

Caused by: org.apache.commons.io.IOIndexedException: IOException #0: Cannot delete file: /tmp/EmbeddedZooKeeper/log
	at org.apache.commons.io.function.IOStream.lambda$forAll$11(IOStream.java:352)
	at java.base/java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:1024)
	at java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:762)
	at org.apache.commons.io.function.IOStream.forAll(IOStream.java:343)
	... 38 more
Caused by: java.io.IOException: Cannot delete file: /tmp/EmbeddedZooKeeper/log
	at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:1392)
	at org.apache.commons.io.function.IOStream.lambda$forAll$11(IOStream.java:345)
	... 41 more
Caused by: java.nio.file.NoSuchFileException: /tmp/EmbeddedZooKeeper/log/version-2
	at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
	at java.base/sun.nio.fs.UnixException.asIOException(UnixException.java:115)
	at java.base/sun.nio.fs.UnixFileSystemProvider.newDirectoryStream(UnixFileSystemProvider.java:477)
	at java.base/java.nio.file.Files.newDirectoryStream(Files.java:481)
	at org.apache.commons.io.file.PathUtils.isEmptyDirectory(PathUtils.java:1075)
	at org.apache.commons.io.file.DeletingPathVisitor.postVisitDirectory(DeletingPathVisitor.java:139)
	at org.apache.commons.io.file.DeletingPathVisitor.postVisitDirectory(DeletingPathVisitor.java:37)
	at java.base/java.nio.file.Files.walkFileTree(Files.java:2803)
	at java.base/java.nio.file.Files.walkFileTree(Files.java:2857)
	at org.apache.commons.io.file.PathUtils.visitFileTree(PathUtils.java:1730)
	at org.apache.commons.io.file.PathUtils.deleteDirectory(PathUtils.java:518)
	at org.apache.commons.io.file.PathUtils.delete(PathUtils.java:477)
	at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:1388)
	... 42 more

ankitsultana avatar Sep 27 '24 06:09 ankitsultana

I added the module recently, let me look in to it. Thanks

abhioncbr avatar Sep 27 '24 11:09 abhioncbr

@abhioncbr : Shaurya was planning to pickup the TPCH one, does that work? @shauryachats

Also, there's an error with pinot-kafka-2.0 also. Just added it to the list.

ankitsultana avatar Sep 27 '24 13:09 ankitsultana

Got it. Thanks for the more context.

abhioncbr avatar Sep 27 '24 14:09 abhioncbr

Kafka one was fixed in #14458

Jackie-Jiang avatar Nov 15 '24 23:11 Jackie-Jiang