CI Failure (timeout transferring leadership after restart) in `ManyPartitionsTest`.`test_many_partitions`

[jcsp]

Two very similar failures on arm64 in this job, in both the regular and the compacted variant of the test: https://buildkite.com/redpanda/vtools/builds/8347#01892282-acd8-41e8-9d58-b091d460ee38

Module: rptest.scale_tests.many_partitions_test
Class:  ManyPartitionsTest
Method: test_many_partitions

Logs have many warnings about timeouts transferring leadership:

RedpandaService-0-281472777116384/ip-172-31-6-250/redpanda.log:WARN  2023-07-04 20:58:45,280 [shard 1] tx - [{kafka/scale_000000/11734}] - rm_stm.cc:1818 - [pid: {producer_identity: id=29, epoch=0}] replication of batch with first seq: 2 failed at consensus level - error: raft::errc::replicated_entry_truncated
RedpandaService-0-281472777116384/ip-172-31-6-250/redpanda.log:WARN  2023-07-04 20:59:06,902 [shard 3] raft - [group_id:5776, {kafka/scale_000000/5775}] consensus.cc:3073 - transfer leadership: timed out waiting on node {id: {9}, revision: {66}} recovery
RedpandaService-0-281472777116384/ip-172-31-6-250/redpanda.log:WARN  2023-07-04 20:59:16,346 [shard 0] raft - [group_id:9717, {kafka/scale_000000/9716}] consensus.cc:3073 - transfer leadership: timed out waiting on node {id: {9}, revision: {66}} recovery

  File "/home/ubuntu/redpanda/tests/rptest/scale_tests/many_partitions_test.py", line 879, in test_many_partitions_compacted
    self._test_many_partitions(compacted=True)
  File "/home/ubuntu/redpanda/tests/rptest/scale_tests/many_partitions_test.py", line 1072, in _test_many_partitions
    self._single_node_restart(scale, topic_names, n_partitions)
  File "/home/ubuntu/redpanda/tests/rptest/scale_tests/many_partitions_test.py", line 534, in _single_node_restart
    wait_until(
  File "/usr/local/lib/python3.10/dist-packages/ducktape/utils/util.py", line 57, in wait_until
    raise TimeoutError(err_msg() if callable(err_msg) else err_msg) from last_exception
ducktape.errors.TimeoutError

[rystsov]

it returned: https://buildkite.com/redpanda/vtools/builds/9277

[vbotbuildovich]

*https://buildkite.com/redpanda/vtools/builds/9284 *https://buildkite.com/redpanda/vtools/builds/9297 *https://buildkite.com/redpanda/vtools/builds/9309

[github-actions[bot]]

This issue hasn't seen activity in 3 months. If you want to keep it open, post a comment or remove the stale label – otherwise this will be closed in two weeks.

[github-actions[bot]]

This issue was closed due to lack of activity. Feel free to reopen if it's still relevant.