Yu Ning

WIP, it only supports tail-read now.

Controller should return the correct number of sub-responses when an error occurs

1

related #603 server.log ``` [2023-12-27 03:31:44,875] ERROR Response size: 0 not match request size: 1 (kafka.log.stream.s3.network.ControllerRequestSender) [2023-12-27 03:31:44,882] ERROR [Stream id=14886 epoch=2] trim fail (com.automq.stream.s3.S3Stream) java.lang.RuntimeException: Response size not match...

When I try to delete 512 topics (7 partitions in each), I got: server.log ``` [2024-01-04 06:43:30,093] ERROR [KafkaApi-1] Unexpected error handling request RequestHeader(apiKey=DELETE_TOPICS, apiVersion=6, clientId=adminclient-1, correlationId=4, headerVersion=2) -- DeleteTopicsRequestData(topics=[...],...

In server.log, log entries are repeatedly printed: ``` [2023-10-10 08:40:46,626] INFO [Consumer clientId=AutoBalancerControllerConsumer-consumer--7526509455016652825, groupId=AutoBalancerControllerConsumerGroup-group--7526509455016652825] Discovered group coordinator 10.0.1.142:9092 (id: 2147483643 rack: null) (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator) [2023-10-10 08:40:46,626] INFO [Consumer clientId=AutoBalancerControllerConsumer-consumer--7526509455016652825, groupId=AutoBalancerControllerConsumerGroup-group--7526509455016652825] Request...

server.log: ``` [2023-10-12 09:28:08,713] INFO log append cost, permitAcquireFail={count=0, avg=0ns, max=0ns}, remainingPermit=1073712638/1073741824 ,append={count=17692, avg=91us, max=19ms}, callback={count=17694, avg=1ms, max=30ms}, ack={count=15819, avg=385us, max=39ms} (kafka.log.streamaspect.ElasticLog) [2023-10-12 09:28:08,713] INFO [KafkaApi-0] produce cost, produce={count=7293, avg=3ms,...

The write latency experiences spikes

1

``` ubuntu@ip-10-0-0-182:~$ sudo java -ea -cp s3stream-0.3.0-SNAPSHOT-jar-with-dependencies.jar com.automq.stream.s3.wal.benchmark.WriteBench -p /mnt/data/testfile --duration 30000 --throughput 120000000 --record-size 40960 -d 8 --threads 1 SLF4J: No SLF4J providers were found. SLF4J: Defaulting to no-operation...

[Good First Issue] Write Unit Tests for `WALConfirmOffsetCalculator`

7

### Problem Description The `com.automq.stream.s3.S3Storage.WALConfirmOffsetCalculator` class currently lacks comprehensive unit tests. ### Required Changes 1. **Read and understand the original implementation**: 1. `S3Storage` sends `append` requests to the WAL, but...

refactor(s3stream/object-wal): complete appends sequentially

1

resolve #2432 ## Performance Test - Before ``` 2025-04-18 07:42:14 - INFO Summary | Prod rate 1600.46 msg/s / 79.71 MiB/s | Prod total 0.48 M msg / 23.36 GiB...

[Good First Issue] Add Consumer Percentage Control for Catch-up Reading in Performance Test

5

### Background AutoMQ provides the [`automq-perf-test.sh`](https://www.automq.com/blog/how-to-perform-a-performance-test-on-automq) script to test performance under different scenarios. Currently, it supports testing "catch-up read" (cold read) scenarios by: 1. Sending a batch of messages. 2....

Memory Leak in `CompactionUploader` During Shutting Down

3

``` 2025-03-18 15:55:10.063 +0800 ERROR [io.netty.util.ResourceLeakDetector] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information. Recent access records: Created at: io.netty.buffer.AbstractByteBufAllocator.compositeDirectBuffer(AbstractByteBufAllocator.java:224) com.automq.stream.s3.ByteBufAlloc.compositeByteBuffer(ByteBufAlloc.java:128) com.automq.stream.s3.operator.ProxyWriter$ObjectWriter.(ProxyWriter.java:127) com.automq.stream.s3.operator.ProxyWriter.(ProxyWriter.java:33) com.automq.stream.s3.operator.ProxyWriter.(ProxyWriter.java:48) com.automq.stream.s3.operator.AbstractObjectStorage.writer(AbstractObjectStorage.java:200)...