self-hosted
self-hosted copied to clipboard
Published
4 months ago
•
getsentry
getsentry
Kafka volume keeps growing
Self-Hosted Version
24.10.0
CPU Architecture
x86_64
Docker Version
Docker version 27.1.1, build 6312585
Docker Compose Version
Docker Compose version v2.29.1
Steps to Reproduce
On a fresh installation of 24.9.0, with partial restore (following docs) from a different server, the Kafka volume seems to keep growing.
Expected Result
On our old instance the kafka volume took ~1.3GB, compared to clickhouse volume taking 35GB.
On the new instance, it is by far the largest volume:
VOLUME NAME LINKS SIZE
sentry-kafka 1 39GB
sentry-self-hosted_sentry-smtp-log 1 2.583kB
sentry-self-hosted_sentry-vroom 2 200.3MB
sentry-symbolicator 2 5.078GB
c03a740937a046d6f7a8bb5f46cf351f31e444f53b34c68b6b1d787f8c6c4f18 0 0B
sentry-clickhouse 1 11.78GB
b1d8a2343bbf7a96bc388a2d96d2a816224d3ef2217b80ca5058d4a2cc3271e5 1 0B
sentry-postgres 1 16.94GB
sentry-redis 1 28.15MB
sentry-self-hosted_sentry-clickhouse-log 1 1.642GB
ebe982febdedd07374d4f81e8f3022a0cfeba5806ec9db104d7447d29a3569a5 1 0B
sentry-self-hosted_sentry-nginx-www 1 0B
409b13a88bedd0278b4d25fda55bbf112a0492af8f4cc397804d8ae04907634b 1 0B
789a48023c997fb402483c9540331141684be4122cb0c834084ba130dee7b8d4 0 0B
sentry-self-hosted_sentry-nginx-cache 1 0B
sentry-self-hosted_sentry-secrets 1 0B
sentry-self-hosted_sentry-smtp 1 0B
e614865d9139b4c6e127efd81400302f105bfa5def2b997eb6323f46070cb335 0 0B
sentry-self-hosted_sentry-kafka-log 1 0B
sentry-data 24 620.3MB
Actual Result
All containers seem to be running fine, see docker compose ps (restarted recently after an upgrade - I was hoping it might fix the issue).
NAME IMAGE COMMAND SERVICE CREATED STATUS PORTS
sentry-self-hosted-attachments-consumer-1 sentry-self-hosted-local "/etc/sentry/entrypo…" attachments-consumer 13 minutes ago Up 13 minutes 9000/tcp
sentry-self-hosted-billing-metrics-consumer-1 sentry-self-hosted-local "/etc/sentry/entrypo…" billing-metrics-consumer 13 minutes ago Up 13 minutes 9000/tcp
sentry-self-hosted-clickhouse-1 clickhouse-self-hosted-local "/entrypoint.sh" clickhouse 15 minutes ago Up 15 minutes (healthy) 8123/tcp, 9000/tcp, 9009/tcp
sentry-self-hosted-cron-1 sentry-self-hosted-local "/etc/sentry/entrypo…" cron 13 minutes ago Up 13 minutes 9000/tcp
sentry-self-hosted-events-consumer-1 sentry-self-hosted-local "/etc/sentry/entrypo…" events-consumer 13 minutes ago Up 13 minutes 9000/tcp
sentry-self-hosted-generic-metrics-consumer-1 sentry-self-hosted-local "/etc/sentry/entrypo…" generic-metrics-consumer 13 minutes ago Up 13 minutes 9000/tcp
sentry-self-hosted-ingest-feedback-events-1 sentry-self-hosted-local "/etc/sentry/entrypo…" ingest-feedback-events 13 minutes ago Up 13 minutes 9000/tcp
sentry-self-hosted-ingest-monitors-1 sentry-self-hosted-local "/etc/sentry/entrypo…" ingest-monitors 13 minutes ago Up 13 minutes 9000/tcp
sentry-self-hosted-ingest-occurrences-1 sentry-self-hosted-local "/etc/sentry/entrypo…" ingest-occurrences 13 minutes ago Up 13 minutes 9000/tcp
sentry-self-hosted-ingest-profiles-1 sentry-self-hosted-local "/etc/sentry/entrypo…" ingest-profiles 13 minutes ago Up 13 minutes 9000/tcp
sentry-self-hosted-ingest-replay-recordings-1 sentry-self-hosted-local "/etc/sentry/entrypo…" ingest-replay-recordings 13 minutes ago Up 13 minutes 9000/tcp
sentry-self-hosted-kafka-1 confluentinc/cp-kafka:7.6.1 "/etc/confluent/dock…" kafka 15 minutes ago Up 15 minutes (healthy) 9092/tcp
sentry-self-hosted-memcached-1 memcached:1.6.26-alpine "docker-entrypoint.s…" memcached 14 minutes ago Up 14 minutes (healthy) 11211/tcp
sentry-self-hosted-metrics-consumer-1 sentry-self-hosted-local "/etc/sentry/entrypo…" metrics-consumer 13 minutes ago Up 13 minutes 9000/tcp
sentry-self-hosted-monitors-clock-tasks-1 sentry-self-hosted-local "/etc/sentry/entrypo…" monitors-clock-tasks 13 minutes ago Up 13 minutes 9000/tcp
sentry-self-hosted-monitors-clock-tick-1 sentry-self-hosted-local "/etc/sentry/entrypo…" monitors-clock-tick 13 minutes ago Up 13 minutes 9000/tcp
sentry-self-hosted-nginx-1 nginx:1.25.4-alpine "/docker-entrypoint.…" nginx 11 minutes ago Up 11 minutes 0.0.0.0:9000->80/tcp, :::9000->80/tcp
sentry-self-hosted-post-process-forwarder-errors-1 sentry-self-hosted-local "/etc/sentry/entrypo…" post-process-forwarder-errors 13 minutes ago Up 13 minutes 9000/tcp
sentry-self-hosted-post-process-forwarder-issue-platform-1 sentry-self-hosted-local "/etc/sentry/entrypo…" post-process-forwarder-issue-platform 13 minutes ago Up 13 minutes 9000/tcp
sentry-self-hosted-post-process-forwarder-transactions-1 sentry-self-hosted-local "/etc/sentry/entrypo…" post-process-forwarder-transactions 13 minutes ago Up 13 minutes 9000/tcp
sentry-self-hosted-postgres-1 postgres:14.11 "docker-entrypoint.s…" postgres 14 minutes ago Up 14 minutes (healthy) 5432/tcp
sentry-self-hosted-redis-1 redis:6.2.14-alpine "docker-entrypoint.s…" redis 15 minutes ago Up 15 minutes (healthy) 6379/tcp
sentry-self-hosted-relay-1 getsentry/relay:24.10.0 "/bin/bash /docker-e…" relay 12 minutes ago Up 11 minutes 3000/tcp
sentry-self-hosted-sentry-cleanup-1 sentry-cleanup-self-hosted-local "/entrypoint.sh '0 0…" sentry-cleanup 13 minutes ago Up 13 minutes 9000/tcp
sentry-self-hosted-smtp-1 tianon/exim4 "docker-entrypoint.s…" smtp 14 minutes ago Up 14 minutes 25/tcp
sentry-self-hosted-snuba-api-1 getsentry/snuba:24.10.0 "./docker_entrypoint…" snuba-api 14 minutes ago Up 14 minutes 1218-1219/tcp
sentry-self-hosted-snuba-errors-consumer-1 getsentry/snuba:24.10.0 "./docker_entrypoint…" snuba-errors-consumer 13 minutes ago Up 13 minutes 1218-1219/tcp
sentry-self-hosted-snuba-generic-metrics-counters-consumer-1 getsentry/snuba:24.10.0 "./docker_entrypoint…" snuba-generic-metrics-counters-consumer 13 minutes ago Up 13 minutes 1218-1219/tcp
sentry-self-hosted-snuba-generic-metrics-distributions-consumer-1 getsentry/snuba:24.10.0 "./docker_entrypoint…" snuba-generic-metrics-distributions-consumer 13 minutes ago Up 13 minutes 1218-1219/tcp
sentry-self-hosted-snuba-generic-metrics-gauges-consumer-1 getsentry/snuba:24.10.0 "./docker_entrypoint…" snuba-generic-metrics-gauges-consumer 13 minutes ago Up 13 minutes 1218-1219/tcp
sentry-self-hosted-snuba-generic-metrics-sets-consumer-1 getsentry/snuba:24.10.0 "./docker_entrypoint…" snuba-generic-metrics-sets-consumer 13 minutes ago Up 13 minutes 1218-1219/tcp
sentry-self-hosted-snuba-group-attributes-consumer-1 getsentry/snuba:24.10.0 "./docker_entrypoint…" snuba-group-attributes-consumer 13 minutes ago Up 13 minutes 1218-1219/tcp
sentry-self-hosted-snuba-issue-occurrence-consumer-1 getsentry/snuba:24.10.0 "./docker_entrypoint…" snuba-issue-occurrence-consumer 13 minutes ago Up 13 minutes 1218-1219/tcp
sentry-self-hosted-snuba-metrics-consumer-1 getsentry/snuba:24.10.0 "./docker_entrypoint…" snuba-metrics-consumer 13 minutes ago Up 13 minutes 1218-1219/tcp
sentry-self-hosted-snuba-outcomes-billing-consumer-1 getsentry/snuba:24.10.0 "./docker_entrypoint…" snuba-outcomes-billing-consumer 13 minutes ago Up 13 minutes 1218-1219/tcp
sentry-self-hosted-snuba-outcomes-consumer-1 getsentry/snuba:24.10.0 "./docker_entrypoint…" snuba-outcomes-consumer 13 minutes ago Up 13 minutes 1218-1219/tcp
sentry-self-hosted-snuba-profiling-functions-consumer-1 getsentry/snuba:24.10.0 "./docker_entrypoint…" snuba-profiling-functions-consumer 13 minutes ago Up 13 minutes 1218-1219/tcp
sentry-self-hosted-snuba-profiling-profiles-consumer-1 getsentry/snuba:24.10.0 "./docker_entrypoint…" snuba-profiling-profiles-consumer 13 minutes ago Up 13 minutes 1218-1219/tcp
sentry-self-hosted-snuba-replacer-1 getsentry/snuba:24.10.0 "./docker_entrypoint…" snuba-replacer 13 minutes ago Up 13 minutes 1218-1219/tcp
sentry-self-hosted-snuba-replays-consumer-1 getsentry/snuba:24.10.0 "./docker_entrypoint…" snuba-replays-consumer 13 minutes ago Up 13 minutes 1218-1219/tcp
sentry-self-hosted-snuba-spans-consumer-1 getsentry/snuba:24.10.0 "./docker_entrypoint…" snuba-spans-consumer 13 minutes ago Up 13 minutes 1218-1219/tcp
sentry-self-hosted-snuba-subscription-consumer-events-1 getsentry/snuba:24.10.0 "./docker_entrypoint…" snuba-subscription-consumer-events 13 minutes ago Up 13 minutes 1218-1219/tcp
sentry-self-hosted-snuba-subscription-consumer-metrics-1 getsentry/snuba:24.10.0 "./docker_entrypoint…" snuba-subscription-consumer-metrics 13 minutes ago Up 13 minutes 1218-1219/tcp
sentry-self-hosted-snuba-subscription-consumer-transactions-1 getsentry/snuba:24.10.0 "./docker_entrypoint…" snuba-subscription-consumer-transactions 13 minutes ago Up 13 minutes 1218-1219/tcp
sentry-self-hosted-snuba-transactions-consumer-1 getsentry/snuba:24.10.0 "./docker_entrypoint…" snuba-transactions-consumer 13 minutes ago Up 13 minutes 1218-1219/tcp
sentry-self-hosted-subscription-consumer-events-1 sentry-self-hosted-local "/etc/sentry/entrypo…" subscription-consumer-events 13 minutes ago Up 13 minutes 9000/tcp
sentry-self-hosted-subscription-consumer-generic-metrics-1 sentry-self-hosted-local "/etc/sentry/entrypo…" subscription-consumer-generic-metrics 13 minutes ago Up 13 minutes 9000/tcp
sentry-self-hosted-subscription-consumer-metrics-1 sentry-self-hosted-local "/etc/sentry/entrypo…" subscription-consumer-metrics 13 minutes ago Up 13 minutes 9000/tcp
sentry-self-hosted-subscription-consumer-transactions-1 sentry-self-hosted-local "/etc/sentry/entrypo…" subscription-consumer-transactions 13 minutes ago Up 13 minutes 9000/tcp
sentry-self-hosted-symbolicator-1 getsentry/symbolicator:24.10.0 "/bin/bash /docker-e…" symbolicator 14 minutes ago Up 14 minutes 3021/tcp
sentry-self-hosted-symbolicator-cleanup-1 symbolicator-cleanup-self-hosted-local "/entrypoint.sh '55 …" symbolicator-cleanup 13 minutes ago Up 13 minutes 3021/tcp
sentry-self-hosted-transactions-consumer-1 sentry-self-hosted-local "/etc/sentry/entrypo…" transactions-consumer 13 minutes ago Up 13 minutes 9000/tcp
sentry-self-hosted-vroom-1 getsentry/vroom:24.10.0 "/bin/vroom" vroom 13 minutes ago Up 13 minutes 8080/tcp
sentry-self-hosted-vroom-cleanup-1 vroom-cleanup-self-hosted-local "/entrypoint.sh '0 0…" vroom-cleanup 13 minutes ago Up 13 minutes 8080/tcp
sentry-self-hosted-web-1 sentry-self-hosted-local "/etc/sentry/entrypo…" web 13 minutes ago Up 13 minutes (healthy) 9000/tcp
sentry-self-hosted-worker-1 sentry-self-hosted-local "/etc/sentry/entrypo…" worker 13 minutes ago Up 13 minutes 9000/tcp
No modifications in docker-compose or other files:
On branch 24.10.0
Untracked files:
(use "git add <file>..." to include in what will be committed)
sentry/backup.json
nothing added to commit but untracked files present (use "git add" to track)
Kafka seems to be working fine, no errors:
kafka-1 | ===> User
kafka-1 | uid=1000(appuser) gid=1000(appuser) groups=1000(appuser)
kafka-1 | ===> Configuring ...
kafka-1 | Running in KRaft mode...
kafka-1 | ===> Running preflight checks ...
kafka-1 | ===> Check if /var/lib/kafka/data is writable ...
kafka-1 | ===> Running in KRaft mode, skipping Zookeeper health check...
kafka-1 | ===> Using provided cluster id MkU3OEVBNTcwNTJENDM2Qk ...
kafka-1 | Log directory /var/lib/kafka/data is already formatted. Use --ignore-formatted to ignore this directory and format the others.
kafka-1 | ===> Launching ...
kafka-1 | ===> Launching kafka ...
kafka-1 | [2024-10-18 16:15:50,061] INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$)
kafka-1 | [2024-10-18 16:15:51,567] INFO Updated connection-accept-rate max connection creation rate to 2147483647 (kafka.network.ConnectionQuotas)
kafka-1 | [2024-10-18 16:15:51,624] INFO [SocketServer listenerType=CONTROLLER, nodeId=1001] Created data-plane acceptor and processors for endpoint : ListenerName(CONTROLLER) (kafka.network.SocketServer)
kafka-1 | [2024-10-18 16:15:54,356] INFO Initialized snapshots with IDs SortedSet(OffsetAndEpoch(offset=2419659, epoch=9), OffsetAndEpoch(offset=2426857, epoch=9), OffsetAndEpoch(offset=2434054, epoch=9), OffsetAndEpoch(offset=2441252, epoch=9), OffsetAndEpoch(offset=2448450, epoch=9), OffsetAndEpoch(offset=2455648, epoch=9), OffsetAndEpoch(offset=2462846, epoch=9), OffsetAndEpoch(offset=2470044, epoch=9), OffsetAndEpoch(offset=2477242, epoch=9), OffsetAndEpoch(offset=2484440, epoch=9), OffsetAndEpoch(offset=2491638, epoch=9), OffsetAndEpoch(offset=2498836, epoch=9), OffsetAndEpoch(offset=2506034, epoch=9), OffsetAndEpoch(offset=2513232, epoch=9), OffsetAndEpoch(offset=2520430, epoch=9), OffsetAndEpoch(offset=2527628, epoch=9), OffsetAndEpoch(offset=2534826, epoch=9), OffsetAndEpoch(offset=2542024, epoch=9), OffsetAndEpoch(offset=2549222, epoch=9), OffsetAndEpoch(offset=2556420, epoch=9), OffsetAndEpoch(offset=2563618, epoch=9), OffsetAndEpoch(offset=2570816, epoch=9), OffsetAndEpoch(offset=2578014, epoch=9), OffsetAndEpoch(offset=2585212, epoch=9), OffsetAndEpoch(offset=2592409, epoch=9), OffsetAndEpoch(offset=2599607, epoch=9), OffsetAndEpoch(offset=2606805, epoch=9), OffsetAndEpoch(offset=2614002, epoch=9), OffsetAndEpoch(offset=2621200, epoch=9), OffsetAndEpoch(offset=2628398, epoch=9), OffsetAndEpoch(offset=2635596, epoch=9), OffsetAndEpoch(offset=2642794, epoch=9), OffsetAndEpoch(offset=2649992, epoch=9), OffsetAndEpoch(offset=2657190, epoch=9), OffsetAndEpoch(offset=2664388, epoch=9), OffsetAndEpoch(offset=2671586, epoch=9), OffsetAndEpoch(offset=2678784, epoch=9), OffsetAndEpoch(offset=2685982, epoch=9), OffsetAndEpoch(offset=2693180, epoch=9), OffsetAndEpoch(offset=2700377, epoch=9), OffsetAndEpoch(offset=2707575, epoch=9), OffsetAndEpoch(offset=2714772, epoch=9), OffsetAndEpoch(offset=2721970, epoch=9), OffsetAndEpoch(offset=2729168, epoch=9), OffsetAndEpoch(offset=2736366, epoch=9), OffsetAndEpoch(offset=2743564, epoch=9), OffsetAndEpoch(offset=2750761, epoch=9), OffsetAndEpoch(offset=2757958, epoch=9), OffsetAndEpoch(offset=2765155, epoch=9), OffsetAndEpoch(offset=2772352, epoch=9), OffsetAndEpoch(offset=2779549, epoch=9), OffsetAndEpoch(offset=2786746, epoch=9), OffsetAndEpoch(offset=2793943, epoch=9), OffsetAndEpoch(offset=2801140, epoch=9), OffsetAndEpoch(offset=2808337, epoch=9), OffsetAndEpoch(offset=2815534, epoch=9), OffsetAndEpoch(offset=2822731, epoch=9), OffsetAndEpoch(offset=2829928, epoch=9), OffsetAndEpoch(offset=2837125, epoch=9), OffsetAndEpoch(offset=2844322, epoch=9), OffsetAndEpoch(offset=2851520, epoch=9), OffsetAndEpoch(offset=2858718, epoch=9), OffsetAndEpoch(offset=2865916, epoch=9), OffsetAndEpoch(offset=2873114, epoch=9), OffsetAndEpoch(offset=2880311, epoch=9), OffsetAndEpoch(offset=2887508, epoch=9), OffsetAndEpoch(offset=2894706, epoch=9), OffsetAndEpoch(offset=2901903, epoch=9), OffsetAndEpoch(offset=2909101, epoch=9), OffsetAndEpoch(offset=2916298, epoch=9), OffsetAndEpoch(offset=2923495, epoch=9), OffsetAndEpoch(offset=2930692, epoch=9), OffsetAndEpoch(offset=2937890, epoch=9), OffsetAndEpoch(offset=2945087, epoch=9), OffsetAndEpoch(offset=2952284, epoch=9), OffsetAndEpoch(offset=2959481, epoch=9), OffsetAndEpoch(offset=2966678, epoch=9), OffsetAndEpoch(offset=2973875, epoch=9), OffsetAndEpoch(offset=2981072, epoch=9), OffsetAndEpoch(offset=2988269, epoch=9), OffsetAndEpoch(offset=2995466, epoch=9), OffsetAndEpoch(offset=3002663, epoch=9), OffsetAndEpoch(offset=3009860, epoch=9), OffsetAndEpoch(offset=3017057, epoch=9), OffsetAndEpoch(offset=3024255, epoch=9), OffsetAndEpoch(offset=3031452, epoch=9), OffsetAndEpoch(offset=3038649, epoch=9), OffsetAndEpoch(offset=3045847, epoch=9), OffsetAndEpoch(offset=3053044, epoch=9), OffsetAndEpoch(offset=3060242, epoch=9), OffsetAndEpoch(offset=3067440, epoch=9), OffsetAndEpoch(offset=3074637, epoch=9), OffsetAndEpoch(offset=3081834, epoch=9), OffsetAndEpoch(offset=3089032, epoch=9), OffsetAndEpoch(offset=3096229, epoch=9), OffsetAndEpoch(offset=3103426, epoch=9), OffsetAndEpoch(offset=3110623, epoch=9)) from /var/lib/kafka/data/__cluster_metadata-0 (kafka.raft.KafkaMetadataLog$)
kafka-1 | [2024-10-18 16:15:54,386] INFO [raft-expiration-reaper]: Starting (kafka.raft.TimingWheelExpirationService$ExpiredOperationReaper)
kafka-1 | [2024-10-18 16:15:55,014] INFO [kafka-1001-raft-outbound-request-thread]: Starting (kafka.raft.RaftSendThread)
kafka-1 | [2024-10-18 16:15:55,015] INFO [kafka-1001-raft-io-thread]: Starting (kafka.raft.KafkaRaftManager$RaftIoThread)
kafka-1 | [2024-10-18 16:15:55,294] INFO [SocketServer listenerType=CONTROLLER, nodeId=1001] Enabling request processing. (kafka.network.SocketServer)
kafka-1 | [2024-10-18 16:15:55,298] WARN [QuorumController id=1001] Performing controller activation. Loaded ZK migration state of NONE. (org.apache.kafka.controller.QuorumController)
kafka-1 | [2024-10-18 16:15:55,314] INFO Awaiting socket connections on 0.0.0.0:29093. (kafka.network.DataPlaneAcceptor)
kafka-1 | [2024-10-18 16:15:55,433] INFO Updated connection-accept-rate max connection creation rate to 2147483647 (kafka.network.ConnectionQuotas)
kafka-1 | [2024-10-18 16:15:55,442] INFO [SocketServer listenerType=BROKER, nodeId=1001] Created data-plane acceptor and processors for endpoint : ListenerName(PLAINTEXT) (kafka.network.SocketServer)
kafka-1 | [2024-10-18 16:15:55,442] INFO Updated connection-accept-rate max connection creation rate to 2147483647 (kafka.network.ConnectionQuotas)
kafka-1 | [2024-10-18 16:15:55,451] INFO [SocketServer listenerType=BROKER, nodeId=1001] Created data-plane acceptor and processors for endpoint : ListenerName(INTERNAL) (kafka.network.SocketServer)
kafka-1 | [2024-10-18 16:15:55,452] INFO Updated connection-accept-rate max connection creation rate to 2147483647 (kafka.network.ConnectionQuotas)
kafka-1 | [2024-10-18 16:15:55,465] INFO [SocketServer listenerType=BROKER, nodeId=1001] Created data-plane acceptor and processors for endpoint : ListenerName(EXTERNAL) (kafka.network.SocketServer)
kafka-1 | [2024-10-18 16:15:57,159] INFO Starting the log cleaner (kafka.log.LogCleaner)
kafka-1 | [2024-10-18 16:15:57,578] INFO [SocketServer listenerType=BROKER, nodeId=1001] Enabling request processing. (kafka.network.SocketServer)
kafka-1 | [2024-10-18 16:15:57,579] INFO Awaiting socket connections on 0.0.0.0:29092. (kafka.network.DataPlaneAcceptor)
kafka-1 | [2024-10-18 16:15:57,581] INFO Awaiting socket connections on 0.0.0.0:9092. (kafka.network.DataPlaneAcceptor)
kafka-1 | [2024-10-18 16:15:57,582] INFO Awaiting socket connections on 0.0.0.0:9093. (kafka.network.DataPlaneAcceptor)
kafka-1 | [2024-10-18 16:16:12,402] INFO Cleaner 0: Beginning cleaning of log snuba-commit-log-0 (kafka.log.LogCleaner)
kafka-1 | [2024-10-18 16:16:12,403] INFO Cleaner 0: Building offset map for snuba-commit-log-0... (kafka.log.LogCleaner)
kafka-1 | [2024-10-18 16:16:12,461] INFO Cleaner 0: Building offset map for log snuba-commit-log-0 for 1 segments in offset range [119320, 120190). (kafka.log.LogCleaner)
kafka-1 | [2024-10-18 16:16:12,582] INFO Cleaner 0: Offset map for log snuba-commit-log-0 complete. (kafka.log.LogCleaner)
kafka-1 | [2024-10-18 16:16:12,608] INFO Cleaner 0: Cleaning log snuba-commit-log-0 (cleaning prior to Fri Oct 18 15:14:12 UTC 2024, discarding tombstones prior to upper bound deletion horizon Thu Oct 17 14:14:09 UTC 2024)... (kafka.log.LogCleaner)
kafka-1 | [2024-10-18 16:16:12,616] INFO Cleaner 0: Cleaning LogSegment(baseOffset=0, size=181, lastModifiedTime=1729260849098, largestRecordTimestamp=Some(1729260849093)) in log snuba-commit-log-0 into 0 with an upper bound deletion horizon 1729174449098 computed from the segment last modified time of 1729260849098,retaining deletes. (kafka.log.LogCleaner)
kafka-1 | [2024-10-18 16:16:12,620] INFO Cleaner 0: Cleaning LogSegment(baseOffset=119320, size=157285, lastModifiedTime=1729264452933, largestRecordTimestamp=Some(1729264452927)) in log snuba-commit-log-0 into 0 with an upper bound deletion horizon 1729174449098 computed from the segment last modified time of 1729264452933,retaining deletes. (kafka.log.LogCleaner)
kafka-1 | [2024-10-18 16:16:12,651] INFO Cleaner 0: Swapping in cleaned segment LogSegment(baseOffset=0, size=181, lastModifiedTime=1729264452933, largestRecordTimestamp=Some(1729264452927)) for segment(s) List(LogSegment(baseOffset=0, size=181, lastModifiedTime=1729260849098, largestRecordTimestamp=Some(1729260849093)), LogSegment(baseOffset=119320, size=157285, lastModifiedTime=1729264452933, largestRecordTimestamp=Some(1729264452927))) in log Log(dir=/var/lib/kafka/data/snuba-commit-log-0, topicId=Ie7bJXrfT06BIFOgvXNl3w, topic=snuba-commit-log, partition=0, highWatermark=121023, lastStableOffset=121023, logStartOffset=0, logEndOffset=121023) (kafka.log.LogCleaner)
kafka-1 | [2024-10-18 16:16:12,697] INFO [kafka-log-cleaner-thread-0]:
kafka-1 | Log cleaner thread 0 cleaned log snuba-commit-log-0 (dirty section = [119320, 120190])
kafka-1 | 0.2 MB of log processed in 0.3 seconds (0.5 MB/sec).
kafka-1 | Indexed 0.1 MB in 0.2 seconds (0.8 Mb/sec, 62.5% of total time)
kafka-1 | Buffer utilization: 0.0%
kafka-1 | Cleaned 0.2 MB in 0.1 seconds (1.4 Mb/sec, 37.5% of total time)
kafka-1 | Start size: 0.2 MB (871 messages)
kafka-1 | End size: 0.0 MB (1 messages)
kafka-1 | 99.9% size reduction (99.9% fewer messages)
kafka-1 | (kafka.log.LogCleaner)
kafka-1 | [2024-10-18 16:16:12,717] INFO Cleaner 0: Beginning cleaning of log snuba-generic-events-commit-log-0 (kafka.log.LogCleaner)
kafka-1 | [2024-10-18 16:16:12,717] INFO Cleaner 0: Building offset map for snuba-generic-events-commit-log-0... (kafka.log.LogCleaner)
kafka-1 | [2024-10-18 16:16:12,752] INFO Cleaner 0: Building offset map for log snuba-generic-events-commit-log-0 for 1 segments in offset range [23772, 24043). (kafka.log.LogCleaner)
kafka-1 | [2024-10-18 16:16:12,756] INFO Cleaner 0: Offset map for log snuba-generic-events-commit-log-0 complete. (kafka.log.LogCleaner)
kafka-1 | [2024-10-18 16:16:12,757] INFO Cleaner 0: Cleaning log snuba-generic-events-commit-log-0 (cleaning prior to Fri Oct 18 15:14:33 UTC 2024, discarding tombstones prior to upper bound deletion horizon Thu Oct 17 14:16:32 UTC 2024)... (kafka.log.LogCleaner)
kafka-1 | [2024-10-18 16:16:12,757] INFO Cleaner 0: Cleaning LogSegment(baseOffset=0, size=187, lastModifiedTime=1729260992425, largestRecordTimestamp=Some(1729260992420)) in log snuba-generic-events-commit-log-0 into 0 with an upper bound deletion horizon 1729174592425 computed from the segment last modified time of 1729260992425,retaining deletes. (kafka.log.LogCleaner)
kafka-1 | [2024-10-18 16:16:12,760] INFO Cleaner 0: Cleaning LogSegment(baseOffset=23772, size=50650, lastModifiedTime=1729264473871, largestRecordTimestamp=Some(1729264473866)) in log snuba-generic-events-commit-log-0 into 0 with an upper bound deletion horizon 1729174592425 computed from the segment last modified time of 1729264473871,retaining deletes. (kafka.log.LogCleaner)
kafka-1 | [2024-10-18 16:16:12,777] INFO Cleaner 0: Swapping in cleaned segment LogSegment(baseOffset=0, size=186, lastModifiedTime=1729264473871, largestRecordTimestamp=Some(1729264473866)) for segment(s) List(LogSegment(baseOffset=0, size=187, lastModifiedTime=1729260992425, largestRecordTimestamp=Some(1729260992420)), LogSegment(baseOffset=23772, size=50650, lastModifiedTime=1729264473871, largestRecordTimestamp=Some(1729264473866))) in log Log(dir=/var/lib/kafka/data/snuba-generic-events-commit-log-0, topicId=_MM1RCNrSgKed2ujMsr6bA, topic=snuba-generic-events-commit-log, partition=0, highWatermark=24259, lastStableOffset=24259, logStartOffset=0, logEndOffset=24259) (kafka.log.LogCleaner)
kafka-1 | [2024-10-18 16:16:12,791] INFO [kafka-log-cleaner-thread-0]:
kafka-1 | Log cleaner thread 0 cleaned log snuba-generic-events-commit-log-0 (dirty section = [23772, 24043])
kafka-1 | 0.0 MB of log processed in 0.1 seconds (0.7 MB/sec).
kafka-1 | Indexed 0.0 MB in 0.0 seconds (1.2 Mb/sec, 54.2% of total time)
kafka-1 | Buffer utilization: 0.0%
kafka-1 | Cleaned 0.0 MB in 0.0 seconds (1.5 Mb/sec, 45.8% of total time)
kafka-1 | Start size: 0.0 MB (272 messages)
kafka-1 | End size: 0.0 MB (1 messages)
kafka-1 | 99.6% size reduction (99.6% fewer messages)
kafka-1 | (kafka.log.LogCleaner)
kafka-1 | [2024-10-18 16:22:57,932] INFO Cleaner 0: Beginning cleaning of log snuba-transactions-commit-log-0 (kafka.log.LogCleaner)
kafka-1 | [2024-10-18 16:22:57,948] INFO Cleaner 0: Building offset map for snuba-transactions-commit-log-0... (kafka.log.LogCleaner)
kafka-1 | [2024-10-18 16:22:58,795] INFO Cleaner 0: Building offset map for log snuba-transactions-commit-log-0 for 1 segments in offset range [613697, 616114). (kafka.log.LogCleaner)
kafka-1 | [2024-10-18 16:22:58,833] INFO Cleaner 0: Offset map for log snuba-transactions-commit-log-0 complete. (kafka.log.LogCleaner)
kafka-1 | [2024-10-18 16:22:58,835] INFO Cleaner 0: Cleaning log snuba-transactions-commit-log-0 (cleaning prior to Fri Oct 18 15:22:56 UTC 2024, discarding tombstones prior to upper bound deletion horizon Thu Oct 17 14:22:56 UTC 2024)... (kafka.log.LogCleaner)
kafka-1 | [2024-10-18 16:22:58,846] INFO Cleaner 0: Cleaning LogSegment(baseOffset=0, size=182, lastModifiedTime=1729261376009, largestRecordTimestamp=Some(1729261376005)) in log snuba-transactions-commit-log-0 into 0 with an upper bound deletion horizon 1729174976009 computed from the segment last modified time of 1729261376009,retaining deletes. (kafka.log.LogCleaner)
kafka-1 | [2024-10-18 16:22:58,850] INFO Cleaner 0: Cleaning LogSegment(baseOffset=613697, size=438280, lastModifiedTime=1729264976760, largestRecordTimestamp=Some(1729264976756)) in log snuba-transactions-commit-log-0 into 0 with an upper bound deletion horizon 1729174976009 computed from the segment last modified time of 1729264976760,retaining deletes. (kafka.log.LogCleaner)
kafka-1 | [2024-10-18 16:22:58,890] INFO Cleaner 0: Swapping in cleaned segment LogSegment(baseOffset=0, size=182, lastModifiedTime=1729264976760, largestRecordTimestamp=Some(1729264976756)) for segment(s) List(LogSegment(baseOffset=0, size=182, lastModifiedTime=1729261376009, largestRecordTimestamp=Some(1729261376005)), LogSegment(baseOffset=613697, size=438280, lastModifiedTime=1729264976760, largestRecordTimestamp=Some(1729264976756))) in log Log(dir=/var/lib/kafka/data/snuba-transactions-commit-log-0, topicId=WYFjiBlsTaGqEc127bJ9zw, topic=snuba-transactions-commit-log, partition=0, highWatermark=617808, lastStableOffset=617808, logStartOffset=0, logEndOffset=617808) (kafka.log.LogCleaner)
kafka-1 | [2024-10-18 16:22:58,918] INFO [kafka-log-cleaner-thread-0]:
kafka-1 | Log cleaner thread 0 cleaned log snuba-transactions-commit-log-0 (dirty section = [613697, 616114])
kafka-1 | 0.4 MB of log processed in 1.0 seconds (0.4 MB/sec).
kafka-1 | Indexed 0.4 MB in 0.9 seconds (0.5 Mb/sec, 91.8% of total time)
kafka-1 | Buffer utilization: 0.0%
kafka-1 | Cleaned 0.4 MB in 0.1 seconds (5.3 Mb/sec, 8.2% of total time)
kafka-1 | Start size: 0.4 MB (2,418 messages)
kafka-1 | End size: 0.0 MB (1 messages)
kafka-1 | 100.0% size reduction (100.0% fewer messages)
kafka-1 | (kafka.log.LogCleaner)
It seems multiple queues/topics keep growing, not a single one (largest is ingest-transactions-0 with 21gb)
4.0K bootstrap.checkpoint
12K buffered-segments-0
12K buffered-segments-dlq-0
12K cdc-0
4.0K cleaner-offset-checkpoint
51M __cluster_metadata-0
39M __consumer_offsets-0
12K eap-spans-subscription-results-0
40K event-replacements-0
373M events-0
4.2M events-subscription-results-0
969M generic-events-0
12K generic-metrics-subscription-results-0
340K group-attributes-0
12K ingest-attachments-0
12K ingest-attachments-dlq-0
930M ingest-events-0
12K ingest-events-dlq-0
12K ingest-feedback-events-0
12K ingest-feedback-events-dlq-0
12K ingest-generic-metrics-dlq-0
51M ingest-metrics-0
12K ingest-metrics-dlq-0
564K ingest-monitors-0
94M ingest-occurrences-0
3.1G ingest-performance-metrics-0
12M ingest-replay-events-0
81M ingest-replay-recordings-0
21G ingest-transactions-0
12K ingest-transactions-dlq-0
4.0K log-start-offset-checkpoint
4.0K meta.properties
12K metrics-subscription-results-0
12K monitors-clock-tasks-0
524K monitors-clock-tick-0
3.5M outcomes-0
128M outcomes-billing-0
12K outcomes-billing-dlq-0
12K outcomes-dlq-0
380K processed-profiles-0
420M profiles-0
844K profiles-call-tree-0
4.0K recovery-point-offset-checkpoint
4.0K replication-offset-checkpoint
12K scheduled-subscriptions-eap-spans-0
12K scheduled-subscriptions-events-0
12K scheduled-subscriptions-generic-metrics-counters-0
12K scheduled-subscriptions-generic-metrics-distributions-0
12K scheduled-subscriptions-generic-metrics-gauges-0
12K scheduled-subscriptions-generic-metrics-sets-0
12K scheduled-subscriptions-metrics-0
12K scheduled-subscriptions-transactions-0
12K shared-resources-usage-0
252K snuba-commit-log-0
12K snuba-dead-letter-generic-events-0
12K snuba-dead-letter-generic-metrics-0
12K snuba-dead-letter-group-attributes-0
12K snuba-dead-letter-metrics-0
12K snuba-dead-letter-querylog-0
12K snuba-dead-letter-replays-0
12K snuba-eap-mutations-0
12K snuba-eap-spans-commit-log-0
104K snuba-generic-events-commit-log-0
1.5G snuba-generic-metrics-0
284K snuba-generic-metrics-counters-commit-log-0
292K snuba-generic-metrics-distributions-commit-log-0
280K snuba-generic-metrics-gauges-commit-log-0
276K snuba-generic-metrics-sets-commit-log-0
25M snuba-metrics-0
216K snuba-metrics-commit-log-0
12K snuba-metrics-summaries-0
12K snuba-profile-chunks-0
12K snuba-queries-0
3.2G snuba-spans-0
448K snuba-transactions-commit-log-0
3.5G transactions-0
12K transactions-subscription-results-0
12K uptime-configs-0
12K uptime-results-0
Issues and transactions are processed correctly, I don't see any delays in processing. Not sure how to check whether there is some queue piling up. The load average is ~3 on 8vCPU instance.
Honestly, I'm not sure whether this is normal, we have recently added a noisy project. It's just suspicious, since it didn't take this much space in the past.
I'll be happy to provide more details if you point me where to look / what to run.
Thank you 🙏
Event ID
No response
