getsentry
Sentry 24.4.2 and severe log spam from clickhouse/snuba
Self-Hosted Version
24.4.2
CPU Architecture
x86_64
Docker Version
26.1.2
Docker Compose Version
2.27.0
Steps to Reproduce
Ensure you blocked HTTP and HTTPS to your sentry instance.
Simply restart everything with docker compose up -d --force-recreate
Look at the logs.
Expected Result
Logs contain valuable information
Actual Result
Logs are constantly spammed with a high volume of:
snuba-generic-metrics-sets-consumer-1 | {"timestamp":"2024-05-10T21:52:21.842453Z","level":"INFO","fields":{"message":"skipping write of 0 rows"},"target":"rust_snuba::strategies::clickhouse"}
snuba-issue-occurrence-consumer-1 | {"timestamp":"2024-05-10T21:52:22.134422Z","level":"INFO","fields":{"message":"skipping write of 0 rows"},"target":"rust_snuba::strategies::clickhouse"}
snuba-generic-metrics-counters-consumer-1 | {"timestamp":"2024-05-10T21:52:22.165962Z","level":"INFO","fields":{"message":"skipping write of 0 rows"},"target":"rust_snuba::strategies::clickhouse"}
clickhouse-1 | 2024.05.10 21:52:22.166625 [ 300 ] {} <Error> ServerErrorHandler: Poco::Exception. Code: 1000, e.code() = 107, e.displayText() = Net Exception: Socket is not connected, Stack trace (when copying this message, always include the lines below):
clickhouse-1 |
clickhouse-1 | 0. Poco::Net::SocketImpl::error(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) @ 0x13c4ee8e in /usr/bin/clickhouse
clickhouse-1 | 1. Poco::Net::SocketImpl::peerAddress() @ 0x13c510d6 in /usr/bin/clickhouse
clickhouse-1 | 2. DB::HTTPServerRequest::HTTPServerRequest(std::__1::shared_ptr<DB::Context const>, DB::HTTPServerResponse&, Poco::Net::HTTPServerSession&) @ 0x110e6f0b in /usr/bin/clickhouse
clickhouse-1 | 3. DB::HTTPServerConnection::run() @ 0x110e5d6e in /usr/bin/clickhouse
clickhouse-1 | 4. Poco::Net::TCPServerConnection::start() @ 0x13c5614f in /usr/bin/clickhouse
clickhouse-1 | 5. Poco::Net::TCPServerDispatcher::run() @ 0x13c57bda in /usr/bin/clickhouse
clickhouse-1 | 6. Poco::PooledThread::run() @ 0x13d89e59 in /usr/bin/clickhouse
clickhouse-1 | 7. Poco::ThreadImpl::runnableEntry(void*) @ 0x13d860ea in /usr/bin/clickhouse
clickhouse-1 | 8. start_thread @ 0x9609 in /usr/lib/x86_64-linux-gnu/libpthread-2.31.so
clickhouse-1 | 9. __clone @ 0x122293 in /usr/lib/x86_64-linux-gnu/libc-2.31.so
clickhouse-1 | (version 21.8.13.1.altinitystable (altinity build))
snuba-transactions-consumer-1 | {"timestamp":"2024-05-10T21:52:22.204371Z","level":"INFO","fields":{"message":"skipping write of 0 rows"},"target":"rust_snuba::strategies::clickhouse"}
clickhouse-1 | 2024.05.10 21:52:22.239660 [ 289 ] {} <Error> ServerErrorHandler: Poco::Exception. Code: 1000, e.code() = 107, e.displayText() = Net Exception: Socket is not connected, Stack trace (when copying this message, always include the lines below):
clickhouse-1 |
clickhouse-1 | 0. Poco::Net::SocketImpl::error(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) @ 0x13c4ee8e in /usr/bin/clickhouse
clickhouse-1 | 1. Poco::Net::SocketImpl::peerAddress() @ 0x13c510d6 in /usr/bin/clickhouse
clickhouse-1 | 2. DB::HTTPServerRequest::HTTPServerRequest(std::__1::shared_ptr<DB::Context const>, DB::HTTPServerResponse&, Poco::Net::HTTPServerSession&) @ 0x110e6f0b in /usr/bin/clickhouse
clickhouse-1 | 3. DB::HTTPServerConnection::run() @ 0x110e5d6e in /usr/bin/clickhouse
clickhouse-1 | 4. Poco::Net::TCPServerConnection::start() @ 0x13c5614f in /usr/bin/clickhouse
clickhouse-1 | 5. Poco::Net::TCPServerDispatcher::run() @ 0x13c57bda in /usr/bin/clickhouse
clickhouse-1 | 6. Poco::PooledThread::run() @ 0x13d89e59 in /usr/bin/clickhouse
clickhouse-1 | 7. Poco::ThreadImpl::runnableEntry(void*) @ 0x13d860ea in /usr/bin/clickhouse
clickhouse-1 | 8. start_thread @ 0x9609 in /usr/lib/x86_64-linux-gnu/libpthread-2.31.so
clickhouse-1 | 9. __clone @ 0x122293 in /usr/lib/x86_64-linux-gnu/libc-2.31.so
clickhouse-1 | (version 21.8.13.1.altinitystable (altinity build))
snuba-metrics-consumer-1 | {"timestamp":"2024-05-10T21:52:22.239374Z","level":"INFO","fields":{"message":"skipping write of 0 rows"},"target":"rust_snuba::strategies::clickhouse"}
Event ID
No response
I created this ticket following comments from @tm-artemb, @peektoseen and @krzysieqq here: https://github.com/getsentry/self-hosted/issues/2963#issuecomment-2072973387
The spam is caused by the rust consumer. Please refer to this comment https://github.com/getsentry/self-hosted/issues/2876#issuecomment-2006287794 or https://github.com/getsentry/snuba/issues/5707#issuecomment-2027710056
If changing that fixed the problem, please let me know, and I'll close this issue :D
If changing that fixed the problem, please let me know, and I'll close this issue :D
Reverting to Python fixes the issue, but there is a real bug to be fixed in snuba rust consumers. Maybe move the issue on the snuba repo
If changing that fixed the problem, please let me know, and I'll close this issue :D
Reverting to Python fixes the issue, but there is a real bug to be fixed in snuba rust consumers. Maybe move the issue on the snuba repo
The issue's already there, see https://github.com/getsentry/snuba/issues/5707
Thank you for the help here. This can be closed and I'll monitor the one in the snuba project