getsentry
Fix metadata synchronization issues in Snuba migrations for multi-replica/shard ClickHouse clusters
Problem: The Snuba migration system encountered significant stability issues when running migrations in ClickHouse clusters with multiple replicas and shards. Synchronization errors, particularly due to outdated metadata on zookeeper, frequently caused migrations to fail or require manual intervention.
Solution:
-
Retry Mechanism: Added a RetryOnSyncError class that retries operations up to 30 seconds when synchronization errors are encountered (Metadata is not up to date). This mechanism has been integrated into critical migration operations (AddColumn, DropColumn, ModifyColumn, etc.) to handle transient synchronization issues gracefully.
-
DROP TABLE with SYNC: Updated DROP TABLE operations to include the SYNC keyword, ensuring consistency.
Impact: These changes significantly enhance the robustness of the Snuba migration system when operating in distributed ClickHouse environments with complex replication setups. By automatically handling synchronization delays, the risk of migration failures is reduced, minimizing operational overhead and downtime.
Testing:
- Verified migrations in a multi-replica, multi-shard ClickHouse cluster.
Legal Boilerplate
Look, I get it. The entity doing business as "Sentry" was incorporated in the State of Delaware in 2015 as Functional Software, Inc. and is gonna need some rights from me in order to utilize my contributions in this here PR. So here's the deal: I retain all rights, title and interest in and to my contributions, and by keeping this boilerplate intact I confirm that Sentry can use, modify, copy, and redistribute my contributions, under Sentry's choice of terms.
Could you please check this as soon as possible? This is very necessary, especially when running it on multiple instances of Clickhouse.
It's getting close to a year and i think this is important for a lot of people that are self-hosting this. hopefully you will review it.
It's getting close to a year and i think this is important for a lot of people that are self-hosting this. hopefully you will review it.
Hi @pyhp2017 - I'll be reviewing this and will hopefully get back to you by EOW or early next week. Thanks again for putting this up PR and raising this issue!
@pyhp2017 Hey there, I ran into some issue trying to re-run the checks on this PR, just made a new PR https://github.com/getsentry/snuba/pull/7280 that is a copy of yours. Had to fix a test but I think after those pass we should be good to go.
going to close this since https://github.com/getsentry/snuba/pull/7280 is merged