pinot
pinot copied to clipboard
[DRAFT] Implementing federation for logical tables in Pinot
Overview
This PR introduces federation capabilities in Apache Pinot’s broker layer, enabling a single broker to connect to and route queries across multiple independent clusters.
The change is additive and non-intrusive—it does not affect existing single-cluster functionality.
Key Points
- This does not break existing single-cluster setup, the new configurations
pinot.secondary.cluster.namesandpinot.zk.server.secondary.<cluster>enable federation support. - The broker connects to the Helix clusters of the secondary clusters as a spectator only, to avoid showing up in the cross cluster and being included in rate-limit calculations etc.
- The
federatedRoutingManagercontains all the individualbrokerRoutingManagers, one for each ZK cluster we connected to. It also has agetPrimaryRoutingManagerto return the cluster's own routingmanager whenever the query optionenableFederationis not present or not true.