pinot icon indicating copy to clipboard operation
pinot copied to clipboard

[DRAFT] Implementing federation for logical tables in Pinot

Open shauryachats opened this issue 3 months ago • 2 comments

Overview

This PR introduces federation capabilities in Apache Pinot’s broker layer, enabling a single broker to connect to and route queries across multiple independent clusters.
The change is additive and non-intrusive—it does not affect existing single-cluster functionality.


Key Points

  • This does not break existing single-cluster setup, the new configurations pinot.secondary.cluster.names and pinot.zk.server.secondary.<cluster> enable federation support.
  • The broker connects to the Helix clusters of the secondary clusters as a spectator only, to avoid showing up in the cross cluster and being included in rate-limit calculations etc.
  • The federatedRoutingManager contains all the individual brokerRoutingManagers, one for each ZK cluster we connected to. It also has a getPrimaryRoutingManager to return the cluster's own routingmanager whenever the query option enableFederation is not present or not true.

shauryachats avatar Nov 05 '25 03:11 shauryachats