Store Gateway refresh doesn't add new tenant with Shuffled/default sharding strategy

[vennilaa]

Shuffled-sharding strategy is enabled for ingester, compactor and store gateway with tenant size set for new tenant (which has no blocks in object storage) deployed with microservices mode. Afterwards ingesters started receiving samples from a new tenant and observed azure blob object storage has blocks for new tenant shipped by ingesters in 2+ hours Compactor started loading blocks of new tenant from object storage on periodic refresh. Issue: Whereas store-gateway scan doesn't load index header of new tenant as refresh seems to consider only tenants available in blob on its initialisation. Say., store gateway instances=4, with default shard size=1 and new tenant shrd size overidden as 2, observing all tenants available in blob from store gateway initialisation uses 3 instances distributed with one instance empty On redeployment, store gateway started loading index header of new tenant as well. Ie., On start, it started downloading bucket index of new tenant, whereas refresh haven't Expected behavior: Store gateway refresh should load index header of new tenant as well and store in its shard.

Environment:

• Infrastructure: Kubernetes istio environment
• Microservices deployment

[alanprot]

Interesting ..

So if you don’t restart store gateway you cannot query historical data ?

[vennilaa]

Actually periodic refresh couldn't identify and load the new tenant in the object storage. It's basically refresh missing to identify shard for new tenant on runtime

[vennilaa]

This issue is observed in default sharding as well

[stale[bot]]

This issue has been automatically marked as stale because it has not had any activity in the past 60 days. It will be closed in 15 days if no further activity occurs. Thank you for your contributions.

[yeya24]

Still valid.

[stale[bot]]

This issue has been automatically marked as stale because it has not had any activity in the past 60 days. It will be closed in 15 days if no further activity occurs. Thank you for your contributions.