go icon indicating copy to clipboard operation
go copied to clipboard

exp/lighthorizon: Fetch ledgers in parallel whenever possible

Open Shaptic opened this issue 3 years ago • 0 comments

Parent epic: #4317.


TL;DR: Concurrent GetLedger() downloads should be implemented wherever possible.

The handful of Horizon Lite endpoints currently process requests in a single-threaded pipeline: get the ledger, extract the information we care about, add it to the response, repeat until done.

Ledger downloads are a big portion of overall response latency, yet these downloads can be done in parallel while ledgers are read/processed. Caching helps with reducing the amount of downloads, but concurrent downloads ensure network bandwidth is saturated as much as possible and minimize processing "downtime".

There is a big open question (though instinct says "yes") about whether or not this is a worthwhile optimization. Since pubnet ledgers are fairly large, doing processing on them while the network does nothing, even though we know we'll need more downloads later, seems like it would be worth it.

This is especially important for checkpoint-based indexing, since we know for a fact that we need to look through at least 64 ledgers.

Shaptic avatar Jul 20 '22 19:07 Shaptic