Call for input: Performance tricks for batch query multithreaded contract query

[banacer]

Let's say I want to download all the uniswap v2 pairs with their reserves. One will do the following: 1- create the uniswap v2 contract object query, then contract.query AllPairsLength 2- foreach i in AllPairsLength contract.query allPairs(i) => this returns an address 2.1 - create a uniswapv2Pair contract using the address returned 2.2 - query the contract for functions token0 and token1 and getReserves given there are 70k+ pairs, this can be done concurrently, one can simply create a create 10+ threads to concurrently fetch 1000 address per batch and for each of them create batch a batch to query 2.2

I have done something similar to what is described above in both golang (using github.com/lmittmann/w3) and this crate in rust but I have noticed that the rust version is at least 10x slower than golang. Both are running in the same machine query an erigon full node. I have done the following approaches to speed things up: 1- create a single WebSocket and create a web3<batch> and cloned it in every tokio threac 2- created a single WebSocket to get the AllPairsLength but created new WebSocket object inside every thread to have a unique websocket connection. I create this in step 2- of the pseudo code above.

option 2 was even slower. I am not sure why there is such a large performance difference between my rust implementation and golang implementation and I figured I may be using this crate in the wrong way. Can you provide guidance as to whether we should use the same websocket or create a new websocket for faster performance? Any tricks to speed things up.

Appreciate your help and apologies for posting it here as I am not aware of another medium to share this