node icon indicating copy to clipboard operation
node copied to clipboard

Snapshot Restore could benefit from download and untar in a single step

Open CryptoFewka opened this issue 1 year ago • 1 comments

Rather than downloading the archive snapshot in one step, and then unpacking it in another step, we should consider streaming the download in to tar through the pipeline so that the overall process completes more quickly.

https://github.com/zeta-chain/node/blob/3eba70102a12be9ab55f0939b1a5260251ef2d5c/contrib/docker-scripts/start.sh#L140-L144

CryptoFewka avatar Sep 24 '24 14:09 CryptoFewka

To be explicit, it will also cut the free disk space required in half, because it won't need to download first, and then unpack.

Note: this change would cause the download to be unable to resume if interrupted. I do not believe that the current process will use an already downloaded, or partially downloaded snapshot currently though.

We would need to decide which is better:

  1. A process that is slower and requires roughly 9 TB of free disk space.
  2. A process that is faster, requires 4.5 TB of free disk space, but cannot tolerate download interruptions

CryptoFewka avatar Sep 24 '24 14:09 CryptoFewka