celestia-node icon indicating copy to clipboard operation
celestia-node copied to clipboard

fix(eds/store): IPLD blockstore cleanup

Open distractedm1nd opened this issue 2 years ago • 2 comments

Individual IPLD blocks are stored in the edsstore's blockstore as of #2532. These blocks need to eventually be cleaned up. There are two approaches we briefly discussed:

  1. GC approach: Every X minutes/hours all blocks under the datastore prefix are purged. The non-trivial part here is that it may remove blocks that are currently being used for reconstruction.
  2. Post-reconstruction removal: Trigger removal of all of the IPLD blocks of an EDS upon successful retrieval. This can be done by adding a hook somewhere where the CIDs are known (such as inside retriever.go or after indexing). The non-trivial part here is not making it extremely ugly

distractedm1nd avatar Aug 21 '23 14:08 distractedm1nd

shwap

ramin avatar Feb 29 '24 12:02 ramin

Initially, shwap aimed to address this by implementing an enhanced retriever. However, following an internal discussion on shwap's scope, we decided against including retriever improvements. Therefore, this issue will need to be addressed separately from shwap.

walldiss avatar Mar 27 '24 15:03 walldiss