celestia-node
celestia-node copied to clipboard
fix(eds/store): IPLD blockstore cleanup
Individual IPLD blocks are stored in the edsstore's blockstore as of #2532. These blocks need to eventually be cleaned up. There are two approaches we briefly discussed:
- GC approach: Every X minutes/hours all blocks under the datastore prefix are purged. The non-trivial part here is that it may remove blocks that are currently being used for reconstruction.
- Post-reconstruction removal: Trigger removal of all of the IPLD blocks of an EDS upon successful retrieval. This can be done by adding a hook somewhere where the CIDs are known (such as inside retriever.go or after indexing). The non-trivial part here is not making it extremely ugly
shwap
Initially, shwap aimed to address this by implementing an enhanced retriever. However, following an internal discussion on shwap's scope, we decided against including retriever improvements. Therefore, this issue will need to be addressed separately from shwap.