HDDS-10239. Storage Container Reconciliation.

[errose28]

What changes were proposed in this pull request?

Design proposal for container reconciliation. Please comment inline on the markdown document to ask questions and post feedback. Switch to Rich Diff mode for smoother reading.

What is the link to the Apache JIRA

HDDS-10239

How was this patch tested?

N/A

[errose28]

cc @kerneltime and @sodonnel who also worked on this design and document.

[kerneltime]

Similarly to closed containers do we have any goals or limitations for open containers ?

The overall goal is to simplify the post open state handing. Once containers are either open or post open we can revisit how to avoid closing containers and only treating them as full. This should lead to another order of simplification but for this design doc it is out of scope.

[kerneltime]

Thanks @xichen01 for the review.

[ChenSammi]

Thanks @errose28 , @kerneltime and @sodonnel for working on this. Except the benefits mentioned in the document, I believe there is another big benefit which is we can significantly reduce the IO resources used in container replica recovery(container replication) , by copying the whole container replica to copying a few related blocks. It will help to improve the cluster wise overall stability and performance.