error while opening existing TSDBs

[cycwll]

My cortex running on kubernetes as statefulset instance, and uses a NFS as local filesystem block storage. At first it worked steadily. However, after running for about 20 days, Cortex has been restarting and the following errors are reported

level=error ts=2022-02-07T01:38:42.285835494Z caller=ingester_v2.go:1813 msg="error while opening existing TSDBs" err="unable to open TSDB for user BSIRS: failed to compact TSDB: /data/cortex/tsdb/BSIRS: plan compaction: open /data/cortex/tsdb/BSIRS/01FTV12WVF6RCCQ5CYCZCMQ3BV/meta.json: no such file or directory" level=error ts=2022-02-07T01:38:42.330499027Z caller=cortex.go:448 msg="module failed" module=ingester-service err="invalid service state: Failed, expected: Running, failure: opening existing TSDBs: unable to open TSDB for user BSIRS: failed to compact TSDB: /data/cortex/tsdb/BSIRS: plan compaction: open /data/cortex/tsdb/BSIRS/01FTV12WVF6RCCQ5CYCZCMQ3BV/meta.json: no such file or directory"

I noticed that there may be the following reasons？ The block storage of cortex is based on Prometheus TSDB. Because Prometheus TSDB does not support NFS, the block storage of cortex does not support NFS. Is my understanding right? I want to confirm whether the local block storage of cortex does not support NFS.

[friedrich-at-adobe]

yes, you are correct. Cortex uses the same tsdb format as prometheus, so you can expect same behavior in the ingesters.

NFS is supported, to my understanding, if it is POSIX-complaint

Read more about it here https://prometheus.io/docs/prometheus/latest/storage/#operational-aspects

[stale[bot]]

This issue has been automatically marked as stale because it has not had any activity in the past 60 days. It will be closed in 15 days if no further activity occurs. Thank you for your contributions.