ts-odd
ts-odd copied to clipboard
Gracefully handle missing data in the MMPT
Context
This came up while working on #211. The root issue means that the MMPT is missing entries. More concretely:
- The MMPT is basically a key-value store. The values are (CIDs to) blobs of encrypted data.
- It stores both file metadata and file content, but they're at different (MMPT) keys, because they're encrypted with different (AES) keys.
- A malicious/erroneous client can upload something to the fission server that messes up the MMPTs structure. E.g. something that looks like file metadata, but without adding the actual file to the MMPT.
- The fission server has no way of detecting this, as everything is encrypted.
We basically had this happen all the time (missing private file contents) because of an error that dropped MMPT entries.
Impact
At the moment, if a private file of which the content blob is missing from the MMPT is requested, webnative will get stuck.
This causes lots of issues:
- Drive loads indefinitely because it's (probably) trying to fetch private directory metadata, but some of that was missing (due to the bug)
- The auth lobby doesn't finish "warming up", because it's trying to look for existing private files/directories to give permission to. Again, there can be some entries in the MMPT missing, causing the CID lookups to go nowhere
- Apps don't work, because they're trying to look at some private file, but that file just never loads
- Private file history has "holes" in it. Some entries are just missing (at least in this case we finish our request) (Side note: The private file history works without failing, because it looks up the files by MMPT key - the name filter - instead of by CID to some blob, of which we just expect it to also be stored in the MMPT)
Solution
We should make webnative resilient against malformed private file content/missing keys in the MMPT.
I can think of two things we could do:
- When looking up things by CID in js-ipfs, we always carry around a timeout
- When we look up the file's contents, we don't look it up by CID, but instead by MMPT key (& match that against the stored CID)
I'm not sure how what we do in the case we find out there's an error, though. Do we throw an exception? Do we revert back to the last CID? Do we throw away malformed data or do we work around it?