rmfakecloud icon indicating copy to clipboard operation
rmfakecloud copied to clipboard

Refactor storage

Open nemunaire opened this issue 7 months ago • 0 comments

We know that reMarkable uses Google Storage as the backend for its sync service. Many features we observe are inherited directly from the capabilities provided by Google Storage.

In particular, routes like /sync/v3/files/abcdef123456 are effectively aliases for Google Storage routes. The logic behind these routes should remain independent from other routes, to allow using an object storage (e.g., S3) instead of a local filesystem.

However, the routes /sync/v3/root and /sync/v4/root are not tied to object storage. The "root" object, along with its generation number, is used to help merging concurrent modifications, something not handled by Google Storage itself.

Currently, the generation number is returned by the LoadBlob function, even though it's only meaningful for the root. Individual files don't have a generation number. The logic handling the root object and its history should be separated from the file storage logic. The current root index is closer to user state than to a file/blob.

What This PR Changes

  • I’ve extracted the root-specific logic from LoadBlob and introduced two new functions: GetRoot and UpdateRoot. These operate on the user storage layer, not the blob storage layer.

  • Some types from the storage package were moved into the models package to allow them to be shared across different storage implementations.

  • The last commit introduces a new storage backend abstraction for the root index: the RootStorer interface, responsible for managing the root index, its generation number, and the cached tree.

  • All abstractions are now cleanly separated to make it easier to implement alternative storage backends. For example: a database backend for user storage, or a S3-compatible blob storage. All go interface could have a different implementation (users in database, root indexes in filesystem, blobs in S3, ...).

  • All filesystem-related calls are now isolated in the fs package. The top-level storage package now only contains interfaces and logic common to all storage implementations.


Overall, this pull request should not change any user-facing behavior. Please test it with your usual workflow and report any unexpected issues or regressions.

nemunaire avatar Jun 28 '25 08:06 nemunaire