release-cloudflare-worker icon indicating copy to clipboard operation
release-cloudflare-worker copied to clipboard

src, scripts: use kv to cache directories

Open flakey5 opened this issue 2 months ago • 0 comments

Makes use of KV for directory listings. This aims to replace the build-r2-symlinks.mjs file with two different ones:

  • build-directory-cache.mjs - This reads the entire dist-prod bucket and caches it into the KV namespace. This is pretty much only for the initial setup of the cache and for anytime we need to rebuild all of it.
  • update-directory-cache.mjs - This reads only the necessary paths from the dist-prod bucket and updates them in the KV namespace. This will be the one that we will call in the workflow

A USE_KV environment variable is added to the worker that acts as a toggle between KV and S3 listings. This acts as a safeguard in case something goes wrong while we're still testing it, so we can easily switch between the two. Once we're more confident in KV, we can remove it + the entire S3Provider from the worker.

Closes #159 Closes #314

TODO:

  • [ ] Functionality
    • [ ] Make sure we're doing everything we need to in both scripts
    • [ ] Keep updating files that build-r2-symlinks.mjs updates so we can switch back to S3 if needed
    • [ ] Update workflows
  • [ ] Testing
    • [ ] Writing some actual tests for the scripts
    • [ ] Ensure we can switch between using KV and S3 if needed
  • [ ] Create KV namespace and update the wrangler config

flakey5 avatar Nov 15 '25 04:11 flakey5