persister-durable-object-sql-storage times out in fragmented mode with 200kb+

[willhoney7]

Describe the bug

I've been using the persister-durable-object-sql-storage for a little while and all of a sudden I've started getting cloudflare errors, "Durable Object storage operation exceeded timeout which caused object to be reset." I've been using "fragmented" mode, which, after reading the code, I'm noticing is storing every individual CELL separately. I switched to JSON mode and it no longer times out, but I've realized I'm at 225kb of data. I think I'll eventually be hitting that 2MB row limit as time continues on (my app gets more data each day).

What do you think about updating fragmented mode to be per row, instead of per cell? I imagine that would greatly increase write speed (and decrease write costs!), and also help with row limits.

I don't think there's really a good 200kb+ storage option right now here.

Also going to tag @acoreyj since he was the original author of the sql storage implementation.

[mokone-september]

Thanks for raising this — I’ve run into the same problem and I think the per-cell fragmentation is the main culprit.

Proposal: What if fragmented mode stored one row per storage entry instead of one cell?

• This would massively reduce the number of writes (less chance of timeouts, lower cost).

• Still keeps things granular enough at the row level.

To future-proof, larger rows could be automatically chunked if they get close to the 2MB limit (e.g. rowId:chunk0, rowId:chunk1).

This hybrid strategy would:

• Solve current timeout issues for 200kb+ datasets.

• Delay hitting the 2MB limit.

• Scale better for apps that accumulate daily data over time.

Happy to help experiment with this approach if maintainers think it’s worth exploring.

[acoreyj]

@mokone-september @willhoney7

Do one of you have a simple recreation. You could adapt https://github.com/acoreyj/tinybase-tester.

I wasn't able to recreate the issue using this which just lets you add a bunch of rows to some tables.

I could try pushing it harder though.

[willhoney7]

Sorry, I was going to try to replicate with the repo you shared but at this point I’m rethinking my whole database situation. I’m just not sure tinybase is well equipped for apps that accumulate data over time. My stores are ~1MB now and will keep growing every day. I think after a year or two they could be 10-20MB. Seems like I might be trying to use the wrong tech.

I do still think it would a good idea to update the fragmented approach to be per row for anyone trying to use it. And who knows, maybe I do stick with tinybase...

[mokone-september]

Thanks, @willhoney7 — totally get it. I’m already working on a row-based persistence prototype that stores one row per storage entry (with optional chunking/compression for very large rows).

Want me to help implement this step-by-step? I can:

• Fork the tester and add a persister-row-mode prototype branch.

• Add a reproducible test script (packages/astro/src/scripts/testRowMode.ts) that writes ~200–500KB and reports whether the Durable Object timed out.

• Share the fork/branch and exact commands so you (or @acoreyj) can run it locally or deploy to Cloudflare.

• If results look good, open a PR with the implementation, tests, and docs — or iterate with you on improvements (chunking, compression, cleanup logic).

Happy to start and share the branch + run instructions — let me know if you’d prefer I open a draft PR or just drop the fork link here.

[mokone-september]

Everything’s done — the persister-row-mode prototype is up and running.

✅ Added packages/astro/src/scripts/testRowMode.ts for reproducible row-mode tests. ✅ Updated packages/tinybase-server/src/statestore.ts with fragmented row persistence. ✅ Verified local sync between server and client via WebSocket.

Happy to open a PR or share the fork + run instructions if you’d like to review or benchmark it further. Let me know which you prefer!

@willhoney7 and @acoreyj