electric-sql
BUG: Inconsistent Sync - UNUSABLE
The sync is inconsistent:
- applies the changes with a delay
- miss some changes
https://github.com/user-attachments/assets/b9e44a87-2342-4e03-9de7-8740e9f08379
The code is pretty simple, I've tried with multiple PGLite options, FS, worker mode. Nothing works. Also the shape sync:
await client.electric.syncShapesToTables({
shapes: {
profiles: {
shape: {
url: 'http://localhost:4000/v1/shape',
params: { table: 'profiles' },
},
table: 'profiles',
primaryKey: ['id'],
shapeKey: 'profiles',
},
inspection: {
shape: {
url: 'http://localhost:4000/v1/shape',
params: { table: 'inspection' },
},
table: 'inspection',
primaryKey: ['uuid'],
shapeKey: 'inspection',
},
},
key: 'syncBase_v1',
});
Docker
ElectricSync Docker: 1.12
Package Json
"@electric-sql/pglite": "0.3.0"
"@electric-sql/pglite-sync": "0.3.2"
Hey @AndryHTC
The video doesn't seem to be working, could you try and upload it again? I'm keen to understand what's happening here.
https://drive.google.com/file/d/1tYuj43knlWUMdaiHBBq2PvSBPNe_qoo6/view?usp=sharing
https://github.com/user-attachments/assets/dfdb3e28-679c-441a-8e91-d5e0d3b9149b
@samwillis using the client.electric.syncShapeToTable instead of the client.electric.syncShapesToTables seems to work by the way.
It gives a different error, but seems to not affect the functionality:
Uncaught (in promise) Error: duplicate key value violates unique constraint “inspection_pkey” at BroadcastChannel.a (index.js:209:25)
a
@index.js:209
There is no reason to throw this, I don't even understand what is trying to do
@samwillis using the
client.electric.syncShapeToTableinstead of theclient.electric.syncShapesToTablesseems to work by the way.It gives a different error, but seems to not affect the functionality:
Uncaught (in promise) Error: duplicate key value violates unique constraint “inspection_pkey” at BroadcastChannel.a (index.js:209:25) a @index.js:209There is no reason to throw this, I don't even understand what is trying to do
Ok this could be caused by duplicated subscriptions. I'll try to move the sync instructions to the worker
@samwillis using the
client.electric.syncShapeToTableinstead of theclient.electric.syncShapesToTablesseems to work by the way.It gives a different error, but seems to not affect the functionality:
Uncaught (in promise) Error: duplicate key value violates unique constraint “inspection_pkey” at BroadcastChannel.a (index.js:209:25) a @index.js:209There is no reason to throw this, I don't even understand what is trying to do
Ok this could be caused by duplicated subscriptions. I'll try to move the sync instructions to the worker
Moved to the worker seems to have inconsistency similar to the multi table one.
Yeah. With client.electric.syncShapeToTable it works. But.. moving that inside the pglite worker, I face the same behaviour of @AndryHTC's video.
The only difference I've spot from the video is that.. by using client.electric.syncShapeToTable inside the pglite worker when I refresh the page, I see shapes correctly synced. With client.electric.syncShapesToTables even after page's refresh I had to wait for them to sync (around 20 seconds).
Could this be the cause of the issue? https://github.com/electric-sql/pglite/issues/652
Do you guys need more info or are you investigating? We would like to pilot a GOV AI product and we need this working in a few weeks, otherwise we may create our own pull engine for supabase.
Hey @balegas, as mentioned by @AndryHTC we are going to pilot a gov AI product and right now we are relying on electric-sync. Can you please give us news about this issue?
Hei, I've been traveling and haven't looked at it. Next week we'll have that fixed
Hei, I've been traveling and haven't looked at it. Next week we'll have that fixed
Thank you for the update
Hei, I've been traveling and haven't looked at it. Next week we'll have that fixed
Just a ping to check if you have been able to replicate the issue, or if you need more info. I'll be glad to help
Hey, yes! I've isolated and reproduced the issue.
My setup is to sync a couple of shapes with syncShapesToTables and setting up a live query for both tables.
It looks like the issue happens if syncShapesToTables is called before the live query. I was able to workaround it by making sure the live queries were started before.
Do you want to check if that works for you, while we work on the issue?
@balegas I'll try ASAP at least for testing the product, until you'll be able to fix it.
I'll get back if it doesn't work ❤
Can you isolate the issue and share a repo with the code?
I've just invited to my personal repo called pglite-sync-issue. Instructions are inside readme.md.
Let me know if you need any help with it.
Can you isolate the issue and share a repo with the code?
I've just invited to my personal repo called
pglite-sync-issue. Instructions are inside readme.md. Let me know if you need any help with it.
@balegas Have you been able to replicate it? I can't believe we are the only ones using Electric SQL experiencing this
@balegas Have you been able to replicate it? I can't believe we are the only ones using Electric SQL experiencing this
you're not the only ones, I'm experiencing this as well.
I'm not entirely sure how electric works but I'll try to explain what I saw from poking around.
electric makes a request that takes 20 seconds and keeps repeating indefintely, my guess is the server holds on to the request listening for changes, and refreshes every 20 seconds in order not to lose connection.
when data gets updated in the backend db, a new request gets made that updates pglite, and a new 20 second request gets made.
from what I saw, the first and second 20 second requests made after the data has been updated do nothing, and only after the second update (~40 seconds) do updates get synced to pglite, which seems to line up (ish?) with your video.
I've opened a PR that shall address the issue -- at least it's solve for my reproduction of the issue. I don't think everyone is experiencing this, since it depends on how you set your shapes and live queries.
I've opened a PR that shall address the issue -- at least it's solve for my reproduction of the issue. I don't think everyone is experiencing this, since it depends on how you set your shapes and live queries.
What's so specific about our shapes that caused the problem?
It's not the shape definitions specifically. It's a race condition between setting up live queries and when the syncShapesToTable is called. If the two calls are mutually exclusive, you'd never miss any changes. I guess a lot of people will setup their applications differently. The patch should address issues with races.
We've encountered some more issues, but we really didn't have the time to work more on it yet.