audius-protocol
audius-protocol copied to clipboard
Reduce export batch size from 10,000 to 1000
Description
We see a few issues as a result of exports being 10k records including:
- statement connection timeout
- sql tx blocks and connections open for a while because CNodeUser is updated and the connection is held open while data is saved to disk
- sync wallet locks expiring and multiple sync operations running concurrently
The simplest way to alleviate this pressure is to lower the export batch size and do more requests with smaller amounts of data which means
- transactions should commit quicker because it's less db operations at once
- tx's aren't held open as long because it's less data to fetch
- sync wallet locks should not expire (no way to guarantee this in either scenario though)
Tests
Mad dog and integration tests pass. Will also test on staging by updating a user replica set for a user with tens of thousands of records
Monitoring - How will this change be monitored? Are there sufficient logs / alerts?
Fully synced % should go up, particularly those with clock values in 10s to 100s of thousands