andryou
Sync Enhancements
For the past week I've put hours into enhancing ScriptSafe's syncing capabilities (https://github.com/andryou/scriptsafe/commits/master), so I decided to create this to document what I've been working on:
- When syncing settings to Google, any unused sync-specific data is cleaned up from the local device
- When an update is synced from the current device, the Options page - if open on one or more other devices - will auto-refresh to reflect the latest settings
- Checks have been added to ensure that sync data is newer than the current local data before asking to import
- Checks have been added to ensure that synced list data is verified to be complete before importing them
- Support for handling a new and improved syncing method will be included in v1.0.9.2, which:
- Ensures all user lists will be synced across all devices reliably
- Uses the lz-string library to better support extremely large user lists (Google sync data has a limit of 102,400 bytes (https://developer.chrome.com/extensions/storage#property-sync-QUOTA_BYTES), so this will provide more efficient space usage)
- Allows for additional timestamped verification
- All lists (general whitelist, fingerprint whitelists, and general blacklist) will continue to use the legacy method
My plan is to gradually phase-in the new syncing method for the lists several releases down in the future (at least 6 months from the date v1.0.9.2 is publicly released); it cannot be immediate as pre-v1.0.9.2 instances wouldn't know how to handle the compressed synced data. v1.0.9.2+ will be able to handle the compressed sync data.
As syncing involves user settings (more specifically, the potential of overwriting settings), I am putting in more time doing personal testing before any sort of beta is released to the public.
My testing set-up includes 3 Chrome instances running:
- v1.0.9.1 (for backwards compatibility testing)
- v1.0.9.2 (proposed next release, with only handling of new syncing method implemented added, no actual data is synced using it for the sake of backwards compatibility with <= v1.0.9.1 instances)
- v1.0.9.3 (theoretical release, which syncs using the new method to ensure that the new syncing handling support in v1.0.9.2 will work as intended when the sync method is switched)
In terms of testing data, I am using a whitelist and blacklist containing 6,000+ entries each to ensure that very large datasets are synced properly.
Unfortunately I very quickly discovered that Chrome struggles to handle large sync.set requests, which led me to code in checks to verify imported data is complete before committing them to localStorage.
Haven't heard anything from you for quite some time... just checking in to see if development is still ongoing?
@andryou Just happened across this.. I've got a whopping 734 items in my whitelist and I'm getting the dreaded QUOTA_BYTES &c. I notice this is suggesting it's a single item that's bumping up against this limit; is there any possibility of chunking parts of this config, like the whitelist, &c out to multiple keys?
Just got it again: QUOTA_BYTES_PER_ITEM
Just taking wc on a stroll through the file.. IDK if it matters but line 12 in the config is over 16kB
