Merge updates for sources, destinations and api_keys
Problem
At the moment, configurator UI updates all entities such as source destination and api_keys with a single query that overrides existing data in storage (redis). This approach is potentially vulnerable to edit conflicts - whoever send an update request the last will overwrite all previous edits.
Also, sources collection can become pretty large with recent additions from airbyte and the initial app load takes a while. Partial get can make UX nicer
Partial updates
Here's a request
POST /configurations/${collectionName}?id=${key} #example POST /configurations/destinations?id=${key}
{
"arrayPath": "destinations",
"object": {
"idFieldPath": "_uid",
"value": "vvd5ss1ue8qj26ex853gze"
},
"patch": {
"_formData": {
"newField": "test",
"pghost": "updatedValue",
"pgschema": null
}
}
}
Server should do the following:
- Aquire lock
collectionName/id - Find a node at
arrayPathnode. Assume that the path is an array (fail otherwise) - Find an object within array where
idFieldPathisvalue - Merge that object with
patch - Put updated object to redis
- Release lock
Also, we need to acquire locks before pulling data from Redis (if we can implement read/write locks quickly - cool, otherwise let's stick with plain locks)
Let's also implement partial reads as an optional feature for GET configurations/{collectionName}?id={id} calls:
-
GET configurations/{collectionName}?id={id}&remove_from_array={arrayPath}&fields=catalog,config/config -
GET configurations/{collectionName}?id={id}&filter_array={arrayPath}&idPathField=_uid&value=vvd5ss1ue8qj26ex853gze
(this task also includes making changes in HttpServerStorage implements ServerStorage on frontend
Note on json paths: syntax should be a.b.c or /a/b/c. If the latter, a should be equivalent to /a