jitsu icon indicating copy to clipboard operation
jitsu copied to clipboard

Merge updates for sources, destinations and api_keys

Open vklimontovich opened this issue 4 years ago • 1 comments

Problem

At the moment, configurator UI updates all entities such as source destination and api_keys with a single query that overrides existing data in storage (redis). This approach is potentially vulnerable to edit conflicts - whoever send an update request the last will overwrite all previous edits.

Also, sources collection can become pretty large with recent additions from airbyte and the initial app load takes a while. Partial get can make UX nicer

Partial updates

Here's a request

POST /configurations/${collectionName}?id=${key} #example POST /configurations/destinations?id=${key}

{
  "arrayPath": "destinations",
  "object": {
    "idFieldPath": "_uid",
    "value": "vvd5ss1ue8qj26ex853gze"
  },
  "patch": {
    "_formData": {
      "newField": "test",
      "pghost": "updatedValue",
      "pgschema": null
    }
  }
}

Server should do the following:

  • Aquire lock collectionName/id
  • Find a node at arrayPath node. Assume that the path is an array (fail otherwise)
  • Find an object within array where idFieldPath is value
  • Merge that object with patch
  • Put updated object to redis
  • Release lock

Also, we need to acquire locks before pulling data from Redis (if we can implement read/write locks quickly - cool, otherwise let's stick with plain locks)

Let's also implement partial reads as an optional feature for GET configurations/{collectionName}?id={id} calls:

  • GET configurations/{collectionName}?id={id}&remove_from_array={arrayPath}&fields=catalog,config/config

  • GET configurations/{collectionName}?id={id}&filter_array={arrayPath}&idPathField=_uid&value=vvd5ss1ue8qj26ex853gze

(this task also includes making changes in HttpServerStorage implements ServerStorage on frontend

vklimontovich avatar Oct 19 '21 15:10 vklimontovich

Note on json paths: syntax should be a.b.c or /a/b/c. If the latter, a should be equivalent to /a

vklimontovich avatar Oct 19 '21 15:10 vklimontovich