Smartify config sync

[Al2Klimov]

Is your feature request related to a problem? Please describe.

#3387

Describe the solution you'd like

On connect:

Based on the peer's version (645dcbdc9) we either behave as usual or do the following:

1. Collect each zone's FS structure which would have been send and compute a checksum and the last timestamp
2. Send that to the peer as a config::HaveZones:

{
  "zones": {
    "myZone": {
      "checksum": "da39a3ee5e6b4b0d3255bfef95601890afd80709",
      "timestamp": 1234567890
    }
  }
}

Depending on what matches, the peer answers with config::WantZones:

{
  "zones": [ "myZone" ]
}

We answer with config::HaveFiles:

{
  "zones": {
    "myZone": {
      "myFile": {
        "checksum": "da39a3ee5e6b4b0d3255bfef95601890afd80709",
        "timestamp": 1234567890
      }
    }
  }
}

The peer answers with config::WantFiles:

{
  "zones": {
    "myZone": [ "myFile" ]
  }
}

We answer with config::PartialUpdate with the same structure as config::Update, but with just the stuff from config::WantFiles.

The peer can handle it almost like config::Update.

Describe alternatives you've considered

#7994

[N-o-X]

That's basically what I was thinking when talking about this a few days ago. I like the concept 👍

[Al2Klimov]

Admittedly #8215 is complicated.

@julianbrost Any security objections regarding compressing (just) config::Update instead?

[julianbrost]

Compression as in #7994, i.e. generic compression algorithm like gzip? I mean you could probably construct an attack where someone can edit parts of the configuration but can't see other parts and wants to reveal them. But that doesn't sound very practical as they'd have to deploy many versions of configuration and there will probably be quite some noise on the connection making it hard to obtain any information.

So much for the question you asked, but how would you even partially compress a JSON-RPC stream? That sounds like it would get quite messy.

Also, what would be the problem with something like described in this issue? Sending and comparing a checksum is probably way more efficient than sending gzip(largeConfig) on every connection.

[Al2Klimov]

What I had in mind while asking you

• New, say, config::Update2 (TODO: better name) which does basically the same as config::Update
• But its payload is at best (lib)xz-ed
• As our "transport layer" is still JSON-RPC, the binary xz data is then armored with something close to base64
• TODO: larger alphabet guaranteeing not to hit JSON escape sequences (everything smaller than Unicode code point TODO, but >= " " ex. "\")

[Al2Klimov]

💡 Or even better...

• E.g. v2.15 is able to receive NetString(xz(JSON)), not just NetString(JSON)
• v2.16 makes use of this, sends config::Update xz-compressed