Record Contents: Single Source of Truth

[nils-wisiol]

This lets nslord and api run off the same database to avoid duplication in data storage and to reduce pain from concurrent API requests.

nslord accesses data stored in the api database tables through views (some of them are updateable). Hence, nslord on our data is always up-to-date. This also saves a lot of change tracking and reduces the number of pdns API requests necessary.

Open Issues

• [x] ~~Check if record content format of the API is fully compatible with pdns~~ (covered by e2e2 tests)
• [x] Make sure nslord answers DNS requests for zones it knows
• [x] Simplify pdns change tracker code
• [x] Figure out upgrade path
• [ ] Remove access to private keys for the API container
• [ ] docker networking: currently, nslord, dbapi, api are all in the same network
• [ ] do not delete cryptokeys when domains are deleted?
• [x] SOA-EDIT-API meta data setting required?
• [x] make sure the record's ordername isn't needed
• [x] SOA MNAMES must not have trailing dot in pdns storage?
• [x] set auth field in records table correctly
• [x] TODOs mentioned in the code
• [x] are serial updates still happening? (I believe not.)
• [ ] make sure the domains notified_serial isn't needed
• [ ] make sure pdnsutil commands are working
• [ ] check pdns AXFRs for SERVFAIL (in case of invalid record content)

Improvements by this PR

1. Can perform atomic updates to multiple domains with a single transaction.
2. One fewer mariadb
3. fewer pdns API calls for RR updates (now O(1))
4. atomic transactions for individual RRs
5. allows hosting of unsigned/presigned zones by adjusting the domains table
6. allows customization of serial updates (beyond capabilities of pdns)
7. Tests run with NS record set in place (that's more of a bugfix for the tests)

Problems introduced by this PR

1. Weakens access control to private keys.

Deployment of this PR

Data from 7 tables need to be migrated. 2 tables need to be copied to dbapi, 5 tables are redundant and need checking if data is as expected and can be deleted.

1. shutdown api access, shutdown nslord
2. backup dblord, dbapi database
3. Verify that the following tables at dblord are empty:
   • supermasters
   • comments
4. progress: 2/7 tables done
5. Verify that the domainmetadata table contains two records for each domain that are precisely a 1 0 0 - NSEC3OARAM and DEFAULT SOA-EDIT-API.
6. progress: 3/7 tables done
7. Verify that all domains listed in dbapi also appear in dblord
8. Verify that all domains listed in dblord also appear in dbapi
9. Verify that all domains listed in dblord have
   • type = MASTER
   • master = NULL
   • account = NULL
10. progress: 4/7 tables done
11. add SOA records for all domains to dbapi (without change tracker)
12. progress: 5/7 tables done and can be discarded
13. opportunity to bail out and just restart nslord and open api access
14. pull new images
15. run api migrations
16. copy cryptokeys and tsigkeys from dblord to dbapi
17. progress: 7/7 tables done
18. shutdown nsmaster
19. restart api, nslord
20. query nslord for desec.io, desec.org AXFR
21. start nsmaster
22. enable api access
23. adjust backup (dblord volume is stale now)