Lars Preliminary Backlog

[larskuhtz]

Database backend

• [ ] Detect consensus db schema changes
• [x] Command line flag to reset consensus db
• [x] Documentation for rocks db layer
• [ ] More efficient TreeDB branches implementation (via reverse iterator) Note that reverse that rocksdb reverse iteration isn't as performant as forward iteration. So, we should also consider chaining the order, assuming that we iterate backward most of the time.
• [ ] RocksDb API and in-memory Rocksdb implementation (either existing or based on hash map)
• [ ] Use RocksDb column families to implement tables? However, check rocks db documentation about use cases for column families and different dbs. Column families may not be the best fit for us, or at least provide little benefits.
• [x] Atomic writes across Rocksdb tables
• [ ] fork RocksDB package and add support for prefix seek.
• [ ] database repair flag
• [ ] Add Table API
• [ ] use LZ4 compression for rocks db
• [ ] efficient membership queries

ChainDbs

• [x] Atomic writes for payload data (that lives in different cas). We may have to change the type of Payload db, so that the rocks db table is exposed from the type parameter.
• [x] Cut persistence
• [ ] flag to re-evaluate chain db by (initializing cutdb with genesis cut and possibly injecting adding local cuts before accepting new cuts)
• [ ] wrapper for payload db that adds tx-hash index (not a CAS, needs updates when tx are reintroduced)
• [x] Move optimizations in limitCut into TreeDB.
• [ ] Add NFData constraint to all TreeDB functions that return a stream (to prevent leaking access to the backend from the bracket).
• [ ] consider if payload db should be moved out of consensus, so that it can be sharded (Probably not a good idea)
• [ ] option to check / rebuild secondary payload index (payloadHash -> blockHeight) for recent blocks. (e.g. in case it gets corrupted during a crash).
• [ ] low-priority background check for secondary payload index

Validation

• [ ] Forward/Iterative block-chain validation for existing db (cf. cut persistence)
• [ ] Include epoch start into block header and validate target value
• [ ] Comprehensively document validation

Cut Processing

• [ ] Sample network consensus for estimating if and how much the local node is out of sync
  • [ ] [optional/optimization] Short POW certificates to allow a node to prove that a fork received the claimed amount of work without further validation

REST API

• [ ] Suppress origin and suppress advertising peer (--invisible flag)

• [ ] Sync payloads for “invisible nodes” (attach new block header and payload to cuthashes)

• [ ] Binary encoding for cut hashes on the wire

• [ ] Support content-type header (json and octet) for block-header and cut hashes

• [ ] connection manager (withConnectionManasger) that uses openssl as backend for https connections.

• [ ] client authentication

Logging / Telemetry

• [ ] Better mempool telemetry
• [ ] Fix file handle logging
• [x] suppress pact info logs during replay
• [x] Roll Elasticsearch indices daily
• [ ] Per component log configuration (log-level and possibly backends)
• [ ] Add a logger field to all handles, instead of passing the logger separately.
• [ ] move cut logs from base-backend to telemetry and make them more compact
• [ ] support random sampling of logs (on a per log level base + possibly per log message level)

Code cleanup

• [ ] break cycles in rejoin spite *.Types modules
• [ ] Read only versions of chain db handles
• [ ] Hide constructor fields for BlockHeader
• [ ] Hide constructor fields for payload data

Misc

• [ ] Option to run node (bootstrap node) without Mempool
• [ ] function to sample network consensus (estimate consensus state without validation)
• [ ] fix server startup in Chainweb.Chainweb (make sure exception in server are propagated).

P2P

• [ ] P2P db api for reputation management
• [ ] Increase default size of P2P db and use random sampling for P2P db sync (sync only subset)
• [ ] add more jitter to peer selection
• [ ] document relation between peer propagation and peer authentication
• [ ] increase cost of peer creation (use pow?)

[enobayram]

This is a great list, we should probably go over these, spot the ones that are still relevant and turn them into tickets so that they can be tracked independently.

[larskuhtz]

Phew, good to see that quite a few of these go actually resolved over the past years :-)

[edmundnoble]

Checking things off as other issues are created for them.