(Draft) Multi-View Support
This is a big one, requiring a complete rewrite of most of the Autobase internals and several breaking API changes, but it's worth it for the multi-views:
This PR adds support for processing an Autobase into multiple views (each in a separate output Hypercore), such that those views are always updated atomically. This is particularly useful for scenarios when you want to generate many different indexes over a common dataset (e.g. one view could be a "raw" list of causally-ordered chat messages, and another could store counts of unique words in those messages):
// Assuming `base` is a previously-constructed Autobase
base.start({
version: 1,
views: 2,
open: (core1, core2) => [core1, new Hyperbee(core2)],
apply: async (core, bee, batch) => {
await core.append(batch)
const b = bee.batch()
for (const node of batch) {
// Index the node inside the Hyperbee
}
await b.flush()
}
})
Importantly, you can now also include a version option when creating views, which describes the version of the apply/open functions. When you make a change to these functions, you're expected to increment version such that previously-computed views can be truncated to 0 (and regenerated using the new code).
This is a hefty, API-breaking change, but it is storage-compatible with existing Autobase. Perhaps the most breaking change is that Autobase now takes a Corestore as a constructor arg, and the other constructor opts (inputs and outputs) are expected to be Hypercore keys, not the cores themselves. This makes sense because every Autobase now has to work with lots of Hypercores (N output cores per output, depending on the view count). It also simplifies resource-management, since the Autobase has ownership over all its cores.
Some additional changes:
- All tests moved to Brittle
- Added
lib/bisect.jsfor doing a galloping search over Hypercores, which should be extracted into a separate modules. - Output core keys are derived using
hypercore-crypto-tweak, based on the key of the first output. - The
Nodeschema contains a new optionalclocksfield (one clock for each output) - The
Headerschema contains a newversionuint field (for busting outdated views)