[RFC] Live diff

[chrisdmacrae]

Summary

Add the liveDiff functionality to the package. The liveDiff functionality will calculate all the operations needed to be done on the index, and then perform a live batch on the index to update it.

Background

LiveDiff should require no new methods outside of the actual liveDiff functionality.

It should implement the client to:

• Initialize the index
  • Allow configuring the index's settings
• Make a copy of the remote index
• Make a copy of the manifest
• Calculate the following:
  • Records not in the remote [add]
  • Records in the remote that have changed (do so by comparing the remote uniqueID to local uniqueID if set) [update]
    • If not set in local index, generate uniqueID from the record content.
  • Records that are not in the local but in the remote [delete]
  • Records that are in both the remote and local
• Update the existing index
• Generate a new manifest
  • ObjectID lookup, 100 objectIDs per record, configurable