featurebase icon indicating copy to clipboard operation
featurebase copied to clipboard

Refactor consistent hashing implementation

Open shaqque opened this issue 6 years ago • 1 comments

Overview

This PR refactors the Jump consistent hashing implementation to allow using other consistent hashing algorithms. Shard distribution to nodes now happen on a per-index basis, which means different indexes can be distributed using different algorithms.

All pre-existing indexes will default to the current Jump algorithm, while new indexes will be allocated using the default shard distribution algorithm set when the server starts/restarts.

Fixes #2035

Pull request checklist

  • [x] I have read the contributing guide.
  • [x] I have agreed to the Contributor License Agreement.
  • [x] I have updated the documentation.
  • [x] I have resolved any merge conflicts.
  • [x] I have included tests that cover my changes.
  • [x] All new and existing tests pass.
  • [x] Make sure PR title conforms to convention in CHANGELOG.md.
  • [ ] Add appropriate changelog label to PR (if applicable).

Code review checklist

This is the checklist that the reviewer will follow while reviewing your pull request. You do not need to do anything with this checklist, but be aware of what the reviewer will be looking for.

  • [ ] Ensure that any changes to external docs have been included in this pull request.
  • [ ] If the changes require that minor/major versions need to be updated, tag the PR appropriately.
  • [ ] Ensure the new code is properly commented and follows Idiomatic Go.
  • [ ] Check that tests have been written and that they cover the new functionality.
  • [ ] Run tests and ensure they pass.
  • [ ] Build and run the code, performing any applicable integration testing.
  • [ ] Make sure PR title conforms to convention in CHANGELOG.md.
  • [ ] Make sure PR is tagged with appropriate changelog label.

shaqque avatar Jul 31 '19 20:07 shaqque

@jaffee the tests are failing because the protobuf files are not yet generated. Unfortunately, I have the latest version of protoc/protobuf on my computer which adds trash fields like XXX_blahblah. Otherwise, all tests pass locally on my computer with the generated protobuf files and this PR should be ready.

An older version of protoc/protobuf should generate the protobuf files without the trash fields.

shaqque avatar Aug 09 '19 02:08 shaqque