ray icon indicating copy to clipboard operation
ray copied to clipboard

[Datasets] Use generators for block splitting

Open stephanie-wang opened this issue 3 years ago • 0 comments

Why are these changes needed?

Update Datasets block splitting to use dynamic generators instead of ray.put and owner reassignment. This should be more stable long-term and reconstruction works out of the box.

Unfortunately dynamic generators aren't yet supported for actor tasks (#28681). To unblock other testing work, I've temporarily changed it to raise a NotImplementedError if block splitting is enabled with compute="actors".

Checks

  • [ ] I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
  • [ ] I've run scripts/format.sh to lint the changes in this PR.
  • [ ] I've included any doc changes needed for https://docs.ray.io/en/master/.
  • [ ] I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
  • Testing Strategy
    • [ ] Unit tests
    • [ ] Release tests
    • [ ] This PR is not tested :(

stephanie-wang avatar Sep 21 '22 22:09 stephanie-wang