ray
ray copied to clipboard
[Datasets] Use generators for block splitting
Why are these changes needed?
Update Datasets block splitting to use dynamic generators instead of ray.put and owner reassignment. This should be more stable long-term and reconstruction works out of the box.
Unfortunately dynamic generators aren't yet supported for actor tasks (#28681). To unblock other testing work, I've temporarily changed it to raise a NotImplementedError if block splitting is enabled with compute="actors".
Checks
- [ ] I've signed off every commit(by using the -s flag, i.e.,
git commit -s) in this PR. - [ ] I've run
scripts/format.shto lint the changes in this PR. - [ ] I've included any doc changes needed for https://docs.ray.io/en/master/.
- [ ] I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
- Testing Strategy
- [ ] Unit tests
- [ ] Release tests
- [ ] This PR is not tested :(