ray
ray copied to clipboard
[core] Object returned by a generator with num_returns="dynamic" should throw an error if reconstruction fails
What happened + What you expected to happen
If a dynamic generator stores some objects in shared memory and these are later lost, then Ray will try to reconstruct them by re-executing the generator task. Currently, if the re-execution stores some of these objects but then errors (e.g., because the generator returned more values than expected), the already-stored objects will be accessible to other tasks through the object store.
Instead, we should store an error for all of the task's return values.
Versions / Dependencies
3.0dev
Reproduction script
See TODO in this unit test that triggers recovery for a dynamically created object, then forces the re-executed task to generate more objects than before.
Issue Severity
No response