data
data copied to clipboard
Add `SingleProcessReadingService` for `DataLoader2` by default
🚀 The feature
Add SingleProcessReadingService to adapt the graph for the sake of:
- Shuffle seed setting per epoch
- Set shuffle/sharding
- etc.
Pros:
This would prevent making DataLoader2 more complicated. The reading service should take charge of all adaptation of the graph of DataPipes.
Cons:
By adding SingleProcessReadingService, we need mixed ReadingServices for the most of cases.
Motivation, pitch
To align the behavior of DataLoader2 with or without MPRS.
Alternatives
No response
Additional context
No response