data icon indicating copy to clipboard operation
data copied to clipboard

Add `SingleProcessReadingService` for `DataLoader2` by default

Open ejguan opened this issue 3 years ago • 0 comments

🚀 The feature

Add SingleProcessReadingService to adapt the graph for the sake of:

  • Shuffle seed setting per epoch
  • Set shuffle/sharding
  • etc.

Pros: This would prevent making DataLoader2 more complicated. The reading service should take charge of all adaptation of the graph of DataPipes.

Cons: By adding SingleProcessReadingService, we need mixed ReadingServices for the most of cases.

Motivation, pitch

To align the behavior of DataLoader2 with or without MPRS.

Alternatives

No response

Additional context

No response

ejguan avatar Aug 05 '22 18:08 ejguan