data icon indicating copy to clipboard operation
data copied to clipboard

Enable Append Mode in SaverIterDataPipe

Open rravu3 opened this issue 1 year ago • 1 comments

🚀 The feature

Currently, Saver only allows write mode and only users to choose byte vs text mode. It might be useful to allow the flexibility to append to an existing file.

Motivation, pitch

Allowing the flexibility to append could be useful for building data pipes that can enable large scale transformations on original dataset like tokenizing, randomizing and/or splitting across training and evaluation.

Alternatives

No response

Additional context

No response

rravu3 avatar Jun 10 '24 23:06 rravu3

Thanks for the feature request @rravu3 , unfortunately we will be deprecating and then deleting DataPipes/DataLoaderV2, please see this issue: Future of torchdata and dataloading

andrewkho avatar Jun 12 '24 16:06 andrewkho