Merging two FFCV .beton datasets

[manideep2510]

Hi

Let's say we have two datasets written to two FCCV dataset files say dataset1.beton and dataset2.beton. Assume both these datasets have the exact same image and label format and are written using the same method. Is it possible to take these two .beton dataset files and merge them into a single merged-dataset.beton FFCV dataset file?

This will be very useful when we want to continuously increase the size of the training datasets through continuous data collection.

Thanks!

[IlyaMescheryakov1402]

+1

It would be very helpful in production systems

[kschuerholt]

• 1 Maybe this could be done by passing more than just one fname to a Loader. Indexing for the Loader could be extended over both .beton files?

[AlexSunNik]

+1 Any update on this issue? This feature will be significantly useful for our usecase as well.