space icon indicating copy to clipboard operation
space copied to clipboard

Loading from TFDS/Parquet without copying files results in files at two locations

Open coufon opened this issue 2 years ago • 0 comments

The features of loading a TFDS datasets (append_array_record) and Parquet files (append_parquet) don't copy/rewrite the source files. As the consequence, a Space's dataset will be split across two locations: the original files and the new Space storage directory.

To support an option that first copies or moves the source files to the Space storage directory. Note that such copy should be still faster than writing files in the normal append methods.

coufon avatar Jan 21 '24 17:01 coufon