Davide Lasagna
Davide Lasagna
Hi, nice package! Have there been any update on the parallelisation?
(Friendly bump) Is there any chance this issue can be easily addressed or worked around? I have several tens of thousand .jld files and reading data from all of those...
Thanks for chiming in. Unfortunately I am out of ideas for alternative formats... Maybe using numpy .npy via PyCall, but I have not tested it yet. It has to be...
@eschnett I could store everything in a large file, but I do not want to do that for several reasons, e.g. avoid massive data corruption, being able to delete/move/reorganise files...
Isn't 1.10 backward compatible?
Cool, Thanks!
Ok, back from vacations... Here is the script I used for timing and profiling. ``` julia using JLD # create `Nfiles` JLD files in `dir`, each containing some small amount...
I have tested the code in the original issue on a Linux machine with hdf 1.10. Same behaviour occurs.
I have tried using python h5py to read the same files and see if the issue is in HDF or HDF.jl. The julia code to write and read the files...
(Friendly bump) It seems that h5py handles thousands of files quite well. This would suggest that the problem might be in the way we use the hdf5 library. Unfortunately, (for...