array_storage_benchmark icon indicating copy to clipboard operation
array_storage_benchmark copied to clipboard
verified publisher icon mverleg

Add hdf5

Open mverleg opened this issue 8 years ago • 4 comments

mverleg avatar Jun 05 '17 10:06 mverleg

bm_random

I'm skeptical that this measures real write speed, because it's faster than binary.

I have a flush in there, I don't know what else could be done.

mverleg avatar Apr 13 '20 09:04 mverleg

will h5 file have problem with CPU memory？I encounter some problem which I think is caused by the memory consumption of loading h5file，especially when using multi-workers

gaopinghai avatar Mar 29 '22 07:03 gaopinghai

@PingHGao I think this might not be the best place to get help with your problem sorry

mverleg avatar Apr 02 '22 17:04 mverleg

bm_random

I'm skeptical that this measures real write speed, because it's faster than binary.

I have a flush in there, I don't know what else could be done.

Hi, is this related to chunk caching of HDF5? If you're writing or reading the same content(chunk) multiple times, real I/O will be executed once.

If you're using h5py, you can set the rdcc_* arguments when opening a file to disable the cache effect. https://docs.h5py.org/en/stable/high/file.html#chunk-cache

Nimrod0901 avatar Oct 31 '22 02:10 Nimrod0901