O_DIRECT on lscratchrza (LLNL)
dcp --synchronous option is not working on lscratchrza (that we know of so far). When this option is turned on lscratchrza we need to fix the alignment and sizes.
For now the options are: run dcp then dcmp to check src matches dest directory run an srun --drop-caches after running dcp (LLNL)
You should be able to get the block size of a file by performing a stat on it. Once you do that, you should be able to allocate a buffer of that size. This will likely complicate how files are split up in dcp I imagine.
Isn't the root of the problem here that ZFS does not support O_DIRECT in 0.7.x (though fixed in 0.8.x)?