builder icon indicating copy to clipboard operation
builder copied to clipboard

Run make_wheel_record parallel in background

Open xwang233 opened this issue 2 years ago • 3 comments

By running make_wheel_record parallel in background, this saves ~8 minutes on my 12-core intel machine with a full cuda wheel build. It basically makes the loop "instant".

https://unix.stackexchange.com/questions/42544/does-redirecting-output-to-a-file-apply-a-lock-on-the-file/42564#42564 This answer suggested that each line will still be a line while written in parallel to the record file.

https://unix.stackexchange.com/questions/103920/parallelize-a-bash-for-loop/436713#436713 If overloading CPU is a concern, we may try this native wait -n bash trick to limit the concurrency.

xwang233 avatar Jun 29 '23 05:06 xwang233

cc @malfet @atalman @ptrblck

xwang233 avatar Jun 29 '23 05:06 xwang233

No ciflow labels are configured for this repo. For information on how to enable CIFlow bot see this wiki

pytorch-bot[bot] avatar Jun 29 '23 19:06 pytorch-bot[bot]

@xwang233 please open test PR in pytorch/pytorch for this change and attach ciflow/binaries label to it so that it will actually trigger binary builds

atalman avatar Jun 29 '23 19:06 atalman