aaronriekenberg
Support `--pipe`
Is support for piping input to each of the job's stdin, planned? like --pipe in GNU Parallel.
Great project, thank you!
Requesting support for --pipe --blocksize
I would like to second the request for support of --pipe and would like to add a request for support of --blocksize for handling large input from STDIN.
I mainly use GNU Parallel for its great ability to take lines from STDIN, then delegate them in chunks to multiple threads on which I run python scripts.
Example:
input_script | parallel --pipe -j 16 --blocksize 10000 | python some_output_script.py > results.txt
Supporting --block-size for batch jobs is import to avoid memory overload. I am not sure if rust_parallel for each threads opens and closes a program, but in the case of Python that is not sustainable since python scripts need to compile on the fly, hence the need for --block-size for batch jobs.
I am happy to see this package come to Rust, Rust is simply best in class in parallelization, performance and memory security. Unfortunately I cannot use it for most of my needs since I need to pipe blocks of input data for efficiency and need directly outputted results. In most cases I only expect one input to lead to a result and need to directly see this output, not when all jobs are finished. Since I only expect 1 input to lead to an output, waiting for all jobs to finish will make me wait 2x as long since on average at 50% of my input lines I expect to get the result I need.
