chaining commands that generate large output is impractical

[abg]

subprocess.run uses Popen.communicate() which buffers data in memory. So some tasks that might be desirable to run (e.g. mysqldump | gzip ) will use too much memory and fail for many cases. Ideally this would instead just link up pipes across processes and handle the file descriptor management behind the scenes, so it could handle arbitrary command chains.

[xando]

Thanks for the comment. You are totally right, it looks like I was super optimistic about https://github.com/xando/subprocess.run/blob/master/run/init.py#L146

Will try to improve this process.