replicate
WIP: VMSplice/STDOUT consumer
@technillogue see if this works for you, we should generally order the requests for slices.
io.copy also handles backpressure from full pipes elegantly (AFAICT). We also do have a greedy reader to capture wire data even if the consumer is blocking.
this could definitely work! would it work with multifile and writing multiple files to stdout in order?
We can make that happen! Wouldn't be too hard. This was a first pass to see if it looked good.
It turns out the only reason we don't order downloads which causes you the problem is a historical reason for using a map instead of a list.
Additionally there is the maximum concurrent files option already that could limit concurrency -- additionally we can set it to automatically use a max file concurrency of one for stdout/vmsplice modes.
@technillogue Rebased. Download order is now guaranteed. You will want to make sure to use --max-concurrent-files=1 and -o stdout or -o vmsplice
Please let me know if this works. if so we can easily spin up this output consumer.
this doesn't build
Error: pkg/consumer/linux_vmsplice.go:15:18: cannot use &VMSpliceConsumer{} (value of type *VMSpliceConsumer) as Consumer value in variable declaration: *VMSpliceConsumer does not implement Consumer (wrong type for method Consume)
have Consume(io.Reader, string, int64) error
want Consume(io.Reader, string) error
Error: pkg/consumer/stdout.go:9:18: cannot use &StdoutConsumer{} (value of type *StdoutConsumer) as Consumer value in variable declaration: *StdoutConsumer does not implement Consumer (wrong type for method Consume)
have Consume(io.Reader, string, int64) error
want Consume(io.Reader, string) error