mention merge* in parallel doc
Wasted a good deal of my time trying to figure out why parallel was not performing as efficiently as I expected.
Hopefully this will save others from the same tar pit!
@dvisztempacct I wonder if you could share any more insight into the difference between parallel and merge?... I have a scenario where I don't care about order, but I do want to constrain the amount of parallelism. e.g.
_(nodeReadableStream)
.map(wrapAsync(async (obj) => {
const transformedObj = await transformObj(obj);
return transformedObj;
}))
.parallel(100) // I want to limit the concurrent reads to 100, but would rather it was pushed to the writable stream ASAP
.pipe(nodeWritableStream); // Doesn't care what order the `transformedObj`s are written in
I was wondering if Highland.prototype.merge would:
a) honour backpressure, or is it like turning on the tap full blast?
b) prevent Highland.prototype.map from being called over n times in parallel?
I think Highland.prototype.mergeWithLimit is exactly what I'm after -- I had glossed over it for some reason 🤷♂️