highland icon indicating copy to clipboard operation
highland copied to clipboard

mention merge* in parallel doc

Open dvisztempacct opened this issue 6 years ago • 2 comments

Wasted a good deal of my time trying to figure out why parallel was not performing as efficiently as I expected.

Hopefully this will save others from the same tar pit!

dvisztempacct avatar Dec 12 '19 08:12 dvisztempacct

@dvisztempacct I wonder if you could share any more insight into the difference between parallel and merge?... I have a scenario where I don't care about order, but I do want to constrain the amount of parallelism. e.g.

_(nodeReadableStream)
   .map(wrapAsync(async (obj) => {
          const transformedObj = await transformObj(obj);
          return transformedObj;
   }))
   .parallel(100) // I want to limit the concurrent reads to 100, but would rather it was pushed to the writable stream ASAP
   .pipe(nodeWritableStream); // Doesn't care what order the `transformedObj`s are written in

I was wondering if Highland.prototype.merge would:

a) honour backpressure, or is it like turning on the tap full blast? b) prevent Highland.prototype.map from being called over n times in parallel?

richardscarrott avatar Sep 17 '20 18:09 richardscarrott

I think Highland.prototype.mergeWithLimit is exactly what I'm after -- I had glossed over it for some reason 🤷‍♂️

richardscarrott avatar Sep 18 '20 09:09 richardscarrott