conc icon indicating copy to clipboard operation
conc copied to clipboard

`iter` would benefit from configurable concurrency.

Open sudhirj opened this issue 3 years ago • 3 comments

Map and ForEach currently use the GOMAXPROCS value to determine how many go routines run in parallel, but there's lots of situations (like making one network request per item in the slice) that would benefit from making that concurrency configurable. Yes, Stream is available, but iter has more fitting constructs. Would suggest an additional set of methods to override.

I can work on this the we're ok with the idea.

sudhirj avatar Jan 12 '23 05:01 sudhirj

I agree. For reference, there was a bit more discussion about this in https://github.com/sourcegraph/conc/issues/36. I think there are some open questions about the design of such a feature. I'd be happy to accept a contribution, but I think it might be worth thinking through some designs first. What do you have in mind?

camdencheek avatar Jan 12 '23 05:01 camdencheek

I was thinking MapN and ForEachN, of the form

func MapN[T, R any](input []T, f func(*T) R, parallelism int) []R {}

sudhirj avatar Jan 12 '23 05:01 sudhirj

And Map and ForEach would delegate to their N counterparts passing in runtime.GOMAXPROCS(0).

sudhirj avatar Jan 12 '23 05:01 sudhirj

I've opened a PR with another take on configurable concurrency for iter: https://github.com/sourcegraph/conc/pull/63

bobheadxi avatar Jan 14 '23 00:01 bobheadxi

Oh nice, I like the #63 design much better. Closing this issue.

sudhirj avatar Jan 14 '23 05:01 sudhirj