tkf
ThreadsX.jl and ThreadedIterables.jl
Hi, I'm the author of the ThreadedIterables.jl package. It seems silly to me to have 2 packages to do the same thing in the Julia ecosystem... Could we port over the functionality of one into the other and deprecate the old one?
Not proposing any "roadmap" or anything, just opening a discussion... So some questions which I think are relevant:
- Which package is more used? Is there a way to get this info? My package is quite a bit older (2017), on the other hand, it was released outside of our company only in 2019 I think, so :shrug: ....
- Which one is more general? I think ThreadsX, but would have to check.
- Which one is faster? :shrug: Would probably have to benchmark.
Overall, I think ThreadsX is much more mature package building on modern Julia features and ecosystem, whereas ThreadedIterables is rather simplistic. So, I probably see deprecating ThreadedIterables in favor of ThreadsX as the better option (that could change depending on the answers to the 3 questions above). What do you think?
Hi, thanks for the interest! Good to know that you want to improve the thread-based parallelism landscape in Julia!
- Which one is more general?
ThreadsX.jl is actually a restricted interface built on top of Transducers.jl to match with Base API and so that people don't have to learn too many new things to use threading. So, I'd definitely say that Transducers.jl is more general than ThreadsX.jl. This is probably true for any API modeled after Base.
I've been thinking about the basic interface we need to build a composable parallel ecosystem. I think the combination of Transducers.jl's interface and SplittablesBase.jl is a good starting point. For example, I just wrote FoldsCUDA.jl over the last weekend based on Transducers.jl interface that brings Transducers.jl and FLoops.jl-based for-loop onto GPU. So I'm somewhat more confident that the direction of Transducers.jl might cover a large class of computation people want to do.
I do want to improve package interop more, though. I should probably separate out FoldsBase.jl out of Transducers.jl so that people can implement things on top of FoldsBase.jl without pulling in the entire Transducers.jl.
- Which package is more used? Is there a way to get this info?
For registered packages, you can just look at "Dependents" tab. FYI ThreadsX 0.1.7 has just one dependent but ThreadedIterables 0.2.0 has two dependents. Of course, these are not enough information for things used in private or by "end users."
- Which one is faster?
I don't think it matters much. If you have a faster implementation, you can just copy it (within what the license allows). I think what matters is that if the interface allows us to optimize things on a wide range of execution paradigms (SIMD/threads/distributed/GPU).
