haskell-distributed
Is there any way that Closure/Static could be factored into a separate package?
Perhaps this is too much to hope for, but it would be ideal for packages such as meta-par, HDpH and distributed-process to be able to share basic RPC functionality (Closure/Static).
With meta-par we copied and hacked a version of the original Cloud Haskell ("Remote") Closure. The ugly bit was that it had hard-coded recognition of the IO and ProcessM monads, and we had to tweak that to include monad-par's "Par" monad.
I don't understand the current Closure implementation, but it looks like it may be the case that CP.hs (the part that deals with Process values) is pretty well isolated from the rest. Does that mean that everything but CP.hs could become its own package?
It looks like TH.hs also deals with Process presently. TH.hs would either need to be replicated in all consumers of the hypothetical factored library, or it would need to become more extensible. When we were using the Closure mechanism in monad-par/meta-par this was the sticking point -- we didn't see a way to do it without adding extra arguments to the compile-time TH functions (e.g. remotable), which would be very ugly.
Any good ideas here?
Yes, the old implementation support for Static/Closure was very tightly interwoven with the rest. This is no longer the case, although you are right that the Template Haskell stuff does recognize Process for convenience. It should be possible to factor most of this out. I think that's a good idea, even just for clarifying the current implementation. I will have a look at this (right after I give you your performance benchmarks :).
I have made a lot of progress on this today but haven't quite finished yet. Hopefully I'll be able to push something tomorrow.
I have pushed (and released to Hackage) a new version of CH where static is a separate package (Control.Distributed.Static -- http://hackage.haskell.org/package/distributed-static-0.1.0.0). This package does not provide any Template Haskell support but the TH layer in Cloud Haskell is now so thin, and so little of it is not Cloud Haskell specific, that I'm not sure it's worth abstracting it further (in fact, C.D.Static is quite usable without TH at all). Please have a look at let me know what part of the TH would be useful in metapar (or elsewhere) and would be worthwhile moving from CH proper to C.D.Static.
Awesome, thanks. We will switch over to this as soon as we get a chance and let you know how it goes.
On Fri, Aug 10, 2012 at 12:12 PM, Edsko de Vries [email protected]:
I have pushed (and released to Hackage) a new version of CH where static is a separate package (Control.Distributed.Static -- http://hackage.haskell.org/package/distributed-static-0.1.0.0). This package does not provide any Template Haskell support but the TH layer in Cloud Haskell is now so thin, and so little of it is not Cloud Haskell specific, that I'm not sure it's worth abstracting it further (in fact, C.D.Static is quite usable without TH at all). Please have a look at let me know what part of the TH would be useful in metapar (or elsewhere) and would be worthwhile moving from CH proper to C.D.Static.
— Reply to this email directly or view it on GitHubhttps://github.com/haskell-distributed/distributed-process/issues/21#issuecomment-7649171.
Moving to Misc/Admin as
(a) the move has already taken place and (b) we've not heard anything back for >= 3months
Ryan, are you happy for me to close or is there more to us here?
To address an earlier question raised by @rrnewton on this issue. The closure/static representation in HdpH was also cut away from the rest of HdpH back in July 2012. At the time this was done to improve the chances of a unified representation for meta-par, hdph and cloud-haskell. @edsko has made similar de-coupling efforts from the CH side. The motivated design of the closure representation is described in Section 3.2 of the first hdph paper, in particular pay attention to the dual closure representation paragraph. The key implementation facts about static in HdpH is in the documentation throughout HdpH.Closure.Static.hs , and the template haskell code for mkClosure and so on is in HdpH.Closure.Internal.hs . I realise that this comes months after the CloudHaskell 2.0 closure implementation. I also, however, see many benefits to having one one representation across all distributed haskells. This would for example enable CloudHaskell's task layer to benefit from load balancing in HdpH's network-based work-stealing scheduler.
It is worth adding that a considerable HdpH refactor will soon be pushed onto github, which will see adoption of the network-transport-tcp backend, and an upload of the library to hackage.
Hi @robstewart57 - thanks for explaining a bit of the history behind this.
a unified representation for meta-par, hdph and cloud-haskell
We should definitely aim for that if possible.
I also, however, see many benefits to having one one representation across all distributed haskells.
Me too - lots and lots of them. I'd really like to be able to mix and match.
It is worth adding that a considerable HdpH refactor will soon be pushed onto github, which will see adoption of the network-transport-tcp backend, and an upload of the library to hackage.
Great. After that happens, we should look at properly unifying these layers then. Ryan (or someone on his behalf) will need to let one of us know what's left to be done before all three libraries can share the same implementation. From @edsko's last comment, it sounds like the main thing that's left to be done is to possibly share some of the TH code, though I'm not clear on which bits would be useful - are we talking about all of mkClosure available across all three libraries?
I'm happy to look into this, but it's a bit further down the priority stack than some of our pressing bug fixes right now and I know little about TH so it might not get done until I've had time to chat with @edsko about it. Unless someone wants to either contribute a pull request or give me some pointed ideas about how to achieve it, I'd estimate we're looking at post 0.2.0. Sorry I realise that's not the answer everyone was hoping for, but I want to give a realistic picture of what I'm likely to get done in my spare time over the next couple of months. I will probably drop a note out onto the parallel-haskell mailing list asking if anyone wants to step into the gap and assist with this specific task.
Note that the dual closure representation cannot be implemented without using unsafePerformIO (or proper implementation of 'static'). That's the main reason I didn't adopt it yet.
Note that the dual closure representation cannot be implemented without using unsafePerformIO (or proper implementation of 'static'). That's the main reason I didn't adopt it yet.
Hmn, I'm inclined to hold off on adopting it for the same reason.