[DPL] (probably a feature request) ExternalFairMQDeviceProxy with inputs as well

[aphecetche]

Question summary

If I understand correctly (and that's not granted ;-) ) how ExternalFairMQDeviceProxy works , its main usage is to inject data into a DPL workflow, i.e. a DPL workflow using such a proxy knows only about what the proxied device is supposed to create, not what it requires as inputs.

So the question is whether there is a way to express the input requirement as well, and thus freely mix and match DPL and non-DPL devices within a DPL-controlled workflow ?

Use case

For instance, I was trying to make a simple workflow :

hit-generator -> digitizer -> pre-clusterizer

where hit-generator is a "sampler" (only output), digitizer a "processor" (input and output) and pre-clusterizer a "sink" (no output).

schematically :

{
      "mch-hit-generator",
      noInput,
      of::Outputs{{"MCH", "HITS", detElemId, of::OutputSpec::Lifetime::Timeframe}},
      of::AlgorithmSpec{initHitGenerator},
},
{
     "mch-digitizer",
      of::Inputs{{"mch-hits", "MCH", "HITS", detElemId, of::InputSpec::Lifetime::Timeframe}},
      of::Outputs{{"MCH","DIGITS",detElemId, of::OutputSpec::Lifetime::Timeframe}},
      of::AlgorithmSpec{initDigitizer},
},
{
 "mch-preclusterizer",
      of::Inputs{{"mch-digits", "MCH", "DIGITS", detElemId, of::InputSpec::Lifetime::Timeframe}},
      noOutput,
      of::AlgorithmSpec{[](of::ProcessingContext& ctx) { // some code here ... }
 }

This works fine.

❯ mch-hit-generator -b -g
digraph structs {
  node[shape=record]
  "mch-hit-generator" [label="{{}|mch-hit-generator(1)|{<from_mch-hit-generator_to_mch-digitizer>from_mch-hit-generator_to_mch-digitizer}}"];
  "mch-digitizer" [label="{{<from_mch-hit-generator_to_mch-digitizer>from_mch-hit-generator_to_mch-digitizer}|mch-digitizer(2)|{<from_mch-digitizer_to_mch-preclusterizer>from_mch-digitizer_to_mch-preclusterizer}}"];
  "mch-preclusterizer" [label="{{<from_mch-digitizer_to_mch-preclusterizer>from_mch-digitizer_to_mch-preclusterizer}|mch-preclusterizer(1)|{}}"];
  "mch-hit-generator":"from_mch-hit-generator_to_mch-digitizer"-> "mch-digitizer":"from_mch-hit-generator_to_mch-digitizer" [label="22000"]
  "mch-digitizer":"from_mch-digitizer_to_mch-preclusterizer"-> "mch-preclusterizer":"from_mch-digitizer_to_mch-preclusterizer" [label="22001"]
}

Now what if I want to replace the middle device (digitizer) by an external one ?

I tried to replace the mch-digitizer spec by a spec using a proxy :

of::DataProcessorSpec digotizerSpec(uint detElemId)
{
  of::OutputSpec outspec{ "MCH", "DIGITS", detElemId, of::OutputSpec::Lifetime::Timeframe };

  return of::specifyExternalFairMQDeviceProxy(
    "mch-digotizer",
    of::Outputs{outspec},
    "type=sub,method=connect,address=tcp://localhost:6060,rateLogging=1",
    digo2digi(outspec));
 }

but then it seems my workflow no longer knows that the first step should be the hit-generator :

❯ mch-hit-generator -b -g
digraph structs {
  node[shape=record]
  "mch-digotizer" [label="{{}|mch-digotizer(1)|{<from_mch-digotizer_to_mch-preclusterizer>from_mch-digotizer_to_mch-preclusterizer}}"];
  "mch-preclusterizer" [label="{{<from_mch-digotizer_to_mch-preclusterizer>from_mch-digotizer_to_mch-preclusterizer}|mch-preclusterizer(1)|{}}"];
  "mch-digotizer":"from_mch-digotizer_to_mch-preclusterizer"-> "mch-preclusterizer":"from_mch-digotizer_to_mch-preclusterizer" [label="22000"]
}

which I assume is to be expected as the fact that the digotizer consumes the output from the hit-generator is not explicitly expressed in the workflow, am I right ? Or do I just misunderstand something basic that would allow to perform what I want ?

[ktf]

Yes, correct, so you would like to have a way to attach a DPL input signature to something which is effectively using an external source, correct? I think it's a valid use case, albeit I need to think how this could be done.

[ktf]

I am working on something which could provide this as a side effect.

[ktf]

So I think this can be modelled using a special "lifetime" (say "External"). The idea would be that the record expires immediately and it triggers a Receive on a channel which gets configured with appropriate ConfigParamSpecs (possibly constructed by default and injected in the configuration automatically, if not yet present. How does that sound?

[aphecetche]

well, I've been disconnected from the DPL for a while, so I guess I first have to get back to it a bit before I can comprehend your comment ;-)

[ktf]

Ok. ;-) I am away this week. I can prepare you an example once I am back.