MachineLearningNotebooks icon indicating copy to clipboard operation
MachineLearningNotebooks copied to clipboard

Potential feature request: PipelineArgument dynamically derived from PipelineParameter

Open mparq opened this issue 3 years ago • 2 comments

Hello,

I couldn't find a pre-existing issue or documentation on this topic. Is it possible to create pipeline parameters which are dynamically generated from the values of other PipelineParameters?

e.g. the below shows the basic scenario we're looking for.


customer_project = PipelineParameter(name="customer_project")
publish_version = PipelineParameter(name="publish_version")

# following is generated from above pipeline parameters but can still be used as an argument in Pipeline Steps
input_data_parameter = PipelineArgument(name="input_data", value_spec="base_input/{param:customer_project}/{param:publish_version}/train")
input_data = (input_data_parameter, DataPathComputeBinding(mode='mount'))
...
train_step = PythonScriptStep(
  ...
  inputs=[input_data]
)

Is there a better way to do what I'm trying to do here?

Currently we have some workarounds like have a "parent" PythonScriptStep which just handles submitting child runs, and generating the right paths from pipeline arguments, but it seems like an extra step and ability to have something like above would be nicer for us.

mparq avatar Apr 26 '22 21:04 mparq

Hello,

I couldn't find a pre-existing issue or documentation on this topic. Is it possible to create pipeline parameters which are dynamically generated from the values of other PipelineParameters?

e.g. the below shows the basic scenario we're looking for.

customer_project = PipelineParameter(name="customer_project")
publish_version = PipelineParameter(name="publish_version")

# following is generated from above pipeline parameters but can still be used as an argument in Pipeline Steps
input_data_parameter = PipelineArgument(name="input_data", value_spec="base_input/{param:customer_project}/{param:publish_version}/train")
input_data = (input_data_parameter, DataPathComputeBinding(mode='mount'))
...
train_step = PythonScriptStep(
  ...
  inputs=[input_data]
)

Is there a better way to do what I'm trying to do here?

Currently we have some workarounds like have a "parent" PythonScriptStep which just handles submitting child runs, and generating the right paths from pipeline arguments, but it seems like an extra step and ability to have something like above would be nicer for us.

singha04 avatar Apr 27 '22 07:04 singha04

@singha04 , I think some of your response got cut off. I only see you quoting my post

mparq avatar Apr 27 '22 14:04 mparq