dbx icon indicating copy to clipboard operation
dbx copied to clipboard

[BUG] Can't specify parameters with python_wheel_task

Open klesouza opened this issue 3 years ago • 1 comments

Expected Behavior

I can pass --parameters using dbx launch --as-run-submit for python_wheel_task

Current Behavior

It fails when --parameters or --parameters-raw is specified. Exception

Traceback (most recent call last):
  File ".venv/bin/dbx", line 8, in <module>
    sys.exit(cli())
  File ".venv/lib/python3.8/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File ".venv/lib/python3.8/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "venv/lib/python3.8/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File ".venv/lib/python3.8/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File ".venv/lib/python3.8/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File ".venv/lib/python3.8/site-packages/dbx/commands/launch.py", line 148, in launch
    run_data, job_id = run_launcher.launch()
  File ".venv/lib/python3.8/site-packages/dbx/commands/launch.py", line 297, in launch
    task_key = [k for k in job_spec.keys() if k in POSSIBLE_TASK_KEYS][0]
IndexError: list index out of range

Steps to Reproduce (for bugs)

Create a deployment.yml with a python_wheel_task and run dbx launch --as-run-submit --job job_name --parameters="param=param"

Context

I think it's just the matter of adding the python_wheel_task to this list https://github.com/databrickslabs/dbx/blob/main/dbx/commands/launch.py#L28.

Unless it was left out on purpose

Your Environment

  • dbx version used: 0.4.1
  • Databricks Runtime version: 9.1

klesouza avatar Mar 22 '22 17:03 klesouza

This seems to not work for all types of tasks (i.e. for me also spark_python_task - dbx 0.6.2) in the multi-task setup as the task key is one level lower inside tasks array. It works in the UI so I assume that there parameters are passed to all tasks from the list?

collerek avatar Jun 28 '22 14:06 collerek

fixed in the referenced PR, will be released in 0.7.0, thanks a lot for opening the issue @klesouza

renardeinside avatar Aug 19 '22 21:08 renardeinside