cli icon indicating copy to clipboard operation
cli copied to clipboard

PythonMutator: propagate source locations

Open kanterov opened this issue 1 year ago • 0 comments

Changes

Add a mechanism to load Python source locations in the Python mutator. Previously, locations pointed to generated YAML. Now, they point to Python sources instead. Python process outputs "locations.json" containing locations of bundle paths, examples:

{"path": "resources.jobs.job_0", "file": "src/examples/job_0.py", "line": 3, "column": 5}
{"path": "resources.jobs.job_0.tasks[0].task_key", "file": "src/examples/job_0.py", "line": 10, "column": 5}
{"path": "resources.jobs.job_1", "file": "src/examples/job_1.py", "line": 5, "column": 7}

Such locations form a tree, and we assign locations of the closest ancestor to each dyn.Value based on its path. For example, resources.jobs.job_0.tasks[0].task_key is located at job_0.py:10:5 and resources.jobs.job_0.tasks[0].email_notifications is located at job_0.py:3:5, because we use the location of the job as the most precise approximation.

This feature is not yet enabled by default because PyDABs don't yet ignore unknown arguments, and this will be only supported with 0.6.0.

experimental:
  pydabs:
    enabled: true
    import: [...]
    load_locations: true

Note: for now, we don't update locations with relative paths, because it has a side effect in changing how these paths are resolved

Example

% databricks bundle validate

Warning: job_cluster_key abc is not defined
  at resources.jobs.examples.tasks[0].job_cluster_key
  in src/examples/example.py:10:1

Tests

Unit tests and manually

Stacked on top of https://github.com/databricks/cli/pull/1782

kanterov avatar Sep 20 '24 15:09 kanterov