Neil Best

https://github.com/databrickslabs/overwatch/blob/63d43d57629ec5c4cb2224a8240c176ad007a11b/docs/dataengineer/pipeline_management/index.html#L1797 Browser provides no feedback when clicking on this non-existent anchor link in the "Reloading Data Example" section within the red "Warning" `div`. This leaves the reader to wonder what...

Consider adding flag `export_only_success_libraries` or similar to cluster stanza in export config YAML.

secret scope names containing slashes cause failure

3

```bash databricks secrets list-scopes --profile bethesda2 ``` returns: ``` Scope Backend KeyVault URL --------------------------------------------- ---------- -------------- adaptive-planning-scope DATABRICKS N/A BI/Databricks/ForumUsers DATABRICKS N/A . . . ``` Secrets migration: ```bash python...

Recursive workspace listing omits directories

5

I would argue that directories should be included in the result set as they are when `recursive=False`. Users of Unix `find` will recognize and appreciate this convention. Let the caller...

preserve directories in recursive workspace listing

1

## Changes ## Tests - [x] `make test` run locally - [x] `make fmt` applied - [x] relevant integration tests applied

adjust Silver Job Runs module configuration

2

# enable auto-optimized shuffle for module 2011 originally implemented for Spark 3.1.2 in commit https://github.com/databrickslabs/overwatch/commit/d751d5fc75c939892b73f877cb0e5542eb2cc030 on branch [`1228-silver-job-runs-spark312-r0812`](https://github.com/databrickslabs/overwatch/tree/1228-silver-job-runs-spark312-r0812) as part of #1253. This PR removes all of the new utilities...

Refactor lookups in Silver Job Runs

8

[ Original description moved [here](https://github.com/databrickslabs/overwatch/pull/1256#issuecomment-2229012714) in #1256. ] The net effect of this PR is to refactor a number of transformations that are part of Silver Job Runs (module 2011)...

Add descriptive Job Group ID and `NamedTransformation`s to Spark UI

5

This makes the Spark UI more developer-friendly when analyzing Overwatch runs. Job group IDs have the form : Any use of `.transform( df => df)` may be replaced with `.transformWithDescription(...