qziyuan

[FEATURE]: cluster assessment should assess MLR with shared mode cluster

2

### Is there an existing issue for this? - [X] I have searched the existing issues ### Problem statement Right now the cluster assessment treat MLR as supported. However, if...

[FEATURE]: Migrate tables in unsupported filesystem

1

### Is there an existing issue for this? - [X] I have searched the existing issues ### Problem statement External tables stored in `adl://` and `wasbs://` will be crawled and...

[FEATURE] `ValidationResultOperation` for UC storage credential validation should include `PATH_EXISTS` and `HIERARCHICAL_NAMESPACE_ENABLED`

1

**Problem Statement** When validate UC storage credential using SDK, the returned response will contain two None operations. It's because `PATH_EXISTS` and `HIERARCHICAL_NAMESPACE_ENABLED` are missing in [ValidationResultOperation](https://github.com/databricks/databricks-sdk-py/blob/a282eee599c3176ccf33a00429356a4627c1b977/databricks/sdk/service/catalog.py#L5376C7-L5382) **Proposed Solution** Support those...

Fix missing `security_policy` when updating SQL warehouse config

1

1. Fix `InvalidParameterValue: Endpoint security policy is required and must be one of NONE, DATA_ACCESS_CONTROL, PASSTHROUGH.` error in #2405 2. Print new SQL warehouse data access config in the log...

[BUG]: `create-uber-principal` command failed when updating the SQL warehouse config

1

### Is there an existing issue for this? - [X] I have searched the existing issues ### Current Behavior 1. `create-uber-principal` will fail with `ERROR [d.l.ucx.create-uber-principal] InvalidParameterValue: Endpoint security policy...

### Is there an existing issue for this? - [x] I have searched the existing issues ### Problem statement **This issue is a place holder for now, because the upstream...

Uses SparkSession/Dataframe over SparkContext/RDD when writing model metadata

1

With https://issues.apache.org/jira/browse/SPARK-48909, Spark ML is now using SparkSession and Dataframe API to write model metadata for example: `spark.createDataFrame(Seq(Tuple1(metadataJson))).write.text(metadataPath)` However, SynapseML still relies on SparkContext and the RDD API in places...