[FEATURE]: cluster assessment should assess MLR with shared mode cluster

[qziyuan]

Is there an existing issue for this?

• [X] I have searched the existing issues

Problem statement

Right now the cluster assessment treat MLR as supported. However, if the customer has MLR cluster that are being shared by multiple users (HC cluster without table ACL or standard cluster), this cluster cannot be simply upgraded to shared mode cluster, since MLR is not supported in shared mode cluster. Further discussion is required for upgrading this type of cluster:

• Is service principal cluster acceptable?
• Do users use unsupported ML libs? Can they use normal DBR and install required and supported ML libs as cluster or session level libraries?
• Can users separate their training code to a single user mode cluster with MLR and run non ML code in shared mode cluster?

Proposed Solution

1. At least we should call out all MLR clusters, and inform the customer that further discussion maybe required.
2. Furthermore, we can use similar solution in #810 to leverage the audit system table to verify if a MLR cluster is really being shared by multiple users. If shared, then we call out that upgrade this cluster need further discussion.

Additional Context

No response

[dmoore247]

Covered by https://github.com/databrickslabs/ucx/pull/1123 Enhance assessment.md and interactive report according to specs above.

[pohlposition]

Do users use unsupported ML libs? Can they use normal DBR and install required and supported ML libs as cluster or session level libraries?

Is it possible to detect and assess this? If so, that could be very useful (and worth a separate FEATURE)