[FEATURE]: Regression testing harness for post-migration validation of configurations (Workload Assets)

[chase-edwards-db]

Is there an existing issue for this?

• [X] I have searched the existing issues

Problem statement

Problem statement

• #927 for Data-related component As a customer, much of the risk and work of a large migration is testing and validation. There is currently no opinionated pattern from Databricks on how to do regression testing. UC migrations provide an impetus to build one, as this type of migration has high risk.

Proposed Solution

The feature should:

• Account for any workload assets being migrated, including dependencies (cluster configurations, policies, ...)
• When run, validate post-migration workloads against their pre-migration counterparts. This could take the form of completeness e.g. has the migration of assets occurred AND/OR a separate process to measure successful execution of migrated workloads. These should be split processes.
• Show a view of the outcome of the migration, including rollup success metrics e.g. "% of workflows/clusters/policies/etc successfully migrated", "# of workflows/jobs migrated". Regression can be classified as workflow-related asset migration failure for the asset migration AND/OR metrics related to the success/failure of workflow execution post-migration. Failures can be detected by capturing failures from the workflow asset migration tooling in UCX (pending) as well as failures upon first run of migrated assets.
• Provide a useful changelog of asset changes made that can be analyzed further. Similar to a system table. Example rough schema (includes many optional fields, can be split into denormalized asset-specific tables):

For migration completeness: asset_type | asset_name | migrated_flag

For migration workflow success: workflow_name | execution_duration | success_flag | error_message

Additional Context

If any amount of rollback is possible in these kinds of migrations, that should also be considered in this feature's design.