ucx icon indicating copy to clipboard operation
ucx copied to clipboard

[FEATURE]: migrate jdbc tables to Lakehouse Federation

Open pohlposition opened this issue 1 year ago • 0 comments

Is there an existing issue for this?

  • [X] I have searched the existing issues

Problem statement

Some customers register jdbc tables in their HMS. These tables need to be registered as Lakehouse Federation 'FOREIGN' catalogs within UC (with the appropriate connection). This functionality has not been built into UCX yet. Lakehouse Federation is now Public Preview.

Proposed Solution

Assessment

  1. Highlight all tables that are jdbc tables.
  2. Nice to have: parse out the URL to show the database server and database that each table maps to.
  3. Nice to have: do a test against each table to verify if the underlying table still exists or not.

Mapping Tables

  1. Allow user to map each jdbc table to a new home in UC
  2. In UC all of the tables for a particular database will need to reside within a "FOREIGN CATALOG"
  3. It is possible that the same physical jdbc table is registered multiple times within a HMS and will need to be mapped to a single object within UC

Upgrading Tables

  1. Manage the database credentials securely into a UC Connection Object
  2. There could be multiple credentials used in an HMS to connect via JDBC (with varying levels of access). Need to pick one.
  3. Need to make sure that jobs, queries, and views reference the newly consolidated UC objects in their code

Additional Context

No response

pohlposition avatar Jul 15 '24 20:07 pohlposition