databrickslabs
[FEATURE]: Add a flag to not list all tables when calling lint-local-code
Is there an existing issue for this?
- [x] I have searched the existing issues
Problem statement
When using lint-local-code, the code is listing all the tables in hive_metastore for some reason.
This behaviour should be overriden as it slows down the performance of the command
Proposed Solution
databricks labs ucx lint-local-code --no-table-listing
Additional Context
No response
@william-conti : It does that to update the table migration index so that it can recommend to which targets Hive metastore tables should be upgraded to.
Did you migrate any tables? If not, then it will indeed list the tables all the time. If so, it only needs to do the listing once, which already happened at the end of the table migration workflow.
As a work around, in the config, set include_databases to a dummy database. (It will warn about the dummy database not existing, but continue.) This will scope the tables listing to that database. https://github.com/databrickslabs/ucx/pull/3681 contains a change that allow you to set include_databases to en empty list, excluding all databases.
@JCZuurmond No we haven't migrated any tables to UCX. For additional context, we didn't even ran the assessment in the customer context here.
Our goal here is to leverage lint-local-code to accelerate our code migration and identify all possible places we will need to migrate without having to understand the code complexity. We expected the linter to work like any Python linter, but there are some complexities behind the scenes that confuses us
Oke, thanks for your input, @william-conti . I suggest we introduce a --offline flag for the local-code-linter to hide the specificity of table listing and including possible feature listings (like #2021)
Related issue: #2378