querybook icon indicating copy to clipboard operation
querybook copied to clipboard

feat: Allow metastore loaders to provide table/column descriptions

Open baumandm opened this issue 3 years ago • 0 comments

This change adds description properties to DataTable and DataColumn, and persists these to the database (if not None).

This allows a metastore loader implementation to source the table and column descriptions from a separate system and sync them into Querybook.

Here's an example implementation that hardcodes the same descriptions for every table & column:

from typing import List, Tuple
from lib.logger import get_logger
from lib.metastore.base_metastore_loader import DataColumn, DataTable
from lib.metastore.loaders.sqlalchemy_metastore_loader import SqlAlchemyMetastoreLoader

LOG = get_logger(__name__)

class ExternalMetadataLoader(SqlAlchemyMetastoreLoader):
    def get_table_and_columns(
        self, schema_name, table_name
    ) -> Tuple[DataTable, List[DataColumn]]:
        table, columns = super().get_table_and_columns(schema_name, table_name)

        table = table._replace(description="test table description")

        columns = list(
            map(
                lambda column: column._replace(description="test column description"),
                columns,
            )
        )
        
        return table, columns

Fixes #871.

baumandm avatar May 25 '22 20:05 baumandm