OpenMetadata icon indicating copy to clipboard operation
OpenMetadata copied to clipboard

Add support for the profiler for ingestion time partition in BQ (`_PARTITIONTIME`/`_PARTITIONDATE`)

Open TeddyCr opened this issue 3 years ago • 2 comments

Context

Profiler and data quality currently supports BQ partitioned table with explicit fields (i.e. fields that are present in the table), though BQ allows users to partition tables by ingestion time using pseudo column _PARTITIONTIME or _PARTITIONDATE.

Solution

Implement support for pseudo column predicate in sampler's and runner's query

A.C.

  • Users are able to run profiler and tests against column with ingestion time partition

TeddyCr avatar Jun 15 '22 05:06 TeddyCr

Working on adding Ingestion Time on the ingestion side.

ayush-shah avatar Sep 15 '22 04:09 ayush-shah

Added a fix in https://github.com/open-metadata/OpenMetadata/pull/7506

ayush-shah avatar Sep 19 '22 10:09 ayush-shah