piperider icon indicating copy to clipboard operation
piperider copied to clipboard

Support non-metric based assertions

Open popcornylu opened this issue 3 years ago • 0 comments

Summary

Currently, the assertion rules are all based on the profiling result. However, it is not enough as a data quality tool. There are some assertion can not work by profiling

  1. Format check: The text column match certain criteria.
  2. Column values in set: The value of the column should be only valid in certain set.
  3. Reference check: Like foreign key constraint in db. Check if value in a column exist in another table's certain column

Intended Outcome

  1. Add at least 3 non-metric based test
  2. Design the foundation of non-metric assertions.

How will it work?

  1. Run SQL query rather than validate the value from profiling result.

popcornylu avatar Sep 29 '22 01:09 popcornylu