piperider
piperider copied to clipboard
Support non-metric based assertions
Summary
Currently, the assertion rules are all based on the profiling result. However, it is not enough as a data quality tool. There are some assertion can not work by profiling
- Format check: The text column match certain criteria.
- Column values in set: The value of the column should be only valid in certain set.
- Reference check: Like foreign key constraint in db. Check if value in a column exist in another table's certain column
Intended Outcome
- Add at least 3 non-metric based test
- Design the foundation of non-metric assertions.
How will it work?
- Run SQL query rather than validate the value from profiling result.