pixie icon indicating copy to clipboard operation
pixie copied to clipboard

Track table truncation due to limit nodes

Open sync-by-unito[bot] opened this issue 3 years ago • 0 comments

We use limit nodes for both implicit and explicit table truncation.

  • Explicit truncation is done via px.head in the script.
  • Implicit truncation is done via the max_output_rows_per_table flag.

A user would like to know which tables if any were truncated due to a limit node triggering. Tracking implicit truncation is especially important since we might need to alert the user that some results may be incomplete.

We should add this information at the per table level (instead of the exec stats level) so that we have granular info on which tables were truncated.

Document detailing where this data might come from → Still a WIP: https://www.notion.so/pixielabs/Tracking-Table-Truncation-839e9723934a4f129cbad61e2a678027

┆Issue is synchronized with this Jira Story by Unito

sync-by-unito[bot] avatar Jul 12 '22 05:07 sync-by-unito[bot]