spark icon indicating copy to clipboard operation
spark copied to clipboard

[WIP][SPARK-49631][SQL] Remove the ANSI config suggestion in ARITHMETIC_OVERFLOW

Open mihailomilosevic2001 opened this issue 1 year ago • 2 comments

What changes were proposed in this pull request?

This PR changes the message of ARITHMETIC_OVERFLOW error and adds relevant try expressions:

  • try_make_interval
  • try_conv
  • try_round
  • try_bround

Also this PR fixes hints for:

  • sum
  • average

Some expressions never hit errors, but are present in the inheritance of the base classes, so are addressed as well:

  • ceil
  • floor

This PR also removes _LEGACY_ERROR_TEMP_2042 as it is as it is a suberror of ARITHMETIC_OVERFLOW.

As this PR got pretty big, improvement for MathUtils functions will be done in a separate PR.

Why are the changes needed?

ARITHMETIC_OVERFLOW contains suggested fix for turning off ANSI mode. Now that in Spark 4.0.0 we have moved to ANSI mode on by default, we want to keep suggestions of this kind to the minimum. There exist implementations of try_* functions which provide safe way to get behaviour as for ANSI mode off and suggestions of this kind should be sufficient.

In this case, try expressions were missing so new expressions were added to patch up the missing implementations.

Does this PR introduce any user-facing change?

Yes, error message will change.

How was this patch tested?

Existing tests cover message change, but try expressions are tested separately.

Was this patch authored or co-authored using generative AI tooling?

No.

mihailomilosevic2001 avatar Sep 23 '24 07:09 mihailomilosevic2001

FYI there is a similar effort in https://github.com/databricks/runtime/pull/82979 But it got reverted https://github.com/databricks/runtime/pull/98628/ Double confirm: have we checked with Kent Marten on this one?

gengliangwang avatar Oct 04 '24 18:10 gengliangwang

@mihailom-db how about splitting it to multiple PRs?

gatorsmile avatar Oct 07 '24 23:10 gatorsmile

We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable. If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!

github-actions[bot] avatar Mar 31 '25 00:03 github-actions[bot]