spark icon indicating copy to clipboard operation
spark copied to clipboard

[SPARK-49140][SQL] Widening type promotion from int/long to decimal and support custom decimal logic type for int in `AvroDeserializer`

Open wayneguow opened this issue 1 year ago • 2 comments

What changes were proposed in this pull request?

This PR aims to:

  1. Widening type promotion from int/long to decimal in AvroDeserializer. Supported as following(Avro Type -> Spark Type):
  • Int -> DecimalType;
  • Long -> DecimalType ;
  1. Support custom decimal logic type for int We already support long in #41409.

Why are the changes needed?

Similar to Parquet reader, we'd better to enable type promotion/widening for Avro deserializer.

Does this PR introduce any user-facing change?

Yes, Users can use wider types more flexibly.

How was this patch tested?

Pass GA and add some new test cases.

Was this patch authored or co-authored using generative AI tooling?

No.

wayneguow avatar Aug 07 '24 10:08 wayneguow

cc @dongjoon-hyun @cloud-fan thanks.

wayneguow avatar Aug 07 '24 13:08 wayneguow

@dongjoon-hyun Could you help to also review it when you have some time, cause that this PR contains some similar feature that you reviewed before(related PR: support custom decimal logic type for long). Thanks ~

wayneguow avatar Aug 15 '24 01:08 wayneguow

We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable. If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!

github-actions[bot] avatar Nov 24 '24 00:11 github-actions[bot]