bigdecimal icon indicating copy to clipboard operation
bigdecimal copied to clipboard

Inconsistent max precision affecting dump&load result

Open atitan opened this issue 3 years ago • 0 comments

We've observed that BigDecimal treats number and string differently on max precision allocation.

Ruby 2.6.6

# precs returns [SIZET2NUM(p->Prec*VpBaseFig()), SIZET2NUM(p->MaxPrec*VpBaseFig())]
BigDecimal(123).precs
=> [9, 27]
BigDecimal('123').precs
=> [9, 18]

BigDecimal(123456789).precs
=> [9, 27]
BigDecimal('123456789').precs
=> [9, 18]

BigDecimal(123456789012).precs
=> [18, 27]
BigDecimal('123456789012').precs
=> [18, 27]

Ruby 3.1.2

# precs returns [SIZET2NUM(p->Prec*VpBaseFig()), SIZET2NUM(p->MaxPrec*VpBaseFig())]
BigDecimal(123).precs
=> [9, 9]
BigDecimal('123').precs
=> [9, 18]

BigDecimal(123456789).precs
=> [9, 9]
BigDecimal('123456789').precs
=> [9, 18]

BigDecimal(123456789012).precs
=> [18, 18]
BigDecimal('123456789012').precs
=> [18, 27]

The problem is also applied to dump & load, which is troublesome.

Especially when taking serialized result for comparison to check for duplicates or data changes.

Ruby 2.6.6

BigDecimal(123)._dump
=> "27:0.123e3"
BigDecimal._load("27:0.123e3")._dump
=> "18:0.123e3"

BigDecimal(123456)._dump
=> "27:0.123456e6"
BigDecimal._load("27:0.123456e6")._dump
=> "18:0.123456e6"

BigDecimal(123456789)._dump
=> "27:0.123456789e9"
BigDecimal._load("27:0.123456789e9")._dump
=> "27:0.123456789e9"

BigDecimal(123456789012)._dump
=> "27:0.123456789012e12"
BigDecimal._load("27:0.123456789012e12")._dump
=> "27:0.123456789012e12"

Ruby 3.1.2

BigDecimal(123)._dump
=> "9:0.123e3"
BigDecimal._load("9:0.123e3")._dump
=> "18:0.123e3"

BigDecimal(123456)._dump
=> "9:0.123456e6"
BigDecimal._load("9:0.123456e6")._dump
=> "18:0.123456e6"

BigDecimal(123456789)._dump
=> "9:0.123456789e9"
BigDecimal._load("9:0.123456789e9")._dump
=> "18:0.123456789e9"

BigDecimal(123456789012)._dump
=> "18:0.123456789012e12"
BigDecimal._load("18:0.123456789012e12")._dump
=> "18:0.123456789012e12"

atitan avatar Jun 23 '22 06:06 atitan