DecFP.jl icon indicating copy to clipboard operation
DecFP.jl copied to clipboard

incorrect RoundUp conversion of large strings

Open stevengj opened this issue 2 years ago • 2 comments

This is wrong, since the result should be exactly representable in Dec128:

julia> Dec128(big(10)^1000, RoundUp)
1.000000000000000000000000000000001e1000

Unfortunately, this might be a bug in the underlying library? All we are doing is converting the BigInt to a string and then calling the library's from_string routine.

As a result, we get incorrect results like:

julia> d128"1e100" == big(10)^100 # correct
true

julia> d128"1e1000" == big(10)^1000 # incorrect
false

stevengj avatar Dec 11 '23 16:12 stevengj

The results vary run to run. Most likely a buffer overflow in the C. I'll look into it.

julia> Dec128("1" * "0"^106, RoundUp)
1.000000000000000000000000000000001e106

julia> Dec128("1" * "0"^105, RoundUp)
1.0e105

julia> exit()
CD0004431999 1006% julia
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.9.4 (2023-11-14)
 _/ |\__'_|_|_|\__'_|  |  
|__/                   |

julia> using DecFP

julia> Dec128("1" * "0"^106, RoundUp)
1.0e106

jmkuhn avatar Dec 11 '23 19:12 jmkuhn