DecFP.jl
DecFP.jl copied to clipboard
incorrect RoundUp conversion of large strings
This is wrong, since the result should be exactly representable in Dec128:
julia> Dec128(big(10)^1000, RoundUp)
1.000000000000000000000000000000001e1000
Unfortunately, this might be a bug in the underlying library? All we are doing is converting the BigInt to a string and then calling the library's from_string routine.
As a result, we get incorrect results like:
julia> d128"1e100" == big(10)^100 # correct
true
julia> d128"1e1000" == big(10)^1000 # incorrect
false
The results vary run to run. Most likely a buffer overflow in the C. I'll look into it.
julia> Dec128("1" * "0"^106, RoundUp)
1.000000000000000000000000000000001e106
julia> Dec128("1" * "0"^105, RoundUp)
1.0e105
julia> exit()
CD0004431999 1006% julia
_
_ _ _(_)_ | Documentation: https://docs.julialang.org
(_) | (_) (_) |
_ _ _| |_ __ _ | Type "?" for help, "]?" for Pkg help.
| | | | | | |/ _` | |
| | |_| | | | (_| | | Version 1.9.4 (2023-11-14)
_/ |\__'_|_|_|\__'_| |
|__/ |
julia> using DecFP
julia> Dec128("1" * "0"^106, RoundUp)
1.0e106