ColorVectorSpace.jl icon indicating copy to clipboard operation
ColorVectorSpace.jl copied to clipboard

spurious underflow in norm

Open stevengj opened this issue 3 years ago • 0 comments

From discourse:

julia> using Colors, ColorVectorSpace, LinearAlgebra, FixedPointNumbers

julia> norm(RGB(0.008,0.008,0.008))
0.008

julia> norm(RGB{N0f8}(0.008,0.008,0.008))  # spurious underflow
0.0

We typically expect norm to avoid spurious underflow. It looks like you already do so for RGB{Float64}, since norm([RGB(1e-300, 1e-300, 1e-300)]) correctly returns ≈ 1e-300 rather than underflowing to zero.

I would suggest doing norm computation in the precision of the output (Float64) rather than in the precision of the input, as well as being careful about scaling.

stevengj avatar Aug 04 '22 14:08 stevengj