font-rs icon indicating copy to clipboard operation
font-rs copied to clipboard

f32 -> u8 conversion should multiply by 256-ε instead of by 255

Open nigeltao opened this issue 9 years ago • 1 comments

The Rust code has SIMD and non-SIMD versions of this conversion (255.0 * y) as u8 from an f32 in the range [0.0, 1.0] to a u8 in the range [0, 255].

Due to rounding errors, this sometimes means that a pixel well inside a glyph ends up with an 0xfe value instead of 0xff, since its floating point value is slightly less than 1.

In my Go port, I embiggened the constant from 255.0 to 255.99998, or 0x437fffff as a float32 bit pattern, which eliminated these central "0xfe"s. See the comment at the bottom of https://github.com/google/font-go/commit/7c31afdef0ec8d074834b29413aa097de0d51197

I think that the Rust code should do the same.

nigeltao avatar Aug 28 '16 04:08 nigeltao

I'd suggest using centered quantization which is widely used in games, basically it can be implemented by using round instead of floor in the final step of calculation.

Though I don't think accuracy matters in the case of fonts (apart from avoiding triggering a different codepath due to 0xff turning into 0xfe), so your approach also seems OK.

ishitatsuyuki avatar Nov 04 '20 12:11 ishitatsuyuki