Incorrect mm to pt conversion due to floats

[timbaas]

Hi there!

We use OpenPDF to add bleed and trim boxes to PDF's exported with chrome/puppeteer. The size of the bleed and trim boxes are defined by a Rectangle which constructs out of 4 floats. We use mm to define the sizes, so we convert these to points first and that's when the problem occurs.

For example:

126mm = 357.165354330708661417680 (using BigDecimal)

But, with floats:

126mm = 357.16534423828125 (using Float)

Because of this I'm not able to export with bleed and trim boxes that exactly match the mm values (126mm becomes 126.00164mm).

I understand that floats aren't accurate, but I'm wondering if there is any other way to go around this.

[asturio]

How are you converting mm to pt? Because the error is quite big here. Already at 1E-5. Maybe you can try some rounding.

You are using a constant in the conversions:

1. 2.83464566929133858268
2. 2.83464558919270833333

I calculated this as a constant for transforming mm in pt: 0. 2.83501683501683501683

So 126 * 2.83501683501683501683 = 357.21212121212121212058

And converting back I get: 357.21212121212121212058 / 2.83501683501683501683 = 126

Another thing you can try is to use BigDecimal for all calculations, and then extract the float value from the BigDecimal result.

[asturio]

Can't get where the conversion is made. Closing as not enough info.