strange floating point decimals

mr_moon

Does anyone else notice this with number input in the editor? Extra decimals at the end for some reason. They don't appear until you hover over them and I can't get rid of them.

Megalomaniak

Can't get rid of the tooltip/hint message or the precision error?

cybereality

It's like that for me too. Not sure if it was always like that, I never rolled over the number before. I would assume that the numbers are declared as floats, and you can't necessarily represent whole integers evenly, though I guess it crops it on display.

mr_moon

@Megalomaniak said: Can't get rid of the tooltip/hint message or the precision error?

The tool tip works as expected. I just can't remove the precision error.

cybereality

I'm not sure that's actually an error though. We'd have to know how the editor stores those values internally.

Megalomaniak

https://floating-point-gui.de

https://floating-point-gui.de/basic/ Why don’t my numbers, like 0.1 + 0.2 add up to a nice round 0.3, and instead I get a weird result like 0.30000000000000004? Because internally, computers use a format (binary floating-point) that cannot accurately represent a number like 0.1, 0.2 or 0.3 at all.

When the code is compiled or interpreted, your “0.1” is already rounded to the nearest number in that format, which results in a small rounding error even before the calculation happens.

mr_moon

It may have to do with how the floating point number is represented in binary. I noticed this issue when working with python too.

Source: http://effbot.org/pyfaq/why-are-floating-point-calculations-so-inaccurate.htm