Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix Color precision error in the documentation generated on M4 macOS. #104112

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

bruvzg
Copy link
Member

@bruvzg bruvzg commented Mar 14, 2025

Running doctool on M4 Pro macOS (not reproducible on M1) result in unnecessary diffs.

The issue is related to use of atan2f and precision of π:

ok_color.h header define π as 3.1415926535897932384626433832795028841971693993751058209749445923078164062 which is rounded to the nearest representable float value 3.14159274101257324219, but atan2f(-0.0, -0.0) (and some other math functions with negative argument) seems to round it the other way and return 3.14159250259399414062 (previous representable value, one bit less).

@lawnjelly
Copy link
Member

lawnjelly commented Mar 14, 2025

To be honest I'm not sure this is the right / best way to solve this.

Rounding errors are expected to be different on different platforms.

To elaborate:

  • The documentation is essentially asserting that you'd get the same result when running doctool on a different platform, which is NOT the case for floating point.
  • Changing the core code to bodge around this is kind of a case of the "tail wagging the dog", so to speak. It will slow down these functions, just to work around a bug / design flaw in the doctool.
  • Now there could be a case for doing this for numerical reasons in core, but that case should be made separately as the result of a real problem seen by users, as it has a cost.

As for some other ideas how to solve:

  • Have the doctool use a limited number of sig figs such that these rounding errors don't change the result.
  • Also could cap values close to 0 and 1 (or maybe all whole numbers) to the whole number if they are within a small margin.

@akien-mga
Copy link
Member

As for some other ideas how to solve:

  • Have the doctool use a limited number of sig figs such that these rounding errors don't change the result.
  • Also could cap values close to 0 and 1 (or maybe all whole numbers) to the whole number if they are within a small margin.

Yeah I agree that something like this may be a better approach.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants