Skip to content

Clarify how string conversion functions in signature REAL treat the sign of a NaN value #33

@pclayton

Description

@pclayton

In the signature REAL, the conversion functions fromDecimal and toDecimal ensure that the sign field of the IEEEReal.decimal_approx value matches the sign bit of the real value even if the value is NaN.

The description of the functions scan and fromString does not specify how the sign bit is derived in the case of an argument giving the NaN value, so an implementation may arbitrarily choose a value for the sign bit. Is this underspecified behavior intended? If the intention was to preserve the sign bit through conversion of NaN, could we say:

  • fromString is equivalent to Option.composePartial (fromDecimal, IEEEReal.fromString)?
  • scan is equivalent to Option.composePartial (mapPartialFirst fromDecimal, IEEEReal.scan)
    where fun mapPartialFirst f (a, b) = case f a of SOME a' => SOME (a', b) | NONE => NONE?

The description of the function toString says that a NaN value is converted to the string "nan". This ignores the sign bit of the NaN value. Therefore toString is not equivalent to IEEEReal.toString o Real.toDecimal. Is this intended?

This issue is raised as suggested in smlnj/legacy#73 (comment).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions