`ending-number` documentation doesn't fit shown regular expression

[lemzwerg]

In https://www.w3.org/2021/06/musicxml40/musicxml-reference/data-types/ending-number/ you can read the following (emphasis added by me).

The ending-number type is used to specify either a comma-separated list of positive integers without leading zeros, or a string of zero or more spaces. [...]

However, this description only approximates the shown regular expression. Since what is given in the documentation is hard to read (please show this with a typewriter font!), here is the same using Perl's/Python's 'extended' notation that allows arbitrary whitespace for better legibility.

  (
    [ ]*
  )
|  
  (
    [1-9] [0-9]*
    (
      , [ ]?
      [1-9] [0-9]*
    )*
  )

As can be seen. stuff like 1, 2, 3 is also allowed (note the [ ]? after the comma), which should be documented IMHO. This brings me to a strange feature: The regular expression allows exactly one single space after the comma. Why only one? Wouldn't it be better to use [ ]* instead to allow an arbitrary number of spaces?

I could even imagine to repace[ ] with \s, indicating any type of whitespace (tabs, for example).

[ahankinson]

\t (and any other whitespace character other than space) is not allowed in XML attributes. It will get normalized into #x20: https://www.w3.org/TR/REC-xml/#AVNormalize

[lemzwerg]

Ah, thanks, so please ignore this 🙂

[mscuthbert]

Thanks for the suggestion and repeal Werner.

Just as an FYI -- I'll make a change to monospace/typewriter if I can figure out how to do it quickly, but we are not switching to Perl/Python expanded regexes, mainly because every modern language supports standard regex but not every one supports verbose regex (Javascript, Java). Personally, I find that when discussing a regular expression that is mostly about parsing spaces, verbose regex is harder for me to read than non verbose. [1-9][0-9]* reads much more like a positive integer than [1-9] [0-9]* does to me.

[lemzwerg]

Oh, I didn't suggest to use a verbose regex in the documentation – I just used it here so that it can be quickly seen what I was talking about.

However, I suggest a single change for conformity: Irrespective of this issue, please replace the current

([ ]*)|([1-9][0-9]*(, ?[1-9][0-9]*)*)

with the equivalent

([ ]*)|([1-9][0-9]*(,[ ]?[1-9][0-9]*)*)

Thanks for the suggestion and repeal Werner.

Just to be clear: I repeal my suggestion about using \s. I still think that [ ]? should be replaced with [ ]*.

[ahankinson]

Quoting from the Attribute value normalization docs:

If the attribute type is not CDATA, then the XML processor must further process the normalized attribute value by discarding any leading and trailing space (#x20) characters, and by replacing sequences of space (#x20) characters by a single space (#x20) character.

So, in theory, multiple consecutive spaces are to be normalized by conforming processors.

In reality, you may not be using a conforming processor for data processing. But in effect foo bar and foo bar are equivalent attribute values. Thus [ ]? is really the most conformant regex when validating, and validators are likely the most conformant processors.

In fact I’m not really sure what [ ]* is doing as the first value?

[mscuthbert]

Oh, you're totally right @ahankinson -- I'll update back to the first version.

But only saw this after working with AI to help me write a to40.xsl template in XPath1.0 (much easier in 2.0--maybe we should move to that?) that changes 2, 3 to 2, 3 but leaves alone. Pasting it here below in case we later realize that we do need this, since I'm going to delete it without a commit.

  <!--
    MusicXML 4.1 allows multiple spaces after a comma in ending-number
    within endings.  MusicXML 4.0 and before do not.
  -->
  <xsl:template match="ending/@number">
      <xsl:attribute name="number">
          <xsl:call-template name="normalize-comma-spaces">
              <xsl:with-param name="text" select="."/>
          </xsl:call-template>
      </xsl:attribute>
  </xsl:template>

  <xsl:template name="normalize-comma-spaces">
      <xsl:param name="text"/>
      <xsl:choose>
          <xsl:when test="contains($text, ',  ')">
              <!-- Recursively replace multiple spaces after commas with a single space -->
              <xsl:call-template name="normalize-comma-spaces">
                  <xsl:with-param name="text" select="concat(substring-before($text, ',  '), ', ', substring-after($text, ',  '))"/>
              </xsl:call-template>
          </xsl:when>
          <xsl:otherwise>
              <xsl:value-of select="$text"/>
          </xsl:otherwise>
      </xsl:choose>
  </xsl:template>

[mscuthbert]

Which I'm glad I did save, because ending@number is in fact CDATA:

<!ELEMENT ending (#PCDATA)>
<!ATTLIST ending
    number CDATA #REQUIRED  
    ...

[mscuthbert]

Nope... that's in the old .mod file -- the new .xsd file lists it as xs:token which is derived from string but has multiple spaces removed:

https://www.w3schools.com/Xml/schema_dtypes_string.asp

	<xs:simpleType name="ending-number">
		<xs:annotation>
			<xs:documentation>The ending-number type is used to specify either a comma-separated list of positive integers without leading zeros, or a string of zero or more spaces. It is used for the number attribute of the ending element. The zero or more spaces version is used when software knows that an ending is present, but cannot determine the type of the ending.</xs:documentation>
		</xs:annotation>
		<xs:restriction base="xs:token">
			<xs:pattern value="([ ]*)|([1-9][0-9]*(, ?[1-9][0-9]*)*)"/>
		</xs:restriction>
	</xs:simpleType>

So there is a conflict in the spec (as far as I can tell). As of v4.0 (maybe 3.1? can't remember), the Schema .XSD is the correct one.

(Sorry folks, 2 hrs on MusicXML is the most I can devote to it today. Returning tomorrow).

[mdgood]

In fact I’m not really sure what [ ]* is doing as the first value?

That's to allow an empty string as the attribute value, but there's likely a better way to do that. If [ ]? is indeed equivalent to ? in this context, I can see that adding readability in the second part.

[mscuthbert]

In fact I’m not really sure what [ ]* is doing as the first value?

That's to allow an empty string as the attribute value, but there's likely a better way to do that. If [ ]? is indeed equivalent to ? in this context, I can see that adding readability in the second part.

If all leading and trailing spaces are trimmed then an empty string is equal to one or more spaces.

Given that the older DTD says that spaces are significant here and the newer Schema XSD says they're not, I'm happy to go in whatever direction you remember consciously deciding on (if any) otherwise we can follow the general rule that the XSD is now in charge.

(PS Michael--just got a video of you addressing my MIT class that they put up: you are amazing with the students!)

[ahankinson]

If all leading and trailing spaces are trimmed then an empty string is equal to one or more spaces.

Leading and trailing spaces are discarded according to the docs. So an empty string foo=" " is the same as foo=""

[mdgood]

To reiterate, ending-number is a subset of a token, and tokens "have no internal sequences of two or more spaces." See the definition at https://www.w3.org/TR/xmlschema-2/#token.

[mscuthbert]

See #588 -- to prevent the confusion, the first part is changed to the empty group () rather than expanding the second part to [ ]? (or my temporarily erroneous [ ]*). As an empty group, the first section as " " would have leading/trailing whitespace.