Attribute values do not escape whitespace charecters on output

[fsateler]

As per the XML Specification, a compliant parser must transform all whitespace characters to space (0x20) on parsing. This means that if one wants to preserve the spacing, the value should be encoded. Ox does not do the encoding. Therefore:

el = Ox::Element.new('element')
el["attr"] = "I have\nnewlines"
puts Ox.dump(el)
# 
# <element attr="I have
# newlines"/>

And a compliant parser should discard the \n there and interpret it as . This means that by default Ox does not roundtrip.

Ox should do something like value.gsub(/([^[:ascii:]]|[\s&&[^ ]])/){|i| "&##{i.ord};" } on attribute values

[ohler55]

Have you looked at the :skip option? In particular, setting :skip to :skip_return.

[fsateler]

I believe that option only works for parsing. The problem is that the output will be parsed by any number of xml parsers. In particular, libxml (and all parsers built on top of that, like Nokogiri or other language parsers) will interpret the above xml as <element attribute="I have newlines"/> (ie, drop de newline).

[fsateler]

Just for further information, I am not using Ox directly, but rather indirectly as a dependency of https://github.com/aespinosa/minitest-junit . The errors reported with junit are badly formatted because any newlines are being swallowed. If the original error message was:

Expected: 1
  Actual: 2

Jenkins is showing:

Expected: 1 Actual: 2

[ohler55]

I think I see. You want a writer option to replace white space with a single space. Is that correct?

[fsateler]

No. It'd like Ox to replace the (non regular, 0x20) whitespace with the xml code (&#60 for newlines), so that the documents can be correctly parsed by all xml parsers.

[fsateler]

I realized my above regex was wrong. Here is the fixed regex: /([^[:ascii:]]|[\s&&[^ ]])/ (not ascii, or (whitespace AND not space)). I fixed the original comment

[ohler55]

I hope when you said (&#60 for newline) that was a typo. Newline is &#a; when escaped in an XML string. It is still unclear to me what you are asking for. Do you want an option that follows the XML spec for normalizing values by replacing all whitespace with a space (0x20) character or do you want a custom option that replaces newline, return, and tab with #xD, #xA, and #x9, do you want all whitespace replaced with <?

[fsateler]

Yes, sorry, that was a mistake on my part. Newline is decimal 10, equivalent to hex A. 
 is equivalent to 
 .

What I want is specifically that documents generated by Oj can be parsed by other XML parsers. This means replacing all non-space whitespaces with their equivalent xml escape.

[ohler55]

Okay, so you are requesting output that does not follow the spec. I suppose I could add some output format options. Maybe support no-change-to-whitespace, collapse-whitespace, and escape-whitespace. Does that sound about right?

[fsateler]

I'm confused. I'm arguing that Oj is right now creating non-compliant documents. Or at least, creating documents that will be mis parsed by compliant documents.

The spec is quite clear that upon parsing, non-space whitespace is replaced with spaces. Therefore, outputting a non-space whitespace will result on something different being parsed back.

[ohler55]

Yes, but you asked for whitespace other than 0x20 to be replaced with ordinal values in the output instead of collapsing all whitespace characters to a single space which is not what the spec calls for. I'm okay with offering both options but by following the specification on output result will be different than the input and it will not be possible to recreate the original since the spec indicates the content should be changed.