Markdown in rdf/xml annotations doesn't render paragraphs or lists

[sensespidey]

Describe the bug

When generating HTML from an RDF/XML ontology file with markdown in rdfs:comment annotations, newlines are removed, resulting in missing paragraph and list tags in the rendered HTML.

To Reproduce
Steps to reproduce the behavior:

1. Add some newlines and * or - list items in Markdown format, in the rdfs:comment text for an owl:Class annotation in an RDF/XML serialized ontology.
2. Render docs from said ontology using something like java -jar bin/widoco-1.4.25-jar-with-dependencies_JDK-11.jar -ontFile ONTOLOGY.owl -outFolder docs -oops -rewriteAll -uniteSections -webVowl
3. Open rendered documentation in your browser
4. Look for <p> and <ul><li> tags in the Class description HTML
5. Observe no paragraph or list tags, newlines are removed from the innerText, and - and * characters are left intact.

Expected behavior

I expected the newlines to persist in the Markdown-formatted rdfs:comment annotations, and result in paragraphs and lists rendered in HTML.

Screenshots
Let me know if it's helpful to see an image of the markup or something, but hopefully my description above is clear :)

Desktop (please complete the following information):

• OS: Linux (Ubuntu 22.04)
• Browser Firefox
• Version 138.0.1

(I don't believe this is particularly relevant, but I'm running openJDK 11 and Widoco 1.4.25)

Additional context

I've poked at this to understand that the marked library is running correctly against these Class descriptions, but the text that's in there is missing newlines that are present in the source file I'm pointing Widoco too. I gather from this that the code that's loading my Ontology in order to produce this HTML is sanitizing or otherwise "cleaning" this text somewhere along the line. I haven't been able to track down where that is, but if I find it (or someone more knowledgeable points me to it) I'll happily take a crack at improving this :)

[sensespidey]

Update: I managed to trace the Widoco code to WidocoUtils::loadModelToDocument(), which in turn calls OWLOntologyManager::loadOntologyFromOntologyDocument to load the ontology from disk. I haven't confirmed this, but it looks like that's the point at which our newlines are lost. I've dug into the OWL API code a little bit, but don't see an obvious way (through the OWLOntologyLoaderConfiguration or similar) to tell it to retain newlines when loading the file. Maybe there's a way to work around this in Widoco, but possibly this bug properly lives upstream in OWL API?

[dgarijo]

This is odd indeed. I certainly don't do any postprocessing to remove newlines. Will have to investigate.

[sensespidey]

Further update: I spent some more time last week reading through OWL API code, as well as researching Java mechanisms for loading files, and I still can't see any good reason for my newlines being "lost". I'm not a java developer, but as far as I can tell, the loadOntologyFromOntologyDocument method I had tracked down earlier ends up using a FileInputStream to read in the file. Based on this stackoverflow answer, it appears one would have to go out of their way to remove newlines from an input stream, whereas other methods (eg. BufferedReader) that read line-by-line are more likely to "lose" newlines.

As such, I'm pretty stumped here, but hoping that sharing this extra bit of info may be helpful in getting to the bottom of it :)

[dgarijo]

Does this also happen with ttl files? Sometimes triple quoting things can force the readers to preserve the format.
So instead of having a literal "like this", you have it """like this"""

[sensespidey]

Does this also happen with ttl files? Sometimes triple quoting things can force the readers to preserve the format. So instead of having a literal "like this", you have it """like this"""

I did try it with all the other formats Protege can export, and it didn't seem to help. I'll double-check with a TTL export and make sure I have triple-quoted things, and report back :)

[dgarijo]

See https://linked.earth/ontology/archive/2.1.0/ontology.ttl:

###  http://linked.earth/ontology/archive#ArchiveType
:ArchiveType rdf:type owl:Class ;
             rdfs:comment """The medium in which the response of a sensor to environmental forcing is recorded. Sensu Evans et al. (2013).

Examples of archives include: marine sediments, corals, wood, lake sediments, speleothems, glacier ice, etc."""@en ;
             rdfs:isDefinedBy <http://www.sciencedirect.com/science/article/pii/S0277379113002011> ;
             rdfs:label "Proxy archive"@en .

And in https://linked.earth/ontology/archive/2.1.0/index-en.html#ArchiveType the spaces between new lines are preserved

[sensespidey]

Fascinating! Thanks for finding a relevant example- I have a very class where the relevant section of the TTL serialization looks like this:

cids:Activity rdf:type owl:Class ; # I'm omitting subclass and class restriction annotations for brevity
              rdfs:comment """Activity defines the actions performed by an organization and/or contributing stakeholder to implement a service. The Common Impact Data Standard’s Activity clas

An activity’s type is based on its outcome rather than service or input. This allows activities to be classified by the type of change they produce rather than by what resources they use (inp

Its properties are
- canProduce: Specifies the Outcome that results from performance of the Activity. It is used primarily for representing Outcome Chains.
- oep:partOf: Identifies the Impact Model it is a component of.
- hasInput: Specifies the Input to the Activity.
- hasOutput: Specifies the Output of the Activity.
- hasCode: Specifies zero or more codes created by various organizations to identify a type of Activity. E.g., ICHI – International Classification of Health Interventions activities.
- act:subActivityOf: Specifies the Service or Activity that this Activity is part of.
- hasName: Specifies a name or title for the activity.
- hasDescription: Specifies a description of the activity.
- hasIdentiifier: Specifies a unique identifier for the activity."""@en ;
              rdfs:label "Activity"@en .

However, when I run this through Widoco to generate docs, the HTML I get looks like so:

  <h3>Activity<sup class="type-c" title="class">c</sup> <span class="backlink"> back to <a href="#toc">ToC</a> or <a href="#classes">Class ToC</a> </span></h3>
  <p><strong>IRI:</strong> https://ontology.commonapproach.org/cids-3.0#Activity</p>
  <div class="comment">
   <span class="markdown">Activity defines the actions performed by an organization and/or contributing stakeholder to implement a service. The Common Impact Data Standard’s Activity class is defined to be a subclass of the TOVE Activity (outlined in Section 3.7), and is extended by including properties for Input, Output, and what Service or Activity it is a subActivityOf. An activity’s type is based on its outcome rather than service or input. This allows activities to be classified by the type of change they produce rather than by what resources they use (input) or who performs the activity (service). Its properties are - canProduce: Specifies the Outcome that results from performance of the Activity. It is used primarily for representing Outcome Chains. - oep:partOf: Identifies the Impact Model it is a component of. - hasInput: Specifies the Input to the Activity. - hasOutput: Specifies the Output of the Activity. - hasCode: Specifies zero or more codes created by various organizations to identify a type of Activity. E.g., ICHI – International Classification of Health Interventions activities. - act:subActivityOf: Specifies the Service or Activity that this Activity is part of. - hasName: Specifies a name or title for the activity. - hasDescription: Specifies a description of the activity. - hasIdentiifier: Specifies a unique identifier for the activity.</span>
  </div>

I'm not sure how I'm ending up with a different result here! The one other thread I found in my research was the suggestion that Windows vs Unix newlines could be at play. This doesn't seem likely to me since my workflow is entirely within a Unix environment, but I just tried converting my OWL files with dos2unix and see no change, which would seem to indicate this is not a CRLF vs. LF problem?

Any other ideas for how to identify the source of the problem here are very welcome! I'd very much like to be able to include Markdown syntax in my OWL files directly :)

[dgarijo]

What if you add breaklines in html? It's not ideal, but it may solve the problem. By adding <br/> instead of a newline. It's a little of a hack, but may work.

[sensespidey]

Ah yes, adding <br /> and <ul> tags seems to work- I think I had tried adding HTML markup like this (also html-encoded newlines like &xA;) before, but not in the TTL format. This doesn't work in RDF/XML, presumably because of the syntax overlap, but may be a reasonable workaround for the time being.

Can you think of any other reason the example you mentioned earlier would behave correctly while mine does not? My only concern with taking the approach of including HTML in the TTL file is that it makes it less readable if you're examining that format directly, rather than reading the HTML docs. This may be a minor point, but it's one of the key selling points of Markdown, so it'd be nice to use it if possible (and clearly it works in some cases!)

Thanks for the ideas and feedback thus far- much appreciated!

[dgarijo]

I have no clue. The example was generated with an older version of Widoco, which may have had an older version of the OWLAPI.
I will have to check the reproducibility of this issue with other vocabularies first.

[vChavezB]

I stumbled upon this issue as well. New lines are not in the html generated by widoco. Its been a while that I have tinkered with Widoco but as far as I know the transformation to HTML is done with the XSLT transformation, so nothing to do with the owlapi. I did as well a short test and the owlapi does not remove the \n character.

[dgarijo]

I looked into this. Apparently <xsl:value-of select="text()"/> concatenates all text removing newlines in the xslt transformation. Instead, we should be using <xsl:output method="text"/> for the description extraction. I will give it a try