Suggestion on hyphenation encoding

[matyaskopp]

Hyphenation over pb, cb, and lb elements:

sample encoding:

<lb xml:id="PMCZ.pb1.cb10.l145"/>článek J. V. Nováka: „O vývoji českého 
<lb xml:id="PMCZ.pb1.cb10.l146"/>jazyka spisovného. Zprávy školské se<pc force="weak">-</pc>
<lb xml:id="PMCZ.pb1.cb10.l147" break="no"/>staveny jsou řiditelem p. Jos. Valentou. — 
<lb xml:id="PMCZ.pb1.cb10.l148"/>Obecné vyšší gymnasium ve Sla<pc force="weak">-</pc>
<pb xml:id="PMCZ.pb2" break="no"/>
<cb xml:id="PMCZ.pb2.cb11" break="no"/>
<lb xml:id="PMCZ.pb2.cb11.l1" break="no"/>ném čítalo 16 učitelských sil. Knihovna profes<pc force="weak">-</pc>
<lb xml:id="PMCZ.pb2.cb11.l2" break="no"/>sorská a žákovská čítaly 3342 čísel. Ústav tento

• adding <pc force="weak">-</pc> simplifies further processing, can be (should be ?) removed for TEI.ana version. (doc: https://www.tei-c.org/release/doc/tei-p5-doc/en/html/ref-pc.html#tei_att.force, sample https://www.tei-c.org/release/doc/tei-p5-doc/en/html/AI.html#AIPC-egXML-im)
• break="no" is documented here https://www.tei-c.org/release/doc/tei-p5-doc/en/html/ref-att.breaking.html and the sample is in lb element documentation https://www.tei-c.org/release/doc/tei-p5-doc/en/html/ref-lb.html#gi-lb-egXML-cz

@TomazErjavec does this encoding make sense to you?

[TomazErjavec]

@matyaskopp, I finally found the time to at least look at this - yes, this encoding makes sense, I can (or you can) modify the encoding guidelines to incorporate your suggestions. If I am to do it, it will have to wait for some weeks as I'm very busy now...

And, yes, my idea was not to have the lb/cp/pb elements in the .ana version, the main reasons are complications with processing, and what then to do with the hyphen indicating the break, as this surely does not belong in the word.

[TomazErjavec]

@matyaskopp, I've now modified and extended the discussion of pb/cb/lb in the guidelines, pls. have a look at https://clarin-eric.github.io/PressMint/#sec-facsimile-ref and the new https://clarin-eric.github.io/PressMint/#sec-dipl.

However, this does not (yet) have your suggestions about @break and pc/@force as I've been thinking about this some more, and I am no longer certain it does make sense. Namely, both these addtitions are only relevant for the .ana version, not the plain text one, as the plain text functions just as well without them. Additionally, when making the plain text version, the processing would now require linguistic knowledge to be able to encode fire<pc force="strong">-</pc>proof vs. recom<pc force="weak">-</pc><lb/>mended.

So, if linguistic processing is required anyway, why do this for the plain-text version (and complicate its processing and encoding), rather than directly in .ana, before tokenisation? If you agree, I would add a discussion about this pc into the linguistic analysis section, with the recommendation option to just delete the hyphen (and pb/cb/lb) and option to add <pc force="weak">-</pc>.
The reason I wouldn't actually recommend adding this is that we would then have e.g. <w>recom<pc force="weak">-</pc><lb/>mended</w> which would complicate processing esp. if you want to get the actual word, as you then need to delete not only tags but also content, i.e. the hyphen.

About @break: indeed, it is meant to indicate (the lack of) a word boundary, however, if we already have pc/@force (either in plain text or ana) then, I think, we have all the information needed for proper tokenisation, so the attribute is redundant and so why complicate with it?

In short, I'd not require @break and would only give pc/@force as an option in the Linguistic analysis Section.
Looking forward to comments!

[matyaskopp]

@matyaskopp, I've now modified and extended the discussion of pb/cb/lb in the guidelines, pls. have a look at https://clarin-eric.github.io/PressMint/#sec-facsimile-ref and the new https://clarin-eric.github.io/PressMint/#sec-dipl.

Doc:

PressMint gives priority to the so-called critical (or structural) transcription of the text, i.e. using, as explained above, elements marking divisions, such as individual newspaper articles, and paragraphs inside them. The alternative view is the diplomatic one, which does not encode structure but rather the visual apperance of the text, i.e. what is a page, column or line.

I agree that the main goal is to have good structural transcription. In my opinion, this includes:

• division into sections...
• correct reading order in the section
• no "presentation type" garbage like hyphenation

(I am aware that not every goal will be fulfilled in all corpora, but this is the ideal condition)

So, for those reasons, I think it is good to have the hyphenation characters marked in the TRI version. In the Czech newspaper, 99% of occurrences of <LETTER>+<HYPHEN or DASH><LOWERCASELETTER>+ use the hyphen, so I can mark it as <pc force="weak">-</pc>.

I prefer to have it also in the TEI version, not only in TEI.ana, or at least don't forbid having it in the TEI version.

About @break: indeed, it is meant to indicate (the lack of) a word boundary, however, if we already have pc/@force (either in plain text or ana) then, I think, we have all the information needed for proper tokenisation, so the attribute is redundant and so why complicate with it?

In short, I'd not require @break and would only give pc/@force as an option in the Linguistic analysis Section. Looking forward to comments!

Yes, agree it is duplicitous, but it is a feature of lb/cb/pb that it is not behaving like a space (word separator), so it is nicer to have it included in the element - no need to look around.

[TomazErjavec]

OK, I now added a discussion of this to the guidelines - it was a bit of a nightmare, as the whole set-up is rather complicated. Anway, @matyaskopp, pls. have a look at the end of Sec 5.2.1. and the new Sec 6.1.1., and let me know if you have any comments or corrections.
In case not, pls. feel free to close this issue.