Insertion method for em spaces

[ag-eitilt]

I'd be using five if you provided an entry method for either U+2002 or U+2003; as it is I just fall back on double-spacing the ends of sentences

Feel free to add a separate feature request for that. I can certainly add it, but I don't think there are any sensible keyboard shortcuts left. However, optionally auto-replacing double space with em space could be an option. Possibly a setting "Replace double space with" and the options "Ignore", "En Space" and "Em Space"?

Honestly wasn't even thinking about that when I mentioned it in #2534. The only time I use the Unicode character is if I'm writing HTML or (less consistently) Markdown in Vim, so two space characters is very common for me to look at.

I definitely agree that you've got a pretty full shortcut space for the spaces, so I think responding to <Space><Space> in the same manner as the dashes would probably be best; it's certainly easiest on the memory. It's not a method you can realistically use in Markdown unless you are very strict with your use of tab characters for indentation, but since novelWriter doesn't use Markdown nor source-file indents, it is easily viable here.

As for which character to replace them with, I'm not entirely sure we need to present multiple options rather than just a checkbox, but I also have a feeling the few people who come to novelWriter already using wide Unicode spaces have very settled ideas of which character they prefer, so if it's not too much additional trouble in the code, the dropdown selection is probably the best way to make everyone happy.

[peter88213]

@ag-eitilt : I don't want to say anything against advanced input options, but isn't this topic more about typesetting than manuscript writing?
I take this issue seriously, but I prefer to use a postprocessor or appropriate macros in LibreOffice/OpenOffice (1). This helps me to maintain my writing flow.
What is your opinion? Are your rules for using different spaces suitable for automation?

(1) For my own use, I have written OpenOffice extensions as well as a converter for the DTP software QuarkXpress. This also performs other typographical tasks, such as reducing the size of consecutive capital letters in acronyms by half a point, or using medieval numerals.

[vkbo]

You may have a point here that this is a typesetting matter, and not something to consider while writing. My intention was mainly to support wider spaces between sentences, but avoid any automatic detection of these and leave it to the writer to determine when they should be used.

When writing in mono space it is still quite common, at least in the US it seems, to double-space. I find the practice highly distracting when reading myself, but that's beside the point. It is also common enough to generate the manuscript in mono space font. However, I'm not sure either of these cases are suitable for em spaces instead of simply having two regular spaces.

I tried to do a search to see if there are any typesetting use cases for en and em space in modern text at all. I couldn't really find anything. The thin space has a use case in French, and figure space has a clear use case in numbers, but both these are already supported.

[peter88213]

Just for the record: There is an extension for OpenOffice that is perfect for such use cases: Pepito Cleaner (an undocumented fix for LibreOffice is here). It uses a set of regular expressions for the typographical post-processing of text documents. You can edit these regular expressions and add your own.
The original appears to be designed for Italian and American English. I have developed a separate configuration for German and provided an additional extension that installs this "language package".

[ag-eitilt]

I don't want to say anything against advanced input options, but isn't this topic more about typesetting than manuscript writing?

To one degree, probably, but as Veronica said there's a long history of hitting the space bar twice after ending a sentence, and with Unicode being a thing it feels more idiomatic to represent that as a single character rather than attaching special meaning to doubled occurrences of a separate character. Even so, I still don't see this as a particularly important request, and it's mainly just by observation that there are already a lot of Unicode spaces being provided by novelWriter shortcuts, so this would round out the set. The alternative of just using the two spaces unmodified is perfectly reasonable

By another degree, though, it can be thought of as a manuscript-level concern -- so far as I can figure, the only reasons people consider the space between words and the space between sentences the same are because we now input them with the same key, and because they're both blank gaps that happen to often be rendered the same width. As a highly-contrived example: "Go see that Dr. Smith is here." It can tenuously be read with "that" being demonstrative and "Smith" being a third person, or with "that" meaning "whether" and "Dr. Smith" being a single name. (The better linguistic solution would probably have been to have separate abbreviation and sentence periods, but that's not the level of analysis that Unicode is built on.)

---

I prefer to use a postprocessor or appropriate macros in LibreOffice/OpenOffice (1). This helps me to maintain my writing flow.

Something which I forgot to copy over from my original post:

There's definitely an argument that that's overkill during the manuscript, but it does allow me to write scripts which specifically target a specific division -- for example "split the document into individual sentences" -- which I can then use to analyze the style of the piece while editing.

It's impossible to be fully accurate in programmatically distinguishing sentences when everything is just using the same single space, without either a lot of heuristics (might be possible in LibreOffice, very heavy-weight in Vim) or bringing in an LLM (which has its own problems with "fully accurate") -- throwing enough regexes at it like that extension does might get close, but there's always going to be ambiguity. That's not an argument for Unicode in particular, just for some spacing distinction being helpful.

Most of the other spaces I use do wind up being automatable, but I've also got them set up on two-key macros so they're not actually disruptive to insert (em space is on <Mod>+<Enter>), and novelWriter's <Ctrl>+K sequences are a similar low overhead; it winds up feeling like less effort to put a thin space in manually than to check all the exceptions in the script (what if I transcribe a character texting "???").

---

I tried to do a search to see if there are any typesetting use cases for en and em space in modern text at all. I couldn't really find anything.

This is definitely the biggest problem facing the feature request. There really isn't any modern typesetting which uses broader sentence spacing to begin with, and even if it were I wouldn't be surprised if it were marked up using XML or some other explicit tags rather than relying on Unicode. The en/em spaces do stem from typesetting concerns, and that shows in how they aren't expanded for justified lines. I only started using the Unicode characters because I was writing blog posts (the site's since died) and highly cross-referenced story notes/hyperfiction in HTML, which "helpfully" collapses doubled spaces into a single width.¹

So long as the editor continues to render two spaces as broader than a single one, issues with monospaced fonts aside, I'm perfectly happy to stick with that.

Footnotes

1. To be fair, there is a legitimate technical reason for HTML doing so. I just get a bit salty with the number of guides that try to sell sentence-space "fixing" as a user-experience feature, when it's just an accident of other parts of the system. ↩

[peter88213]

Okay, understood. So the use case is to insert sentence delimiters for text analysis.
Then it's clear to me that double spaces are a potential problem because, as you mentioned, they disappear in HTML, which also applies to (regular) Markdown, which was originally designed for HTML generation.

So, at the very least, you would need to be certain that novelWriter does not automatically “clean up” double spaces, either when loading and saving its own format or when exporting to ODT format.
Incidentally, it should be noted that, as an XML-based file format, ODT doesn't support multiple spaces either.
So if OpenOffice needs to store consecutive spaces, it uses special "spacer" tags internally: <text:s/>. Off the top of my head, I can't say whether the novelWriter ODT exporter uses them.

This naturally raises the question of where the interface of your self-created text analysis lies for you.

[vkbo]

So, at the very least, you would need to be certain that novelWriter does not automatically “clean up” double spaces, either when loading and saving its own format or when exporting to ODT format. Incidentally, it should be noted that, as an XML-based file format, ODT doesn't support multiple spaces either. So if OpenOffice needs to store consecutive spaces (which is, by the way, quite difficult to enter), it uses special "spacer" tags internally: <text:s/>. Off the top of my head, I can't say whether the novelWriter ODT exporter uses them.

I can clarify that. No spaces are stripped inside of text paragraphs. In some contexts leading or trailing spaces are removed though. Like in the context of markup. novelWriter also generates <text:s/> tags for the second consecutive space in ODT files, and with a count value for more than two repeated spaces. Same as LibreOffice does. So multiple spaces are supported.

novelWriter/novelwriter/formats/toodt.py

Lines 1505 to 1520 in 98cdd4e

	elif nSpaces > 2:
	if self._nState in (X_ROOT_TEXT, X_ROOT_TAIL):
	self._xTail = ET.SubElement(self._xRoot, TAG_SPC, attrib={
	TAG_NSPC: str(nSpaces - 1)
	})
	self._xTail.tail = ""
	self._nState = X_ROOT_TAIL
	self._chrPos += nSpaces - 1
	elif self._nState in (X_SPAN_TEXT, X_SPAN_SING):
	self._xSing = ET.SubElement(self._xTail, TAG_SPC, attrib={
	TAG_NSPC: str(nSpaces - 1)
	})
	self._xSing.tail = ""
	self._nState = X_SPAN_SING
	self._chrPos += nSpaces - 1

[vkbo]

It's impossible to be fully accurate in programmatically distinguishing sentences when everything is just using the same single space, without either a lot of heuristics (might be possible in LibreOffice, very heavy-weight in Vim) or bringing in an LLM (which has its own problems with "fully accurate") -- throwing enough regexes at it like that extension does might get close, but there's always going to be ambiguity. That's not an argument for Unicode in particular, just for some spacing distinction being helpful.

This is precisely why I don't want to try to auto-detect these things. A naive detection of detecting .!? is clearly not enough as . can also be used in the context if an abbreviation, and : can sometimes be followed by a full sentence. Then there are all the punctuation rules related to quote marks. Letting the writer indicate the sentence separator offloads all that.

That said, there doesn't seem to be a standard here that can be used as an argument for having this feature other than the typewriter double-space. I'm not sure if there are any manuscript guidelines that demand it. But that would be a definite use case within the scope of novelWriter.

Allowing double-spaces to be replaced by a Unicode space is consistent with how dashes, ellipsis, and quotes are handled, so I am fine in principle with adding it as a non-default option. However, auto-replace comes with a small computational cost even when disabled, so if there is no standard to connect it to, and it's just a matter of personal preference for a small group of users, I'm a little reluctant to add it.

[peter88213]

I can clarify that. [...]

Then everything is clear. Perhaps this could be mentioned in the specification?

In any case, it is possible to identify the sentences with the help of LibreOffice macros. The built-in macro language StarBasic has convenient functions for text cursor control, including jumping to the beginning and end of a sentence. Ambiguities, such as in the example Go see that Dr. Smith is here., could be handled via the additional criterion of double spaces.

[vkbo]

Then everything is clear. Perhaps this could be mentioned in the specification?

Which specification are you thinking of? There is one doc that covers the project XML file format, but that's all. I considered preserving multiple spaces in the manuscript text as default and expected behaviour, so I didn't consider it to be something that needed mentioning. You and I are only aware of this issue because we have the (mis)fortune of knowing the Open Document XML spec. 😅

[peter88213]

Allowing double-spaces to be replaced by a Unicode space is consistent with how dashes, ellipsis, and quotes are handled, so I am fine in principle with adding it as a non-default option.

That immediately came to mind as well. However, my many years of experience as a proofreader tell me that double spaces are among the most common typos. That is why they are blocked by default in some word processors such as LibreOffice Writer.

[peter88213]

Which specification are you thinking of?

You have such a nice and comprehensive manual, don't you?

[vkbo]

That immediately came to mind as well. However, my many years of experience as a proofreader tell me that double spaces are among the most common typos. That is why they are blocked by default in some word processors such as LibreOffice Writer.

I opted to add an (optional) error underline on repeated space instead, because personally I want to avoid them. It may be a good idea to add an option to filter them out in manuscript builds actually.

You have such a nice and comprehensive manual, don't you?

Ah, sure. But that's a level of detail I think is too much for a user guide. It is more appropriate for a format spec, which I only have one of as mentioned.

[peter88213]

But that's a level of detail I think is too much for a user guide.

Why not just tell the average users that they can string together as many spaces as they want without losing anything?
Then they can happily indent paragraphs, format poems, or construct tables ;-)