feat: allow blank lines and comments in dictionary.dict

[hippietrail]

Description

This version would support comments after any whitespace following a dictionary entry with its affix annotation on the same line.

If we do need to support words with spaces then I'll redesign this to require a comment delimiter.

# is the comment delimiter. It is not currently used as an affix annotation flag and perhaps shouldn't be. The current logic will break if becomes one.

How Has This Been Tested?

Fails on lints_lots_of_latin_correctly due to the dictionary containing et al. around line 49,839 as the sole dictionary entry containing a space.

Is that actually intentional and supported?

I've added new tests to cover:

• blank lines
• full-line comments
• line-end comments after entries with and without affix annotations

Checklist

• I have performed a self-review of my own code
• I have added tests to cover my changes

[ficcdaf]

Personally, I think a comment delimiter would be a good idea regardless. Results in much less ambiguity. For example, if someone opens the dictionary and sees #, they'll immediately know it's a comment. And this does leave support for words with spaces in them -- while rare, for example in case of et al. it makes sense.

[hippietrail]

Personally, I think a comment delimiter would be a good idea regardless. Results in much less ambiguity. For example, if someone opens the dictionary and sees #, they'll immediately know it's a comment. And this does leave support for words with spaces in them -- while rare, for example in case of et al. it makes sense.

Yes I'm seeking clarification on the issue of terms with spaces. In tests I found I can add them arbitrarily but I can't get them suggested unless I use the term with the spaces removed, and optional some other characters removed, but not with the space in the wrong place, etc.

There are tons of terms to add if this is going to be a thing.

Only problem with the delimiter is it can also be ambiguous in that it can look like a an annotation flag. # isn't currently used but we're starting to run out and a lot is missing.

I've also started a syntax highlighter so that would also make the comments stand out. The idea is there would be a bunch of whitespace, not just one, at least locally aligned.

Which is not to say I'm closed to the idea of a delimiter (-:

[elijah-potter]

Use # as a comment delimiter. I don't think whitespace after the / should hold meaning in the context you've laid out here.

To clarify the "latin" issue. We previously had issues (solved in #473) with words like et al. which I saw as more "words" than as a higher level phrase. In order for other "words" to be properly recognized like this, they have to be added to the dictionary and an exception has to be inserted to the Document parse stage.

Personally, I see this as out-of-scope for this PR, but we can still discuss it here.

[hippietrail]

Use # as a comment delimiter. I don't think whitespace after the / should hold meaning in the context you've laid out here.

I'll implement # today. As a bonus we'll be able to have full-line comments to divide sections. Mostly "normal words" vs "very domain-specific coding words"

To clarify the "latin" issue. We previously had issues (solved in #473) with words like et al. which I saw as more "words" than as a higher level phrase. In order for other "words" to be properly recognized like this, they have to be added to the dictionary and an exception has to be inserted to the Document parse stage.

In lexicography to avoid ambiguity with the word "word" the terms "lexeme" and "listeme" are used to cover normal words and terms like these.

Personally, I see this as out-of-scope for this PR, but we can still discuss it here.

Me too. But I'll now start gathering Latin and other terms like this to put in an issue somewhere.

I added the first term to a new list in issue 823 here

[elijah-potter]

I made once_cell a dev-dependency since we only use it in tests. Otherwise, this looks good to me!