Feature request: Case-insensitive matching for PEG

[oofoe]

It would be nice to conveniently do case-insensitive matching in a PEG.

One use for PEGs is defining simple command languages. Mixed-case support is often expected in such a language to simplify user interaction. However it's not enough to just force everything to lower (or upper) case before PEG matching because there may be items where case must be preserved (e.g. strings or external function references). Unfortunately, it's rather arduous to set this up:

(def ugly ~{
   :c (set "Cc")  :a (set "Aa")  :t (set "Tt")
  :command (* :c :a :t)
  :main (* (cmt (<- :command) ,string/ascii-upper))})
> (peg/match ugly "CaT")
@["CAT"]                         

Obviously, this is untenable for anything much more complex. However, @veqq came up with a nice way to automate the composition of a case insensitive matcher and I improved on it a bit to handle glyphs that don't need case matching:

(defn ci [x]
    ['* ;(map (fn [b]
               (cond (and (< 64 b) (< b 91))  ['set (string/format "%c%c" b (+ b 32))]
                     (and (< 96 b) (< b 123)) ['set (string/format "%c%c" (- b 32) b)]
                     (string/format "%c" b))) x)])
> (ci "Cat!")
(* (set "Cc") (set "Aa") (set "Tt") "!")

This produces a structure ready to plug into a grammar:

(def sweet ~{
   :command ,(ci "cat")
  :main (* (cmt (<- :command) ,string/ascii-upper))})
> (peg/match sweet "CaT")
@["CAT"]

Currently, I have to unquote the call to ci to use it, but it would be handy if it was built-in to the PEG directives. If possible, the caret ("^") might be a good shorthand notation, i.e. (^ "cat").

Thanks!

[oofoe]

Zulip chat with @veqq about getting this to work -- has a more involved example: https://janet.zulipchat.com/#narrow/channel/409517-help/topic/.E2.9C.94.20State.20of.20the.20art.20for.20case-insensitive.20PEGs.3F/with/599611477

[sogaiu]

For reference, please see the Zulip discussion starting at this message.

[sogaiu]

To summarize a bit of the Zulip discussion, AFAIU upper and lowercase distinctions exist in writing systems other than those representable in ASCII.

Janet's PEGs handle individual bytes and thus this kind of feature seems out-of-place to me for it to be built-in.

---

On a side note, in some writing systems, there are two forms of a lowercase letter (e.g. sigma in Greek [1]) for a single capital letter, so the "calculation" is not just a simple addition or subtraction by some constant amount. I suspect it's not unusual to do table lookups.

---

[1] IIUC, "ς" in word-final position and "σ" elsewhere.

[oofoe]

I am not going to kick to hard on this, since I have a working solution (even if it doesn't work for Greek (sorry guys!)). However, some languages (apparently Python) manage to handle this by using unicode cases.

Is there a way I could modify this feature request that would be more generally usable?

Or, could I ask for consideration of (say) an asciicase tag instead of the more generic ci? (asciicase is just a placeholder for something more clever, but obviously for use with standard ASCII.)

Thanks!

[sogaiu]

The commentary below is just on the "other folks do Unicode" portion of the comment, not touching on the other two points about:

• modify this feature request that would be more generally usable?

• consideration of (say) an asciicase tag instead of the more generic ci?

for further discussion. May be other folks have things to say about them :)

---

However, some languages (apparently Python) manage to handle this by using unicode cases.

Python has pretty extensive support for Unicode AFAIU, in Janet we have "8-bit clean strings":

Strings can be used similarly to keywords, but their primary usage is for defining either text or arbitrary sequences of bytes. Strings (and symbols) in Janet are what is sometimes known as "8-bit clean"; they can hold any number of bytes, and are completely unaware of things like character encodings. This is completely compatible with ASCII and UTF-8, two of the most common character encodings. By being encoding-agnostic, Janet strings can be simple, fast, and useful for other uses besides holding text.

IMO, Unicode's reality is pretty problematic -- it is a large number of specifications which did not account for security early on, and it has resulted in problems such as this [1]. Due to the large number of standards, I think it's rather difficult for that many (if any) individuals to adequately comprehend the details as to be able to reason appropriately. That makes for quite an unfortunate situation. It's also a set of standards that's expected to be ever-changing (and this has historically been true).

Janet also has a (pretty much) policy of not having external dependencies (a relevant point here being that supporting Unicode almost inevitably involves pulling in some external implementation that is expected to change over time and I think core support would likely lead to size increase due to at least inclusion of various tables).

I suspect that programming languages that have "Unicode support" do a combination of:

• having multiple maintainers
• depend on external libraries

AFAIU, these conditions do not match those of Janet's current stance.

---

[1] IMO, it's not a good sign that this older official material is not only not reachable via the original link, but all you get when you go there is:

Not Found
The requested URL was not found on this server.