Add new experimental package ModularGroup for working with finite-index subgroups of SL_2(Z)

[Sebas777-gif]

These are some first basic functions for an OSCAR version of the ModularGroup GAP package. Some of these functions are used in the experimental Origami module, for which there is also an ongoing pull request #5345. The long-term goal is to replace most currently used GAP wrappers with native re-implementations.

Since Origami depends on ModularGroup, but not the other way around, I created this separate branch for ModularGroup. There is still a lot to do, especially writing tests and documentation, but some first feedback from @fingolfin (or others) would be most appreciated. Thanks!

[lgoettgens]

It seems I forgot to submit my comments. Sorry for the delay

[lgoettgens]

IMO, a bunch of these names do not sound like they should be available to all OSCAR users. But as I am not used to the mathematics involved and its conventions, I leave that for @fingolfin to comment on.

[Sebas777-gif]

I talked to @wsinuds and she said that at least in the GAP version of the package, all these functions were indeed meant to be available for end users. Maybe there are some different standards for this question in OSCAR though?

[ThomasBreuer]

Looks good.
I have added a few comments.

One more thought:
In the membership tests, the code computes the permutation induced by the given word, and then checks whether this permutation fixes 1. Wouldn't it be cheaper to map 1 by the sequence of the generating permutations given by the word, and to check whether the result is 1? This would avoid computing many products of permutations.

[ThomasBreuer]

In each call of this function, the objects MatS, MatT, M, and phi are created anew. They are needed only once. Instead of caching the domain of phi, one could simply cache phi.

Is it really intended to return matrices not matrix group elements?
(The conceptual question behind that is what a ModularGroup object actually is:
Is it a group, and if yes, is it a matrix group?
The stored data are a finitely presented group and an isomorphism from this group to the matrix group SL(2, Z) and an epimorphism $\pi$ to a permutation group, all defined on generators for which the defining relations hold; the ModularGroup object describes the subgroup of SL(2,Z) (or of the finitely presented group) that is given by the preimage of the stabilizer of 1 under $\pi$.)

[Sebas777-gif]

Yes, that would probably be better. However, I am wondering if it is advisable to still cache the domain independently. I.e., keep the function _SL2Z_fp(), and then write

const _MATRIX_HOM_CACHE = Ref{GAPGroupHomomorphism{FPGroup, MatGroup{ZZRingElem, ZZMatrix}}}()

function _matrix_hom()
  if !isassigned(_MATRIX_HOM_CACHE)
    MatS = matrix(ZZ, [0 -1; 1 0])
    MatT = matrix(ZZ, [1  1; 0 1])

    M = matrix_group([MatS, MatT])
    MS, MT = gens(M)

    SL2Z, _, _ = _SL2Z_fp()
    _MATRIX_HOM_CACHE[] = hom(SL2Z, M, [MS, MT])
  end

  return _MATRIX_HOM_CACHE[]::GAPGroupHomomorphism{FPGroup, MatGroup{ZZRingElem, ZZMatrix}}
end

For one, because SL2Z, S and T are needed so often that it is slightly more convenient (and more readable) to write _SL2Z_fp() instead of gens(domain(_matrix_hom())). And also for the following reason: _coset_action_hom(G) also has SL2Z as its domain, and we might want to ensure that they really use the same object as their domain?

[fingolfin]

So a ModularGroup is not an Oscar.Group ? I assume that is intentional as it does not implement the Group interface. But perhaps it should?

It really isn't so clear to me what instances of this type are meant to be. OTOH they are matrix groups (by the description above), but OTOH they do not seem to contain matrices... huh

[Sebas777-gif]

No, not implementing the Group interface was not intentional; just an embarrassing oversight. I changed this in 3edbfa5. I am, however, entirely unsure if I did it correctly. I'd appreciate some feedback, thanks!

[Sebas777-gif]

For completeness, I wanted to mention that parts of the code were generated by ChatGPT 5.6 Sol and Grok 4.5 and manually checked by me.