Better ergonomics for lookup types

[cmyr]

so the types we use to represent lookups are annoying; it was one of the trickier parts of the whole scheme. They also haven't been exercised much, since we don't actually have (say) a shaper.

(One of the worst parts of this is the extensions, which have a different type and an additional level of indirection)

In any case, I definitely think there's room to add some methods here that would make this more convenient, and I'm trying to think what exactly would be best given the underlying structure.

I'm going to write up my thoughts here, and then i can convert it into an issue.

Ultimately all we really care about is:

• getting the lookupflag
• getting the markFilteringSet, if present
• getting access to the subtables
• being able unify extension and non-extension lookups (e.g. return the same thing for a pairpos lookup as an extension lookup with pairpos subtables)

Our current structure:

• a generic Lookup<T> type, where T is the type of the subtables
• a pair of generic ExtensionLookup<T> types (one each for GSUB/GPOS) where T is again the type of the subtable
• a pair of enums PositionLookup and SubstitutionLookup with a variant for each lookup type, with each variant wrapping a Lookup<T> where T is the subtable type for that variant.
• another pair of enums for each extension type

Let's imagine we want to simplify this so that, in the general case, the user only needs to think about the top-level PositionLookup and SubstitutionLookup types.

We can have lookupFlag & markFilteringSet fields on this enum.
Then to make accessing the subtables easy, I can current envision two versions of the API:

• we could just have one method for each format, e.g.

impl<'a> PositionLookup<'a> {
    single_pos_subtables(&self) -> Option<ArrayOfOffsets<'a, SinglePos<'a>, Offset16>> { .. }
}

(except this won't exactly work, since extension and non-extensions have a different offset type, which means we don't have a unified return type here; we'd need to introduce an enum to hide those two possibilities)

• or we could have another enum, with a variant for each non-extension lookup, that would look something like,

impl<'a> PositionLookup<'a> {
    fn subtables(&self) -> PositionSubtables<'a> { .. }
}

enum PositionSubtables<'a> {
    SinglePos(SomethingThatActsLikeAnArray<SinglePos<'a>>),
    PairPos(SomethingThatActsLikeAnArray<PairPos<'a>>),
    ...
}

The idea here is to basically change the level of granularity of the generics from the top-level lookup object down to the subtables themselves, at least from the perspective of the API.

How this would relate to the current code is unclear. It would be worth at least considering what this would look like as a systematic change, but it might be more pragmatic to treat it as syntactic sugar, on top of the existing types.

Originally posted by @cmyr in googlefonts/fontc#492 (comment)

[dfrg]

For shaping, we really want to read the lookup/subtable metadata once and cache it for access later when applying features. Whether this is done lazily when lookups are requested or all at once up front would determine whether this cache can be shared across multiple threads without locking. I think we can trivially support both options dynamically but don't hold me to that.

This gist contains the general types for implementing such a cache. In particular, we want a few things from lookup/subtable traversal:

• Absolute offsets to subtables from the base of GSUB or GPOS
• The ability to reconstruct the subtable types directly using the offset and lookup type
• Offset to the "primary" coverage table for each subtable

Upon looking at the code that I pulled the types in the gist from, I realized that this is all already implemented and working based on current read-fonts code and it's not too bad so shaping might just be a case requiring some custom code.

In any case, I still think providing improved ergonomics for the API is beneficial for other cases such as layout-normalizer and autohinting.