Merging of Zng definitons, Type Templates, and Conditional Definitions

[Ryex]

Overview

There are three features I feel would be really nice to have

• The ability to split definitions across multiple files for organizational purposes
• The ability to define a Template definition for generic types
• The ability to specify definitions and members there of conditionally

I group these into one because depending on their implementation they can have significant overlap.

I had at least a preliminary plan for the implementation of each

• Merging

  • Change the zngur_def::ZngurFile::types member form a Vec to an IndexMap that maps a RustType to its ZngurType
  • change add_to_zngur_file to check if it's type is present already and if it is merge
    • Confirm matching layout
    • Append non overlapping fields and Well Known Traits
    • Append new member functions
    • Error on conflict
  • Allow specifying multiple input files to parse, and are considered to be concatenated together

• Template Types

  • syntax similar to

    type<T> ::std::option::Option<T> {
        #layout(size = {T.size}, align = {T.align})
    }
    
    type<T,E> ::std::result::Result<T, E> {
        #layout(size = {max(T.size, E.size)}, align = {max(T.align, E.align)})
    }
    
    type<T> crate::SomeWrapper<T> {
        #layout(size = {T.size + 12}, align = 8)
    }

  • Templates are expanded last, after aliases, ONLY if a field or member function references a type that is not already defined
  • Alternatively, Templates are expanded for every use but are merged with existing definitions deferring to the explicit definitions on conflict

• Conditional definition

  • Use familiar #[cfg(target = "x86_64")] syntax
  • Expanded using Cargo ENV similar to cxx
  • evaluate after parse to remove non matching definitions

Questions

I'd like some input on if these features fit within Zugur and if so what they should look like.

• Syntax
  • I could also see a template syntax like:

    template <T> ::std::option::Option<T> {
       #layout(size = {T.size}, align = {T.align})
    }

  • Conditionals could also use a more Preprocessor like #ifcfg /* ...*/ #endif (not sure I'd prefer that though
• Operations allowed when evaluating template size (min, max +, - ? * ? / ?)
• CLI support for conditional expansion

[hkalbasi]

I would like to have all these features in Zngur! I have some opinion about the details:

Merging

There is also another issue about this (#28). We can keep discussion there.

Template Types

While I liked the #layout(size = {T.size}, align = {T.align}) syntax, I guess it wouldn't work in many cases. Either the value of size and align is constant relative to T (where T is used behind some pointer, like Box<T> and Vec<T>), or it has complex relations to T. For example, size of Option<i32> is 8, size of Option<i64> is 16, and size of Option<&i32> is again 8, so it is not possible to describe its size using T.size and arithmetic operators.

So what I would suggest is something like a preprocessor match:

type<T> ::std::option::Option<T> {
    #match T {
        i32 => #layout(size = 8, align = 4)
        i64=> #layout(size = 16, align = 8)
        &_ => #layout(size = 8, align = 8)
    }
    fn unwrap(self) -> T;
    fn as_ref(&self) -> Option<&T>;

    #match T {
        Foo => fn some_method_special_for_option_foo(&self) -> Foo;
        _ => {},
    }

    #if T == i32 {
          fn some_method_for_i32(self);
     }
}

Then we can add more convenient features on top of that in the future.

Templates are expanded last, after aliases, ONLY if a field or member function references a type that is not already defined

That is nice, but might not play well with merging. In that case, we may want to also consider manual expansion, for example:

type<T in [i32, i64, &i32]> ::std::option::Option<T> {
}

Conditional definition

I think we can expand that preprocessor match and preprocessor if from templates to handle this as well:

type Foo {
#match target_arch {
    "x86_64" => layout(size = 8, align = 8),
    "neon" => layout(size = 8, align = 4),
}
   fn some_method(self);
}

What do you think?

[Ryex]

Merging

There is also another issue about this (#28). We can keep discussion there.

Will do.

Template Types

While I liked the #layout(size = {T.size}, align = {T.align}) syntax, I guess it wouldn't work in many cases. Either the value of size and align is constant relative to T (where T is used behind some pointer, like Box<T> and Vec<T>), or it has complex relations to T. For example, size of Option<i32> is 8, size of Option<i64> is 16, and size of Option<&i32> is again 8, so it is not possible to describe its size using T.size and arithmetic operators.

So what I would suggest is something like a preprocessor match:

type<T> ::std::option::Option<T> {
    #match T {
        i32 => #layout(size = 8, align = 4)
        i64=> #layout(size = 16, align = 8)
        &_ => #layout(size = 8, align = 8)
    }
    fn unwrap(self) -> T;
    fn as_ref(&self) -> Option<&T>;

    #match T {
        Foo => fn some_method_special_for_option_foo(&self) -> Foo;
        _ => {},
    }

    #if T == i32 {
          fn some_method_for_i32(self);
     }
}

Then we can add more convenient features on top of that in the future.

This seems workable. The semantics of the arm patterns will need some work.

• consider the cases where there are _, & _, and &mut _ arms. If only a _ arm is present does it also match &T or &mut T ?
• is crate::SomeType<_> a valid pattern?
• is it possible to reference the size or align of T or only to match against it?
• if referencing a type's size/align is possible, is capturing a Type in a pattern possible ?

    type<T, E> ::std::result::Result<T, E> {
         #match T {
             i32 => #layout(size = 8, align = 4)
             i64=> #layout(size = 16, align = 8)
             &_ => #layout(size = 8, align = 8)
         }
         fn unwrap(self) -> T;
         fn as_ref(&self) -> Option<&T>;
  
          #match T {
               ::std::option::Option<_ @ U> => {
                   fn transpose(self) -> ::std::option::Option<::std::result::Result<U, E>>;
               }
               _ => {}
          }
     }

[Ryex]

Templates are expanded last, after aliases, ONLY if a field or member function references a type that is not already defined

That is nice, but might not play well with merging. In that case, we may want to also consider manual expansion, for example:

type<T in [i32, i64, &i32]> ::std::option::Option<T> {
}

This manual expansion syntax seems useful but might not always be convenient to go in and explicitly state all the ones you want to expand.

Since you didn't explicitly comment on it I want to make sure you didn't miss the alternative I suggested.

Alternatively, Templates are expanded for every use but are merged with existing definitions deferring to the explicit definitions on conflict

so you could have

type crate::MyType {
    #layout(size = 8, align = 8)

    fn foo(&self) -> ::std::option::Option<u8>;
}

type ::std::option::Option<u8> {
    #layout(size = 2, align = 1)
    fn copied(&self) -> ::std::option::Option<u8>;
    fn as_ref(&self) -> Option<&u8>;
}

type<T> ::std::option::Option<T> {
    #match T {
           &_ => #layout(size = 8, align = 8)
           _ => {} // expansion on a type not explicitly defined fails because result lacks layout
     }
    fn unwrap(self) -> T;
}

result in

type crate::MyType {
    #layout(size = 8, align = 8)

    fn foo(&self) -> ::std::option::Option<u8>;
}

type ::std::option::Option<u8> {
    #layout(size = 2, align = 1)
    fn copied(&self) -> ::std::option::Option<u8>;
    fn as_ref(&self) -> Option<&u8>;
    fn unwrap(self) -> u8;
}

type ::std::option::Option<&u8> {
    #layout(size = 8, align = 8)
    fn unwrap(self) -> T;
}

[Ryex]

Conditional definition

I think we can expand that preprocessor match and preprocessor if from templates to handle this as well:

type Foo {
#match target_arch {
    "x86_64" => layout(size = 8, align = 8),
    "neon" => layout(size = 8, align = 4),
}
   fn some_method(self);
}

What do you think?

My only realy concern is the lack of syntax clarity on what target_arch refers to (to us it's clear it's the cfg flag/env, but might not be so clear in all cases). I feel it should be knowable at parse time if the statement is trying to match a cfg flag or a template type.

perhaps

type Foo {
#match #cfg(target_arch) {
     "x86_64" => layout(size = 8, align = 8),
     "neon" => layout(size = 8, align = 4), 
}
    fn some_method(self);
}

or similar?

[hkalbasi]

Since you didn't explicitly comment on it I want to make sure you didn't miss the alternative I suggested.

I would prefer automatic detection of template use, but I guess there would be some difficulty about integrating that and the merging feature.

_ => {} // expansion on a type not explicitly defined fails because result lacks layout

That match does not need to be exhaustive, we can raise a compile time error if there were no matching arms. But including that arm should work as you explained, and will cause an error later.

My only realy concern is the lack of syntax clarity on what target_arch refers to (to us it's clear it's the cfg flag/env, but might not be so clear in all cases). I feel it should be knowable at parse time if the statement is trying to match a cfg flag or a template type.

That's reasonable, and I'm fine with #cfg(target_arch) syntax.

[Ryex]

I'm thinking of starting with support for the #match syntax. I'll see what I can whip up

[nickpdemarco]

@Ryex did you get a chance to work on this?

Regarding conditional definitions, the only thing I really need is platform-specific layout directives. We have a few cases (android32, wasm32) where we copy-paste zng files except for their layouts. #match should work great for this.

[Ryex]

Sorry for the slow response @nickpdemarco
been working on "parsing out" a good structure for how the match operator should be implemented

This is what I've got so far

/// Effective trait alias for verbose chumsky Parser Trait
pub trait ZngParser<'a, Item>:
    Parser<'a, ParserInput<'a>, Item, extra::Err<Rich<'a, Token<'a>, Span>>> + Clone
{
}
impl<'a, T, Item> ZngParser<'a, Item> for T where
    T: Parser<'a, ParserInput<'a>, Item, extra::Err<Rich<'a, Token<'a>, Span>>> + Clone
{
}

trait MatchableParser<'src>: Sized {
    type Pattern: MatchPatternParser<'src>;
    fn parser() -> impl ZngParser<'src, Self>;
}

trait MatchPatternParser<'src>: Sized {
    fn parser() -> impl ZngParser<'src, Self>;
}

trait MatchItemParser<'src>: Sized {
    fn parser() -> impl ZngParser<'src, Self>;
}

struct ParsedMatchCfg<'src> {
    pub item: Vec<&'src str>,
}

#[derive(Clone)]
enum ParsedMatchCfgPattern<'src> {
    Empty, // a `_` pattern
    Matches(Vec<Spanned<&'src str>>),
}

enum ParsedMatchEnumBody<'src, Item: MatchItemParser<'src>> {
    Items(Vec<Spanned<Item>>, PhantomData<&'src Item>),
    Empty,
}

struct ParsedMatchArm<'src, Pattern: MatchPatternParser<'src>, Item: MatchItemParser<'src>> {
    pub pattern: Pattern,
    pub body: ParsedMatchEnumBody<'src, Item>,
}

struct ParsedMatch<'src, Scrutinee: MatchableParser<'src>, Item: MatchItemParser<'src>> {
    pub scrutinee: Spanned<Scrutinee>,
    pub arms: Vec<Spanned<ParsedMatchArm<'src, <Scrutinee as MatchableParser<'src>>::Pattern, Item>>>,
}

impl<'src> MatchPatternParser<'src> for ParsedMatchCfgPattern<'src> {
    fn parser() -> impl ZngParser<'src, Self> {
        spanned(select! {
            Token::Str(c) => c
        })
        .separated_by(just(Token::Pipe))
        .collect::<Vec<_>>()
        .map(ParsedMatchCfgPattern::Matches)
        .or(just(Token::Underscore).to(ParsedMatchCfgPattern::Empty))
    }
}

impl<'src> MatchableParser<'src> for ParsedMatchCfg<'src> {
    type Pattern = ParsedMatchCfgPattern<'src>;
    fn parser() -> impl ZngParser<'src, Self> {
        just([Token::Sharp, Token::Ident("cfg")])
            .ignore_then(
                (select! {Token::Ident(c) => c})
                    .separated_by(just(Token::Dot))
                    .collect::<Vec<_>>()
                    .delimited_by(just(Token::ParenOpen), just(Token::ParenClose)),
            )
            .map(|item| ParsedMatchCfg { item })
    }
}


fn match_item<'src, Scrutinee: MatchableParser<'src>, Item: MatchItemParser<'src> + 'src>()
-> impl ZngParser<'src, ParsedMatch<'src, Scrutinee, Item>> {
    let match_arm = Scrutinee::Pattern::parser()
        .then(
            just(Token::ArrowArm).ignore_then(choice((
                spanned(<Item as MatchItemParser>::parser())
                    .repeated()
                    .collect::<Vec<_>>()
                    .delimited_by(just(Token::BraceOpen), just(Token::BraceClose))
                    .map(|items| ParsedMatchEnumBody::Items(items, PhantomData)),
                spanned(<Item as MatchItemParser>::parser())
                    .map(|item| ParsedMatchEnumBody::Items(vec![item], PhantomData)),
                just([Token::BraceOpen, Token::BraceClose]).map(|_t| ParsedMatchEnumBody::Empty),
            ))),
        )
        .map(
            |(pattern, body)| ParsedMatchArm::<<Scrutinee as MatchableParser>::Pattern, Item> { pattern, body },
        );

    just([Token::Sharp, Token::KwMatch])
        .ignore_then(spanned(<Scrutinee as MatchableParser>::parser()))
        .then(
            spanned(match_arm)
                .repeated()
                .collect::<Vec<_>>()
                .delimited_by(just(Token::BraceOpen), just(Token::BraceClose)),
        )
        .map(|(scrutinee, arms)| ParsedMatch { scrutinee, arms })
}

the idea is you would parse with match_item<ParsedMatchCfg, ParsedItem> at the global or module scope and match_item<ParsedMatchCfg, ParsedItem> in the Type scope

and if we add template types at a future date Matchable trait set could be implemented for a ParsedMatchType or equivalent

during the add_to_zngur_spec phase the match arms can be checked and the first one to match will have it's child items added to the spec.