Emit separate headers for each sub-crate

[nickpdemarco]

With #43 merged, the likelihood of creating massive generated.h files increases exponentially. We should have support for splitting generated.h into multiple headers.

At first glance, I'd suggest creating a header for each crate that contributed symbols to the ZngurSpec. This is easier said than done, since symbols (particularly from std) can be "re-opened" in different zngur files, and we'd expect each zng file that re-opened the symbol to have its FFI code in the crate's generated header.

I think that implies we need to track which .zng files mention which symbols, and then create a header for each set of headers which mention common symbols. That internal, "shared" header would then be included by each individual header to create the desired effect.

// a.zng
mod ::std {
    type option::Option<i32> {
        #layout(size = 8, align = 4);

        constructor None;
        constructor Some(i32);
    }
}

type Box<dyn Fn(i32) -> i32> {
    #layout(size = 8, align = 4);
}

// b.zng
mod ::std {
    type option::Option<i32> {
        #layout(size = 8, align = 4);
        fn unwrap(self) -> i32;
    }

    mod vec {
        type Vec {...}
    }
}

// internal/generated/ab.h (not exposed to users)
// Insert `option::Option FFI` here

// generated/a.h (exposed to users)
#include <internal/generated/ab.h>
// Insert `Box<dyn Fn(i32) -> i32>` FFI here

// generated/b.h (exposed to users)
#include <internal/generated/ab.h>
// Insert `Vec` FFI here

[hkalbasi]

I think it makes somethings simpler if we break the header based on Rust crates, not zngur files. It solves the reopening problem, but there are still challenges. For example, although crates dependency graph is always acyclic in rust, template specializations that we have can create cycles:

mod ::std {
    type option::Option<::some_crate::SomeType> { // where should this go? std.h or some_crate.h?
        #layout(size = 8, align = 4);
        fn unwrap(self) -> i32;
    }
}

In this example, some_crate depends on std, but if we want to put Option in std, we need to have SomeType and so some_crate, which is circular.

[colin-r-schultz]

What if there was
std.h

template<typename ...T>
struct Option;

and
some_crate.h

#inlcude "std.h"
struct SomeType;
template<> Option<SomeType>;

[colin-r-schultz]

I guess there is the issue of where template<> Tuple<crate1::Type1, crate2::Type2> would be defined. I think the answer is that we need some manual organization of the .zng code which does not have circular dependencies.

[hkalbasi]

This is not specific to Tuple. crate1::Type<crate2::Type, crate3::Type> also has this problem. We can put these types in the dominator of all involved crates, and add an imaginary final crate which depends on all crates, ensuring that there is at least one dominator for every set of crates. The problem with this approach is that you may need to include some random crate in order to use a type, but I think it is fine compared to "creating a header for every set of crates" and "manually specifying headers which should contain the items".

[adsnaider]

Any thoughts about the approach that I proposed in #101 where instead of thinking about one header per Rust crate, we set it up as one header per zngur instantiation but we add an import module directive that will change the behavior of import from copy paste to instead assume it needs to include the corresponding generated header for C++ or the corresponding crate in the rust generated sources. This way you can structure your build system however you want around that. You can have one massive zng file or split things up as much as you want. I also think it would be easier to implement that

[hkalbasi]

Spliting headers based on zngur modules addresses some of the challenges (at least make them the problem of the user) so I'm generally in favor. For the goal of reducing compile times by using multiple headers, it may create some manual work relative to the crate based header separation, but I guess it is not feasible to split headers fully automatically, and using zngur modules for declaring the boundries makes sense.

[adsnaider]

@hkalbasi I'm hoping the manual work required can be hidden behind the build system integration. For cargo workflows the original imports are probably the right abstraction since they're more of a monolith anyway.
I'll try to write up a PR that implements this. I'm also trying to write a new implementation for #67. I think we should decide whether the panic handler and the cfgs should be defined in zngur.h or per crate. My thoughts are that zngur.h represents shared behavior across a linked binary, so the panic handler should be done globally in zngur.h since the panic strategy is generally global per binary. cfgs on the other hand are a crate artifact so they should be defined in each instance of the generated crates.

I'm thinking that the CLI should have an option to generate a zngur.h only so that build systems can generate it once and place it in their codebase. The current approach still generates a zngur.h file per zngur instantiation which I think isn't the right approach here. For context cxx.rs shares a prebuilt cxx.h file that projects can use. I'm fine with generating it in order to parametrize behavior but it should be something along the lines of zngur make-shared-header --per-project-flags instead.