Auto-generate main.zng

[kraktus]

Hello, thanks for this promising project.

I’ve a rust crate https://github.com/kraktus/rs-retroboard-chess that has sparked the interest of cpp users. I’d like to make it available.

The good thing is that it mostly returns simple types or wrapper around it, however the api surface is still quite large to make a main.zng file manually. Is there a way to generate it from the rust code? Even imperfectly, as a way to save time.

[hkalbasi]

I added a very low effort implementation of a main.zng generator, you can use it by generating your project's json docs by cargo +nightly rustdoc -Zunstable-options --output-format=json and copy the result into zngur/zngur-autozng/doc.json and then run autozng using cargo run. I did it for your project, and here is the result:

type crate::MoveKind {
    #heap_allocated;
    fn new(Option, Option) -> Result;
    fn is_uncapture(self) -> bool;
    fn is_en_passant(self) -> bool;
    fn is_unpromotion(self) -> bool;
    fn to_retro_uci(self) -> String;
}
type crate::UnMove {
    #heap_allocated;
    fn from_retro_uci(&str) -> Result;
    fn new(Square, Square, MoveKind) -> Self;
    fn to_retro_uci(self) -> String;
    fn is_uncapture(self) -> bool;
    fn uncapture(self) -> Option;
    fn is_unpromotion(self) -> bool;
    fn is_en_passant(self) -> bool;
    fn uncapture_square(self) -> Option;
    fn mirror(self) -> Self;
}
type crate::RetroPockets {
    #heap_allocated;
    fn color(self, Color) -> &RetroPocket;
    fn color_mut(self, Color) -> &mut RetroPocket;
    fn from_str(&str, &str) -> Result;
}
type crate::RetroBoard {
    #heap_allocated;
    fn new_no_pockets(&str) -> Result;
    fn new(&str, &str, &str) -> Result;
    fn push(self, &UnMove);
    fn pseudo_legal_unmoves(self, &mut UnMoveList);
    fn legal_unmoves(self) -> UnMoveList;
    fn board(self) -> &Board;
    fn retro_turn(self) -> Color;
    fn us(self) -> Bitboard;
    fn our(self, Role) -> Bitboard;
    fn them(self) -> Bitboard;
    fn their(self, Role) -> Bitboard;
    fn king_of(self, Color) -> Square;
    fn flip_vertical(self);
    fn flip_horizontal(self);
    fn flip_diagonal(self);
    fn flip_anti_diagonal(self);
    fn rotate_90(self);
    fn rotate_180(self);
    fn rotate_270(self);
}
type crate::ParseRetroPocketError {
    #heap_allocated;
}
type crate::RetroPocket {
    #heap_allocated;
    fn decr(self, Role);
}
type crate::ParseRetroUciError {
    #heap_allocated;
}

It is broken in multiple ways, and you can use it just as a template. You need to fix these things for it to become usable:

• Names should be full qualified, like crate::UnMove, ::shakmaty::Role or ::std::string::String.
• Generics are ignored and should be added
• Generic types used in function arguments and return types should be added as separate types, with appropriate methods like unwrap.
• self types are wrong (I can fix this easily if it is annoying).
• You probably want to do this once for shakmaty crate as well.

The autozng is a simple 200 line script which you can easily read and improve, and ideally we want to have a full featured autozng and it is in the scope of this project, however I currently don't have time to make it complete and polished, but I can add some more features if you desire.

Please tell me if you hit any problems in using zngur, or have some ideas. My own project that triggered me to create zngur became fully Rust, so now there is no real project using it. But I still believe in zngur idea and would really like to see a real project use zngur, so I will help you if you hit any zngur related problem.

[kraktus]

Thanks! I actually did something similar 2 years ago: https://github.com/kraktus/cargo-extern-fn and got quite far using proc macros, but it was based on cxx which turned out to be too limited in terms of fondamental type support (not even Option), and seemingly no active development from the maintainers to add them or accept PRs.

Will definitely check it out once I've more time.

[camio]

I think it's important to note that an autozng will always be incomplete. It's impossible to know, without reading the C++ code, which of the infinite variations of tuples, for example, are needed on the C++ side.

[hkalbasi]

Yes, there never will be a full automatic autozng. The most complete autozng possible could receive a list of end types, for example Vec<i32>, (Foo, &[Bar]), and others that are directly used in C++, and automatically generate all of their methods and types that are needed for those method (in this example, they are i32, Option<i32>, Foo, Bar, &[Bar], ...). But even this can lead to bloat in the generated code and compile times so I think even more manual control is needed at the end.

[Ryex]

I've been letting my brain simmer on this one so the last few months and I think I've reached an approach I'm willing to implement and PR.

from the beginning there a series of issues to work out

• Any autozng impl would be incomplete

  • Mostly Solved: import statements let you generate code for the public rust side and extend it with cpp side or spesfic generic instances.
  • Generation may make it desirable to not require a layout in every type decl. make it a generation time error for it to be missing.

• layout choice.

  • Proposal: let autozng side load a 'size' databases (.json) generated/updated from -Z print-type-sizes (like the work in AutoZNG implementation using builtin rust tooling #58) or by hand that maps target and opt level configuration to size and alignment for fully qualified concrete types. configurably fallback to heap allocation (or a future conservative overallocation option) or error if it tries to generate a type it can't find. database could even include size and offset for fields.

• Generics: how to we emit the correct concrete types for generics?

  • Proposal: type templates and impl declarations.
    Templates can be expanded for full concrete types that appear in impl declarations and possibly for types that would otherwise be missing. Autozng could output templates for generic types and functions. impl declarations would then be left to the main spec.

• Where to get type info for generation. This could be done two ways

  • rustdoc: Requires nightly and you'd need to put the zngur generated module / code relent on it behind a feature to hide it from rustdoc but you get the full benifits of the compiler including conditional compilation and macro expansion.
  • syn parse module files: not as heavy as a compiler call. can be done without nightly. selective of which files to examine. but no macros or conditional compilation.
  • could support both with seperate commands. extra work.

this work would make code gen a 2-3 step process

• Aquire type size info and choose fallback.

  • Generate or write it and commit it along side the code
  • OR Have a nightly compiler in the compilation env and generate it ever time

• Run autozng to generate any interface zng files. import them into the main spec.

• Run zng cli or build script to generate header and module

[hkalbasi]

Is acquiring type size info in a database before knowing the needed specializations possible? Other than that, I'm happy with a two step autozng -> zng -> code flow.

syn parse module files: not as heavy as a compiler call. can be done without nightly. selective of which files to examine. but no macros or conditional compilation.

I don't think it scales well. We need some compiler integration at the end I guess, either via rustdoc or with a custom rustc_driver. Both options requires nightly, but the rustc_driver approach is more prune to breakage (but it is more powerful).

[Ryex]

Is acquiring type size info in a database before knowing the needed specializations possible?

it's mostly possible. -Z print-type-size will print the size and internal layout of any type it "sees" during the compilation. and sadly, that compilation will still fail due to normal compilation errors so the presence or absence of a generated module would need to be hidden from that step. If there are types that would only appear in the zng interface which need size info (like the parameters of a generic type or function) a macro could be used to write code like.

#[cfg(not(feature = "zngur-type-sizes"))]
mod generated;

#[cfg(feature = "zngur-type-sizes")]
mock_sizes_with!(__zngur_type_sizes, [
    Option<&132>,
    MyInterfaceType,
    SpecialGenericInterfaceType<String>,
]);

which would expand to this on a compile pass for size info

#[allow(dead_code)]
fn __zngur_type_sizes(){
    let _ = |_i: Option<&i32> |{};
    let _ = |_i: MyInterfaceType |{};
    let _ = |_i: SpecialGenericInterfaceType<String> |{};
}

if you don't want a nightly compiler in the final compilation env it would be necessary to either maintain the size database and generated json docs in source control or to pre generate zng files for each target you want to support to be selectively included and keep those in source control.

but just like with autozng as a whole it's effectively impossible for the size info to be completely generated which is why I was still allowing for a human writing a json file by hand and fallback strategies for types it can't find size info for.

Other than that, I'm happy with a two step autozng -> zng -> code flow.

I really think this is the best path. it leaves the door open to a lot of control over how and when the IDL is generated.

syn parse module files: not as heavy as a compiler call. can be done without nightly. selective of which files to examine. but no macros or conditional compilation.

I don't think it scales well. We need some compiler integration at the end I guess, either via rustdoc or with a custom rustc_driver. Both options requires nightly, but the rustc_driver approach is more prune to breakage (but it is more powerful).

that's fair. I was thinking it might be nice in the simpler cases where it was possible to place all the types you want to expose in one or two files that don't use macros or code gen.

[Ryex]

so I've been doing some experiments and I think -Z print-type-sizes is a dead end. it seems to plug into the compilation tracing to get it's output so if that stage is cached for a given codegen unit or otherwise skipped no output is given for the types therein. thus you may get partial or no out put for a compiler invocation on a unclean build.

I briefly considered using a doctest style approach of generating code to print the layout of types we're interested in and linking it to the crate but that limits us to the public api and only works if the host and compilation target are the same. trying to read constants out of a target elf or similar format also has issues.

but then I realized we already generate code to check the layout in the generated module. and the compiler will tell us if we get it wrong.

so here's my new idea. use a cached layout database. have autozng make some guesses based on pointer size for types not in that database and give autozng a command that will invoke the compiler with --error-format=json and parse the resulting assertion errors to update the layout database.

as long as we can write the assertions to allow easily identification of the fully qualified type and the real size / offset this should work fairly reliably.

[hkalbasi]

I think rustc_public is also a promising tool. I'm using it for another project, it has a very clean API which I think addresses almost all needs of autozng, and it is a standard and supported solution (parsing elf files or error messages is hacky and may break in unexpected ways in future versions). One problem is that it ties us to a single version of rustc (other solutions are somehow rustc agnostic), but I think it is a tolerable problem and rustc_public team are working to address this problem long term.

[Ryex]

it's certainly an option from looking into it.

it still has the issue of seeming to require a nightly toolchain to compile at all. I'd prefer there be a way to collect and vendor the data needed to run final autozng step and crate compilation on a stable toolchain. not having that option makes distribution quite difficult

[hkalbasi]

Nightly is bad, but it is not a deal breaker. If we release binaries for autozng, users won't need to install a nightly toolchain I guess. Also, rustc_public is going to be published in the crates.io eventually, but that's a long term plan.

[Ryex]

even if autozng ends up statically linking rustc my main concern remains allowing support for vendoring the data needed to compile from source on stable for an arbitrary target.

for example if a project looking to adopt zngur targets mac, linux, windows, bsd, and some riscv. in order to support packagers it needs to ensure the packager doesn't also need to package autozng in order to compile for a given distro.

hence the desire to cache the layout information for targets, vendor it with the source for development purposes, and generate a vendored zng file that could conditionaly use the size info needed for any given platform, falling back to a conservative layout for unknown platforms. that way a build.rs or stable compiled zng-cli could be used during packaging.

that's also why a stable compiled autofix version of autozng that could correct incorrect layout information by reading the assertion errors from trying to compile a crate with incorrect layout seems like a good idea to me.