When making changes to the parser, I am always scared of regressions and properly handling edge cases in the Nix grammar. We want to ensure that rnix-parser parses Nix expressions correctly, but since Nix has no formal specification, the best we can ask in this regard is "does it do the same thing as the reference implementation?" I have built a tool to help us in answering that question: https://github.com/darichey/rnix-parser-tester.
There's a lengthy write-up if you're interested in the details, but the TLDR is you can feed it a Nix expression and it will tell you if rnix-parser and the reference implementation produced the same AST for that expression.
I have used it to verify the correctness of #96, #99, #101, #102, #103, and #106 by parsing all of nixpkgs. The tool provides a way to summarize the comparison results:
$ cargo run -- summary summary/f759fce.json summary/with-patches.json
...
== Summary ==
# equal before: 15924
# not equal before: 8135
# reference impl errors before: 1
# rnix-parser errors before: 1
# equal after: 24060
# not equal after: 0
# reference impl errors after: 1
# rnix-parser errors after: 0
# progressions: 8135
# regressions: 0
This tells us that before applying the patches, 8135 files in nixpkgs were parsed differently by rnix-parser. After applying the patches (and another one I'm still working on), all valid Nix files are parsed equivalently. (There are several caveats to these results which I cover here)
I wanted to make other contributors aware of the tool in case they would find it useful, and also ask if there would be any interest in somehow incorporating it into rnix-parser's testing (probably through CI). Thanks!
When making changes to the parser, I am always scared of regressions and properly handling edge cases in the Nix grammar. We want to ensure that rnix-parser parses Nix expressions correctly, but since Nix has no formal specification, the best we can ask in this regard is "does it do the same thing as the reference implementation?" I have built a tool to help us in answering that question: https://github.com/darichey/rnix-parser-tester.
There's a lengthy write-up if you're interested in the details, but the TLDR is you can feed it a Nix expression and it will tell you if rnix-parser and the reference implementation produced the same AST for that expression.
I have used it to verify the correctness of #96, #99, #101, #102, #103, and #106 by parsing all of nixpkgs. The tool provides a way to summarize the comparison results:
This tells us that before applying the patches, 8135 files in nixpkgs were parsed differently by rnix-parser. After applying the patches (and another one I'm still working on), all valid Nix files are parsed equivalently. (There are several caveats to these results which I cover here)
I wanted to make other contributors aware of the tool in case they would find it useful, and also ask if there would be any interest in somehow incorporating it into rnix-parser's testing (probably through CI). Thanks!