test(spec): walk both presets so mainnet-only cases are emitted by GrapeBaBa · Pull Request #341 · ChainSafe/lodestar-z

GrapeBaBa · 2026-05-03T15:18:45Z

Summary

The EF spec test pack ships asymmetric vector packs: ~97 case dirs only exist under mainnet/tests/mainnet/.../operations/... (e.g. random_*_with_duplicates for sync_aggregate), while hundreds only exist under minimal/tests/minimal/.... The current generator walks just one preset (minimal), so any mainnet-only case is silently dropped at codegen time and never reaches the spec runner.

This PR makes write_spec_tests.zig walk both presets and dedup the union by case key, so every case the EF pack ships gets a generated test entry.

Concrete impact (operations shard, mainnet)

Before	After
1567 / 1734 (167 skip)	1664 / 1831 (167 skip)

+97 newly emitted mainnet-only cases. Skip count is unchanged because those 167 are runner-handled skips, not codegen omissions.

gemini-code-assist · 2026-05-03T15:18:53Z

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request improves the Ethereum Foundation spec test generation process by addressing an issue where asymmetric vector packs caused certain test cases to be omitted. By iterating over both minimal and mainnet presets and implementing a robust deduplication strategy, the generator now captures the full set of available test cases, increasing test coverage for mainnet operations.

Highlights

Preset Coverage: Updated the test generator to walk both 'minimal' and 'mainnet' presets, ensuring that mainnet-only test cases are no longer silently dropped.
Deduplication and Stability: Implemented case key deduplication using a StringHashMap and added sorting to ensure stable, deterministic test output across regenerations.
Error Handling: Refined directory traversal error handling to distinguish between expected missing presets and actual filesystem errors, ensuring failures are reported loudly when appropriate.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

This pull request refactors the test generation logic to support asymmetric vector packs by aggregating and sorting test cases from multiple presets, ensuring consistent output across runs. The review feedback identifies several violations of the repository's style guide, including multiple instances where the 100-column line limit is exceeded, a lack of sufficient assertions in new functions, and the use of abbreviated variable names. Additionally, a redundant assignment in the hash map insertion logic was noted for removal to align with the zero technical debt policy.

gemini-code-assist · 2026-05-03T15:20:26Z

@@ -92,55 +105,116 @@ pub fn writeTestRoot(comptime kinds: []const RunnerKind, writer: *std.Io.Writer)

 pub fn writeTests(


Adhere to the repository style guide (Rule 51) which requires an average of at least two assertions per function. This function currently has none. Consider adding assertions for preconditions, such as std.debug.assert(forks.len > 0);.

References

Assert all function arguments and return values, pre/postconditions and invariants. The assertion density of the code must average a minimum of two assertions per function. ^(link)

gemini-code-assist · 2026-05-03T15:20:26Z

+    @setEvalBranchQuota(8000);
    try TestWriter(kind).writeHeader(writer);

    var root_dir = try std.Io.Dir.openDir(.cwd(), io, spec_test_options.spec_test_out_dir ++ "/" ++ spec_test_options.spec_test_version, .{});


This line exceeds the 100-column limit specified in the repository style guide (Rule 400).

var root_dir = try std.Io.Dir.openDir( .cwd(), io, spec_test_options.spec_test_out_dir ++ "/" ++ spec_test_options.spec_test_version, .{}, );

References

Hard limit all line lengths, without exception, to at most 100 columns. ^(link)

gemini-code-assist · 2026-05-03T15:20:26Z

+            }
+
+            for (presets) |preset| {
+                try collectCases(io, allocator, root_dir, preset, fork_path, comptime handler.suiteName(), comptime kind.hasSuiteCase(), &cases);


This line exceeds the 100-column limit (Rule 400).

try collectCases( io, allocator, root_dir, preset, fork_path, comptime handler.suiteName(), comptime kind.hasSuiteCase(), &cases, );

References

Hard limit all line lengths, without exception, to at most 100 columns. ^(link)

gemini-code-assist · 2026-05-03T15:20:26Z

+            var it = cases.keyIterator();
+            while (it.next()) |k| ordered.appendAssumeCapacity(k.*);


Avoid abbreviations in variable names as per Rule 221 of the repository style guide. Use iterator instead of it and key instead of k.

var iterator = cases.keyIterator(); while (iterator.next()) |key| ordered.appendAssumeCapacity(key.*);

References

Do not abbreviate variable names, unless the variable is a primitive integer type used as an argument to a sort function or matrix calculation. ^(link)

gemini-code-assist · 2026-05-03T15:20:26Z

+            for (ordered.items) |key| {
+                if (comptime kind.hasSuiteCase()) {
+                    const slash = std.mem.indexOfScalar(u8, key, '/').?;
+                    try TestWriter(kind).writeTest(writer, fork, handler, key[0..slash], key[slash + 1 ..]);


This line exceeds the 100-column limit (Rule 400).

try TestWriter(kind).writeTest( writer, fork, handler, key[0..slash], key[slash + 1 ..], );

References

Hard limit all line lengths, without exception, to at most 100 columns. ^(link)

gemini-code-assist · 2026-05-03T15:20:26Z

    }
 }
+
+fn collectCases(


This function lacks assertions, violating Rule 51 of the repository style guide. Consider asserting that preset.len > 0 and fork_path.len > 0.

References

Assert all function arguments and return values, pre/postconditions and invariants. The assertion density of the code must average a minimum of two assertions per function. ^(link)

gemini-code-assist · 2026-05-03T15:20:26Z

+            var suite_dir = handler_dir.openDir(io, entry.name, .{ .iterate = true }) catch |err| switch (err) {
+                error.FileNotFound => continue,
+                else => |e| return e,
+            };


This block exceeds the 100-column limit (Rule 400).

var suite_dir = handler_dir.openDir( io, entry.name, .{ .iterate = true }, ) catch |err| switch (err) { error.FileNotFound => continue, else => |e| return e, };

References

Hard limit all line lengths, without exception, to at most 100 columns. ^(link)

gemini-code-assist · 2026-05-03T15:20:26Z

+            var case_iter = suite_dir.iterate();
+            while (try case_iter.next(io)) |case_entry| {
+                if (case_entry.kind != .directory) continue;
+                const key = try std.fmt.allocPrint(allocator, "{s}/{s}", .{ entry.name, case_entry.name });


This line exceeds the 100-column limit (Rule 400).

const key = try std.fmt.allocPrint( allocator, "{s}/{s}", .{ entry.name, case_entry.name }, );

References

Hard limit all line lengths, without exception, to at most 100 columns. ^(link)

gemini-code-assist · 2026-05-03T15:20:26Z

+    }
+}
+
+fn insertOrFree(allocator: std.mem.Allocator, set: *std.StringHashMapUnmanaged(void), owned_key: []const u8) !void {


The function signature exceeds the 100-column limit (Rule 400).

fn insertOrFree( allocator: std.mem.Allocator, set: *std.StringHashMapUnmanaged(void), owned_key: []const u8, ) !void {

References

Hard limit all line lengths, without exception, to at most 100 columns. ^(link)

gemini-code-assist · 2026-05-03T15:20:27Z

+    if (gop.found_existing) {
+        allocator.free(owned_key);
+    } else {
+        gop.key_ptr.* = owned_key;


This assignment is redundant because getOrPut already inserts the provided owned_key into the map if it was not found. Removing redundant code aligns with the "zero technical debt" policy (Rule 19).

References

TigerBeetle has a “zero technical debt” policy. We do it right the first time.

The EF spec test pack ships asymmetric vector packs — for example ~97 case dirs only exist under `mainnet/tests/mainnet/.../operations/...` while hundreds only exist under minimal. The previous generator walked just `minimal/tests/minimal`, so any mainnet-only case was silently skipped at code-gen time and never reached the spec runner. Walk both presets and dedup the union by case key, so every case the EF pack ships gets a generated test. Adds ~97 mainnet operations cases without touching minimal coverage. Errors from `openDir` are split: `error.FileNotFound` is the legitimate "this preset/fork/handler is not present" gap, anything else is a real filesystem problem and propagates so a corrupt checkout fails loudly instead of silently dropping cases.

GrapeBaBa · 2026-05-07T06:30:10Z

@codex review

chatgpt-codex-connector · 2026-05-07T06:31:45Z

Codex Review: Didn't find any major issues. Another round soon, please!

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

…hack Replaces StringHashMapUnmanaged<void> + manual `<suite>/<case>` string concat + indexOfScalar split with: - Struct key { suite, case } — no string concatenation - ArrayListUnmanaged(Key) + sortUnstable + adjacent dedup - Per-handler ArenaAllocator owning all dup'd slices, freed in one arena.deinit() — no manual key-by-key free, no keyIterator/getOrPut invariant to maintain Generated test_case/*.zig files are byte-identical before/after. Spec tests pass under -Dpreset=minimal.

GrapeBaBa · 2026-05-07T07:22:39Z

@claude review

GrapeBaBa · 2026-05-07T07:23:11Z

@codex review

GrapeBaBa · 2026-05-07T07:23:26Z

gemini review

chatgpt-codex-connector · 2026-05-07T07:26:23Z

Codex Review: Didn't find any major issues. What shall we delve into next?

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

gemini-code-assist Bot reviewed May 3, 2026

View reviewed changes

GrapeBaBa force-pushed the gr/spec-test-coverage-pr branch from 82e77c2 to be97140 Compare May 3, 2026 15:33

GrapeBaBa added 5 commits May 7, 2026 15:00

test(spec): drop redundant comments

6978dbd

test(spec): rename dedup write index to write_idx

59fb86c

test(spec): drop presets motivation comment

5175bde

test(spec): drop FileNotFound comment in collectCases

ea89ae7

GrapeBaBa marked this pull request as ready for review May 7, 2026 07:21

GrapeBaBa requested a review from a team as a code owner May 7, 2026 07:21

test(spec): inline Key.lessThan as method, compact body

7cf000d

GrapeBaBa added 2 commits May 7, 2026 15:31

test(spec): factor Key.eql, dedup loop uses early continue

4a4ef43

test(spec): inline is_dup check

2e8b259

		@@ -92,55 +105,116 @@ pub fn writeTestRoot(comptime kinds: []const RunnerKind, writer: *std.Io.Writer)

		pub fn writeTests(

		var it = cases.keyIterator();
		while (it.next()) \|k\| ordered.appendAssumeCapacity(k.*);

Conversation

GrapeBaBa commented May 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Concrete impact (operations shard, mainnet)

Uh oh!

gemini-code-assist Bot commented May 3, 2026

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist Bot May 3, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot May 3, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot May 3, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot May 3, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot May 3, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot May 3, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot May 3, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot May 3, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot May 3, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot May 3, 2026

Choose a reason for hiding this comment

Uh oh!

GrapeBaBa commented May 7, 2026

Uh oh!

chatgpt-codex-connector Bot commented May 7, 2026

Uh oh!

GrapeBaBa commented May 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

GrapeBaBa commented May 7, 2026

Uh oh!

GrapeBaBa commented May 7, 2026

Uh oh!

chatgpt-codex-connector Bot commented May 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

GrapeBaBa commented May 3, 2026 •

edited

Loading

GrapeBaBa commented May 7, 2026 •

edited

Loading