Mission/rust rewrite benchmark publication - #1560
Open
sagaragas wants to merge 113 commits into
Open
Conversation
…d matrix Create reproducible manifest generation tooling: - scripts/parity/build_fixture_manifest.py: hashes all test fixtures into manifests/fixtures.json with a stable top-level manifest hash - scripts/bench/build_scenario_manifest.py: generates corpus, query, and scenario manifests with a canonical CLI command matrix for ag, rust-ag, rg, and ugrep comparators - 12 benchmark scenarios covering literal, regex, case-insensitive, context, count, and file-listing search patterns - All manifests include schema version, checksums, and --verify support Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
… testing Implements run_matrix.py to execute baseline ag and candidate command variants, capture stdout/stderr/exit-code, and emit normalized machine-diff artifacts. Implements validate_outputs.py to verify artifact completeness and consistency. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
Add reproducible edge-case fixtures covering: ignore-source matrix (.gitignore, .ignore, git/info/exclude, -U flag), hidden files (default vs --hidden), binary files (default vs --search-binary), symlink/device traversal (-f flag with platform skip metadata), large-file behavior (~2MB fixture), zero-length regex semantics (^, contradictory boundaries), and max-count truncation (--max-count). - tests/edge-cases/setup_fixtures.py: Deterministic setup script - tests/edge-cases/platform_metadata.json: Platform applicability notes - 15 edge-case scenarios added to manifests/scenarios.json - 4 edge-case queries added to manifests/queries.json - scripts/parity/run_matrix.py: Support --group edge-cases - scripts/parity/validate_fixture_integrity.py: Integrity validator - scripts/parity/build_fixture_manifest.py: Handle broken symlinks Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
…ation - Add Cargo workspace with rust-ag binary crate - Implement placeholder binary with --version, --help, and ag-compatible flag acceptance - Add CLI smoke tests (5 passing) and parity skeleton tests (9 ignored, awaiting search impl) - Add 'rust' target alias in parity runner for --target rust convenience - Add debug build paths to parity runner binary resolution - Add target/ to .gitignore for Rust build artifacts Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
Add milestone scrutiny review reports and synthesis, and record parity validation commands in shared services for reproducible validator execution.
…arsing Replace naive str.split() with shlex.split() in run_command() so quoted arguments and paths/patterns containing spaces are preserved as single tokens. Add quoted-args-regression scenario group with corpus-with-spaces and pattern-with-spaces test scenarios, fixture files, and unit tests. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
- Add one-device fixture builder with cross-device symlink support (Linux) and platform-conditional marker on macOS/Windows - Add edge-one-device-restricted and edge-one-device-follow scenarios with platform_skip metadata in scenarios.json - Harden large-file fixture with SHA-256 checksum sidecar for determinism verification on clean checkouts - Update validate_fixture_integrity.py with one-device and large-file checksum validation (Stage 7 and Stage 8) - Update setup_fixtures.py --verify to fail loudly when one-device or large-file assets are missing or corrupted - Update run_matrix.py one_device_unavailable skip logic to use fixture marker for authoritative cross-device availability Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
Exclude build-generated artifacts from fixture and corpus manifest hashing using a two-layer filter: 1. git ls-files restricts to version-controlled files only 2. Explicit EXCLUDE rules for .o, .deps/, .dirstamp, config.h, stamp-h1, .pyc, __pycache__, .err, .trs, .Po files Also fixes broken symlink crash in build_scenario_manifest.py by adding symlink-safe stat handling (matching build_fixture_manifest.py). Add 20-test suite verifying manifest reproducibility, build-artifact exclusion, and _should_skip rule coverage. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
…verage Add 8 edge-case scenarios (edge-ignore-source, edge-hidden-files, edge-binary-files, edge-symlink-traversal, edge-one-device, edge-large-file, edge-zero-length-regex, edge-max-count) and 2 new queries (q-edge-needle, q-edge-zero-len) to the benchmark scenario manifest generator. Symlink and one-device scenarios include platform_skip metadata. Regenerated manifests pass both build_scenario_manifest.py --verify and validate_fixture_integrity.py. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
Extract symlink creation into _update_latest_symlink() helper that computes correct relative paths for both default (child of artifacts_base) and custom (external) output directories. Previously, the symlink always used run_dir.name which only worked when the run directory was inside parity-artifacts/. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
… regressing quoted-arg handling
Replace shlex.split(posix=True) with a custom tokenize_command() helper
that uses shlex.split(posix=False) plus quote stripping. This preserves
literal backslashes in unquoted tokens (e.g. \b word boundaries in
regex-complex scenarios) while still merging quoted arguments containing
spaces into single argv tokens.
The previous shlex.split() call in POSIX mode silently ate backslashes,
causing regex patterns like \b[A-Z]{2,}_[A-Z]{2,}\b to be mangled into
b[A-Z]{2,}_[A-Z]{2,}b (no word boundaries), producing zero matches on
the regex-complex scenario.
Tests expanded from 8 to 14 cases covering backslash preservation,
combined backslash+quoted scenarios, and regression guards.
Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
…tion Replace single-directory presence check in run_matrix.py with comprehensive preflight that validates required marker files per fixture category. Partial or missing trees trigger targeted auto-setup instead of silent failures. Changes: - setup_fixtures.py: Add REQUIRED_MARKERS dict, preflight_check() for fast structural validation, setup_categories() for targeted rebuilds - run_matrix.py: Add _resolve_needed_edge_categories() to map scenario selections to fixture categories, _preflight_edge_fixtures() for auto-setup with fail-fast on persistent failures - test_edge_preflight.py: 16 tests covering partial trees, missing dirs, category resolution, and marker consistency Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
…on controls Add complete search implementation for the rust-search-core milestone covering: - Recursive file discovery with sorted traversal (VAL-CORE-001) - Default ignore stack: .gitignore, .git/info/exclude, .ignore (VAL-CORE-002) - -U skip-vcs-ignores semantics matching baseline ag (VAL-CORE-003) - Hidden file behavior with --hidden flag (VAL-CORE-004) - Recursion controls: -r, -n/--norecurse, --depth NUM (VAL-CORE-009) New modules: opts.rs (CLI parsing), ignore.rs (ignore engine with glob matching), walk.rs (directory traversal), search.rs (regex/literal matching). 17 parity tests + 4 unit tests + updated skeleton tests all passing. Parity matrix core-ignore-recursion group: 4/4 pass. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
…line, word-boundary, zero-length regex - Add CaseMode enum to opts.rs with 'last wins' flag precedence tracking for -i/-s/-S, matching baseline ag behavior across all flag combinations - Rewrite search.rs with multiline search engine that faithfully simulates ag's PCRE_MULTILINE print-state-machine for line reporting - Support multiline patterns (literal \n in regex can span lines) while keeping . from matching \n (no PCRE_DOTALL), matching ag's default - Implement --nomultiline as line-by-line search mode - Handle zero-length regex matches safely: advance past zero-length to prevent infinite loops, skip empty files, skip phantom trailing lines - Add edge-case fixture directories: case-sensitivity/, multiline/, word-boundary/ with deterministic test content - Add parity_core_matching.rs with 20 parity tests covering VAL-CORE-005, VAL-CORE-006, VAL-CORE-010, VAL-CORE-012, VAL-CORE-013 - Add core-matching scenario group to scenarios.json with 11 new scenarios - Update parity_skeleton.rs to remove ignore marker from VAL-CORE-005 Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
…-count parity - Binary files: detect null bytes, emit 'Binary file X matches.' for --search-binary/-u mode matching ag output format - Symlinks: skip all symlinks by default (file and directory); follow with -f; report broken symlinks to stderr - One-device: implement --one-device traversal restriction using Unix st_dev with platform-conditional support - Max-count: emit 'ERR: Too many matches' diagnostic to stderr when match count >= max_count, matching ag's boundary behavior - Large-file: verified parity with baseline on ~2MB fixture - Added 20 parity tests in parity_core_edge_cases.rs covering VAL-CORE-007, VAL-CORE-008, VAL-CORE-011, VAL-CORE-014 - Tagged edge scenarios with 'core-edge-cases' group in manifests - All parity matrix scenarios pass (5/5) Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
Add MultilineMode enum (mirroring CaseMode) so --nomultiline/--multiline ordering uses last-wins resolution instead of no_multiline always winning. Add 3 parity tests for conflicting flag ordering and 2 scenario manifest entries for the parity matrix. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
…n/-r last-flag-wins - Add pop_directory() to IgnoreEngine to restore ignore state when leaving a directory, preventing ignore rules from leaking across sibling directories - Add RecurseMode enum with last-flag-wins semantics for -n/-r ordering, matching baseline ag behavior where -n -r recurses and -r -n does not - Add ignore-scope-leak test fixture with sibling directories - Add 7 new parity tests covering scope isolation and flag precedence - Add 3 new parity scenarios to core-ignore-recursion group - Update build_scenario_manifest.py with groups and comparator subset support Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
…ign max-count diagnostics with baseline - Binary-enabled mode now returns real match results when -c or -l is set, printing counts/filenames instead of 'Binary file X matches.' message - Binary file -c counting uses regex match regions (not line count) to match ag's match-region-based counting semantics - max_count_hit diagnostic is now based on positive (non-inverted) match count, matching ag's behavior where the diagnostic fires regardless of -v flag - Invert-match with --max-count now scans for positive matches up to max-count then inverts: lines not matched in the scanned portion plus all lines after the scan cutoff appear in inverted output (matching ag's invert_matches() post-scan behavior) - Added 7 new parity tests covering --search-binary -c/-l and --max-count -v Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
ag treats --max-count=0 / -m0 as 'no limit' (its internal default is 0, guarded by > 0 checks in search.c). Our rust-ag was treating 0 literally, causing: - Spurious 'Too many matches' diagnostics (total >= 0 always true) - Wrong output in -v/invert mode (0 regions collected = all lines inverted) - Panic in debug mode from usize underflow (opts.max_count - 1 when 0) Fix: normalise max_count=0 to the default cap (10000) after parsing, matching ag's 'no limit' semantics. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
…of 10,000 cap ag treats -m0/--max-count=0 as truly unlimited (search.c guards on > 0). The prior fix incorrectly mapped 0 to the default cap of 10,000, which caused: - Truncated output at 10,000 lines on high-match files (ag outputs all) - Spurious 'Too many matches' diagnostic on stderr - Wrong inverted output for -m0 -v (2050 false positives on 12050-line files) Fix: map max_count=0 to usize::MAX so downstream >= comparisons never trigger. Add 12050-line all-match fixture and 4 high-match parity tests covering both multiline/nomultiline modes with and without -v. Preserves --max-count=N (N>0) truncation and diagnostic behavior. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
…tdin stream mode - Fix --count to report regex match occurrences (not line counts) matching ag - Add match_count field to FileSearchResult for accurate occurrence counting - Implement stdin/stream detection and searching via pipe/FIFO detection - Stream -c mode outputs per-line match counts (ag's stream count semantics) - Implement --nofilename: suppresses filename prefix AND line numbers - Implement --nonumbers: suppresses line numbers while keeping filename prefix - Implement --numbers: forces line numbers on (survives stream-mode override) - Match ag's stdin-pipe quirk: line numbers suppressed when stdin is pipe even with explicit paths (ag's stream settings leak to file mode) - Add blank line separators between files in --nofilename multi-file mode - Add 21 parity tests covering VAL-CLI-001, VAL-CLI-002, VAL-CLI-003 Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
…ring, and help/version - Add context output (-A/-B/-C) with correct separators (:/-) and -- group delimiters - Add ANSI color output matching ag's escape codes (path, line number, match highlight) - Add -g filename pattern mode (list files matching regex) - Add -G file-search-regex filtering (filter files before content search) - Add output.rs module for color/context formatting - Update help text to match ag's documented flags - Fix opts to handle -g positional arg parsing and color last-wins semantics - Add 22 parity tests covering VAL-CLI-004/005/006/007 - Add cli-formatting parity scenarios to scenarios.json Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
…olved parity_run_ids - run_clean_checkout_smoke.py: creates git worktree at cited commit for isolated smoke execution (not caller working tree); records provenance fields (requested_commit_sha, checked_out_commit_sha, execution_context) - traceability_check.py: removes latest fallback for parity_run_ids (unresolved IDs are hard failures); validates provenance fields including schema_version >= 2 and execution_context == isolated_worktree - publication_checklist.json: schema_version 3 with explicit pass/fail for lineage and executed clean-checkout reproducibility checks - clean_checkout_reproducibility.json: regenerated with schema v2 provenance Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
…orts Update rewrite-worker, benchmark-worker, and publication-worker SKILL.md to reflect actual expected procedures based on suggestedGuidanceUpdates collected across all scrutiny synthesis rounds. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
…eration errors - Remove --depth 1 from cargo tree invocation to enumerate full transitive dependency closure (6 deps instead of 2 direct deps) - Raise DependencyEnumerationError on cargo not found, non-zero exit, or timeout with actionable diagnostic messages - Report missing transitive deps with clear attribution guidance - Use --prefix none for cleaner line-oriented cargo tree parsing Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
…enchmark data - Enhanced reconcile_metrics.py to numerically validate narrative speedup claims (e.g., '2.0× speedup') against benchmark-derived ratios from sampling_validation.json, failing reconciliation on mismatches beyond configurable tolerance (--speedup-tolerance, default 0.15). - Added reconcile_speedup_table() to validate Table 2.2d per-run-type speedup ratios against claim_gate.json pair evaluations with tight tolerance (0.02). - reconciliation_report.json now includes explicit speedup_claim_reconciliation section with per-claim pass/fail evidence for both narrative claims and table claims (schema_version bumped to 2). - Added 28 tests covering extraction, validation, tolerance boundaries, failure detection, and live data integration. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
…mpliance
Include all canonical comparators (ag, rust-ag, rg, ugrep) in cli-stream-default,
cli-stream-count, and cli-stream-numbers scenarios. Retain {corpus} placeholders
in all command templates while preserving stdin_data field for stream behavior.
Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
…, always-persist report - Add 'speedup over' phrasing pattern to extract_memo_speedup_values() so narrative claims like 'rust-ag achieved a 2.0× speedup over the original ag' resolve their comparator pairs correctly. - Treat unresolved comparator pairs and missing benchmark pair data as reconciliation failures instead of silently skipping them. - Always write publication/reconciliation_report.json before returning, on both pass and fail paths, with explicit speedup-claim failure evidence and diagnostics. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
…scenarios
In both the benchmark harness and parity runner, stdin_data scenarios now
correctly strip the {corpus} positional argument from expanded commands and
pipe stdin_data to the process's stdin. This preserves stdin-driven search
semantics while keeping {corpus} placeholders in manifest templates for
integrity checks.
Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
- Update AGENTS.md test-first policy to distinguish Rust code (strict red-first) from tooling/publication features (scoped-validator-driven), removing the contradiction with rewrite-worker/publication-worker skill procedures. - Add stdin/stream semantics safeguard to benchmark-worker SKILL.md under comparator completeness policy, preventing regressions where template integrity enforcement breaks stdin-only execution paths. - Broaden rewrite-worker SKILL.md with a second example handoff for tooling/manifest/non-Rust features alongside the existing Rust parity example. - Update benchmarking library notes with stdin/stream semantics reminder. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
reconcile_speedup_table was ignoring its tolerance parameter and using a hardcoded table_tol=0.02. Now the caller-provided tolerance flows through consistently so pass/fail outcomes change with configured thresholds. Also records effective_tolerances in reconciliation_report.json and adds table_claims.tolerance to the speedup claim reconciliation section. Added 4 new tests proving tolerance controls outcomes end-to-end. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
Summarize validator outcomes and feature review results for this milestone, including recommended skill-guidance follow-up. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
…guidance - Update reconcile_metrics --speedup-tolerance help to explicitly state tolerance applies to both narrative speedup claims and speedup table values - Add full-vs-scoped validation guidance to publication-worker SKILL.md step 9 - Add deviation logging requirement for scoped validator runs Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
Rewrite README to reflect the Rust rewrite workspace, parity/benchmark harness usage, publication evidence workflow, and current local validation commands. Upstream-only install/build instructions are folded into a collapsible reference section. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
Produces a polished, public-readable blog post draft alongside the validated technical memo. The draft preserves all 31 claim IDs and evidence linkage from claim_evidence_map.json while restructuring the content for a narrative arc suitable for ragas.dev/blogs. Passes check_claims.py (4/4) and style_gate.py (4/4). Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
…raft tasks Expand step 9 with explicit taxonomy: full suite mandatory for claim/evidence changes, scoped validation acceptable for blog-draft derivative outputs and polish-only tasks, skill/tooling doc changes exempt. Strengthen deviation logging requirements with 4-item checklist for handoff quality. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.