Skip to content

eval: preserve first-attempt artifacts when retrying failed cases #1701

Open
#1751
Open
eval: preserve first-attempt artifacts when retrying failed cases#1701
#1751
Assignees
Benkapner
Labels
triaged

Description

@Benkapner
Benkapner
opened on Oct 18, 2026

Problem

eval/run-functional.sh retries cases that failed before the agent ran. When retry output is available, it removes the original case directory and replaces it with the retry's directory.

This loses the original attempt's saved results and diagnostic files from that location. The primary run and retry run then both contain the retry result, making the original failure harder to investigate from uploaded artifacts.

Expected behavior

Preserve the original attempt before replacing the results used for scoring. Both attempts should be identifiable within the existing uploaded artifact tree.

Acceptance criteria

  • Original and retry results remain available when the retry succeeds or fails.
  • If the retry cannot start or produces no replacement output, the original results remain available.
  • Retry eligibility, scoring inputs, aggregate accounting, and exit behavior remain unchanged.
  • Runs without retries retain their existing behavior.
  • Preserved files remain covered by the existing secret-scrubbing and artifact-upload paths.
  • Offline regression tests verify preservation using distinct first-attempt and retry contents.

Scope

Artifact preservation, focused tests, and documentation of the artifact layout. No new retry policy, scoring policy, or workflow changes.

Activity

  1. github-project-automation
    added this to fullsend-ai Workstreamon Oct 18, 2026
  2. github-project-automation
    moved this to Todo in fullsend-ai Workstreamon Oct 18, 2026

  3. Benkapner commented on Oct 18, 2026

    @Benkapner
    Benkapner
    on Oct 18, 2026
    ContributorAuthor

    /fs-triage Check this against current main and search for overlapping issues and PRs. Confirm the problem, implementation location, and acceptance criteria. This is a triage-only request: do not trigger coding or apply ready-to-code yet. Identify any scope or compatibility decision that needs agreement before implementation.

  4. fullsend-ai-triage commented on Oct 18, 2026

    @fullsend-ai-triage
    fullsend-ai-triageBot
    on Oct 18, 2026 – with fullsend-ai-triage

    🤖 Finished Triage · ✅ Success · Started 10:05 AM UTC · Completed 10:07 AM UTC

    Commit: 2da538b · View workflow run →

    Runtime: claude · Model: sonnet → claude-sonnet-5-5 · Effort: high · Cost: $0.24

  5. fullsend-ai-triage
    added
    triaged
    on Oct 18, 2026

  6. fullsend-ai-triage commented on Oct 18, 2026

    @fullsend-ai-triage
    fullsend-ai-triageBot
    on Oct 18, 2026 – with fullsend-ai-triage

    Triage against current main confirms the problem. In eval/run-functional.sh, the retry merge removes RUN_DIR/cases/<case> and copies the retry's directory over it. The retry's own <RUN_ID>-retry run directory also remains, so both locations end up holding retry results. The first attempt's workspace under /tmp/agent-eval is scrubbed but is not in the upload path (only eval/runs/ is uploaded), so the original is lost from artifacts. No overlapping open or closed issues or PRs were found.

    Implementation location: eval/run-functional.sh (retry block), eval/run-functional-test.sh (regression tests) and eval/README.md (layout docs). No workflow changes are needed, because eval/runs/ is already scrubbed and uploaded.

    Acceptance criteria: these are consistent with the code. The retry_rc 1 and 2 branches already keep the original in place, so preservation only needs to happen on the replacement path.

    Decisions to agree before implementation:

    1. Preserved location. The suggested default is a directory inside the run dir but outside cases/ (for example first-attempt/cases/<case>). cases/ is what scoring and the per-case checks iterate, so a sibling like <case>-attempt1 there could be scored as a case. A new top-level sibling run dir with its own run_result.json could instead break existing globs, such as the */run_result.json loop in the tests, and any consumer that enumerates run dirs.
    2. Run-level record. Keep the existing behavior, where the per-case record in run_result.json is replaced and the totals are recomputed, and preserve the first attempt only as files. Preserving it inside run_result.json as well would change a scoring input, which the issue rules out.
    3. Duplication. The <RUN_ID>-retry dir and the merged case dir both hold the retry's output. Whether to dedupe is out of scope for this issue.

    Auto-promotion: as requested, this is triage only. The category is other, so it is not eligible for automatic ready-to-code promotion. A maintainer can apply that once the layout decision above is settled.

    Suggested test cases (in eval/run-functional-test.sh, with distinct first-attempt and retry marker content):

    - retry succeeds: scored case dir has retry content; preserved dir has first-attempt content
- retry runs and fails: both preserved; exit code unchanged
- retry cannot start / output copy fails: original remains in cases/<case>
- no retries: no preserved dir; layout unchanged
- existing eval/runs/<agent>/*/run_result.json globs still pass
- token-shaped string in preserved files is redacted by scrub-eval-results.sh

    Next steps:

    • /fs-code — agent creates a PR to implement this issue
    • /fs-code <your instruction> — agent implements with your specific guidance
  7. Benkapner
    self-assigned this
    on Oct 18, 2026

  8. Benkapner commented on Oct 18, 2026

    @Benkapner
    Benkapner
    on Oct 18, 2026
    ContributorAuthor

    the proposed layout makes sense. im waiting with /fs-code until #1700 merges since both changes touch the same runner and tests. Then Ill re-run triage against the updated main before starting implementation.

  9. Benkapner commented on Oct 22, 2026

    @Benkapner
    Benkapner
    on Oct 22, 2026
    ContributorAuthor

    /fs-code Implement this against current main (it now includes #1700, merged today, which touched the same runner and eval/run-functional-test.sh). Re-read eval/run-functional.sh and eval/run-functional-test.sh on main before writing anything.

    Layout decision (agreed on the issue): preserve the first attempt at ${RUN_DIR}/first-attempt/cases/. Inside the run dir so scrub and upload already cover it, outside cases/ so scoring never sees it as a case.

    Change, in the retry_rc -eq 0 branch of eval/run-functional.sh (around lines 338-364):

    • Before replacing a retried case, move ${RUN_DIR}/cases/ to ${RUN_DIR}/first-attempt/cases/ instead of rm -rf. Use mv, not cp, so the first attempt is relocated rather than duplicated.
    • Fail safe: if the preserve step fails, or if first-attempt/cases/ already exists, log a warning and skip replacing that case, leaving the first attempt in cases/. Never delete the original to make room for the retry.
    • Only create first-attempt/ when a case is actually preserved. No empty dir on runs with no retries, or when the retry produced no output for that case ([[ -d "$retried" ]] already guards this).
    • Do not write a run_result.json, or anything else, at the first-attempt/ top level. Nothing new should match an eval/runs//*/run_result.json glob.

    Do not change: pre_agent_failures eligibility, the run-level run_result.json per_case merge and totals recomputation, the exec_exit/first_exec_exit exit logic, the case-failure loop, scoring, or the <RUN_ID>-retry sibling run dir. Deduping the retry run dir is explicitly out of scope. No workflow changes are needed; confirm that rather than editing .github/workflows/functional-tests.yml.

    Verify before you commit: check how score.py in the eval/.agent-eval-harness submodule enumerates cases. If it globs the run dir recursively rather than reading per_case, first-attempt/cases/ could be picked up as cases, so say so in the PR and pick a layout score.py igno

    Tests, in eval/run-functional-test.sh, using distinct marker content per attempt (have the stub harness key its marker on whether the run id ends in -retry):

    • retry succeeds: cases/ holds retry content, first-attempt/cases/ hxit code unchanged.
    • retry runs but the case fails again: both attempts present, exit behavior unchanged.
    • retry cannot start (rc 1) and retry output copy fails (rc 2): cases/ still holds first-attempt content, no first-attempt/ dir left behind.
    • a run with no retries: no first-attempt/ dir, layout otherwise identical.
    • the existing run-level run_result.json merge test near line 641 still passes unmodified.
      Add one test to eval/scripts/scrub-eval-results-test.sh: a ghp_-shaped token planted in a preserved first-attempt file is redacted.

    Docs: add an artifact layout section to eval/README.md covering eval/runs/// with cases/ (scored), first-attempt/cases/ (preserved originals) and the sibling -retry/ run dir, and state which one scoring and the uploaded artifact consumers read.

    Validation: bash eval/run-functional-test.sh, bash eval/scripts/scrub-eval-results-test.sh, make lint (skillsaw --strict), and shellcheck at the pinned pre-commit version. Note: one pre-existing test reads /dev/zero and may be blocked in a sandbox. Work around it locally if needed, but do not change that test.

    Branch agent/1701-preserve-first-attempt-artifacts

  10. fullsend-ai-coder commented on Oct 22, 2026

    @fullsend-ai-coder
    fullsend-ai-coderBot
    on Oct 22, 2026 – with fullsend-ai-coder
    Contributor

    🤖 Finished Code · ✅ Success · Started 1:08 PM UTC · Completed 1:17 PM UTC

    Commit: c309ce4 · View workflow run →

    Runtime: claude · Model: opus → claude-opus-5-5 · Effort: high · Cost: $2.34

  11. github-project-automation
    moved this from Todo to In progress in fullsend-ai Workstreamon Oct 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

triaged

Type

No type

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions