Skip to content

test(behaviour): step to assert a stage was skipped by its pre-script #8792

Closed
#8793
Closed
test(behaviour): step to assert a stage was skipped by its pre-script#8792
#8793
Labels
featureFeature-category issue awaiting human prioritizationtriagedTriaged but awaiting human prioritization

Description

@waynesun09
waynesun09
opened on Oct 24, 2026

What is missing

No behaviour step can tell that a stage was skipped by its pre-script. "the harness … workflow completes successfully" passes either way, because a skipped run still uploads its artifact. So a scenario cannot prove that a pre-script skip stopped the agent (see fullsend-ai/agents#1782: pre-review posts "Review skipped" but the agent still runs).

What should happen

Add a Then step in pkg/behaviourtest/steps:

  • the harness "<agent>" run was skipped by its pre-script: for the workflow run the scenario is tracking (w.WorkflowRun), read that run's fullsend-<agent> artifact and require that the pre-script skip was recorded and that no agent ran (no agent-result.json, no dummy behaviour-results.json). Fail with a message that names what was found.

Unit tests with a fake CI/artifact reader for: skipped run, run where the agent ran, missing artifact. Document it in docs/guides/dev/behaviour-testing.md.

Merge criterion

make test and make lint green with the new unit tests.

Activity

  1. waynesun09 commented on Oct 24, 2026

    @waynesun09
    waynesun09
    on Oct 24, 2026
    MemberAuthor

    /fs-code
    Do not sign off commits: no git commit -s and no Signed-off-by trailer.

  2. github-project-automation
    added this to fullsend-ai Workstreamon Oct 24, 2026
  3. github-project-automation
    moved this to Todo in fullsend-ai Workstreamon Oct 24, 2026

  4. fullsend-ai-triage commented on Oct 24, 2026

    @fullsend-ai-triage
    fullsend-ai-triageBot
    on Oct 24, 2026 – with fullsend-ai-triage

    🤖 Triage · ⚠️ Cancelled · Ended 7:43 PM UTC

    Commit: 6603bc5 · View workflow run →

  5. fullsend-ai-coder commented on Oct 24, 2026

    @fullsend-ai-coder
    fullsend-ai-coderBot
    on Oct 24, 2026 – with fullsend-ai-coder
    Contributor

    🤖 Code · ❌ Terminated · Started 7:44 PM UTC · Ended 7:53 PM UTC

    Commit: 6603bc5 · View workflow run →

  6. fullsend-ai-triage commented on Oct 24, 2026

    @fullsend-ai-triage
    fullsend-ai-triageBot
    on Oct 24, 2026 – with fullsend-ai-triage

    🤖 Finished Triage · ✅ Success · Started 7:45 PM UTC · Completed 7:48 PM UTC

    Commit: 6603bc5 · View workflow run →

    Runtime: claude · Model: sonnet → claude-sonnet-5-5 · Effort: high · Cost: $0.20

  7. fullsend-ai-triage
    added
    featureFeature-category issue awaiting human prioritization
    triagedTriaged but awaiting human prioritization
    on Oct 24, 2026

  8. fullsend-ai-triage commented on Oct 24, 2026

    @fullsend-ai-triage
    fullsend-ai-triageBot
    on Oct 24, 2026 – with fullsend-ai-triage

    Triage found this issue ready to implement. Notes beyond the issue text:

    • How the skip is recorded: the CLI exits 0 on a pre-script skip, so only the artifact content tells the cases apart. internal/cli/run.go sets the root span attribute fullsend.prescript.skipped (and skip_reason) in run-telemetry.jsonl, which action.yml uploads inside fullsend-<agent>. internal/evalmeasure already parses that file, so the step can likely reuse it.
    • Artifact download: the existing ensureRunArtifacts helper requires behaviour-results.json (or metrics.json) as a marker, which a skipped run never produces. The new step needs its own download path based on DownloadNamedArtifactFromRun for w.WorkflowRun.
    • Open point for implementers: confirm run-telemetry.jsonl is written in behaviour-test runs. If it is not, another recorded skip signal will be needed. That would be a small scope extension.
    • Status: no PR is open for this issue yet. A code run was requested via /fs-code, and the no-sign-off instruction in that comment applies.

    Proposed test case:

    Fake artifact reader:
1. skipped run (skipped=true, no agent-result.json / behaviour-results.json) -> step passes
2. agent ran (agent-result.json or behaviour-results.json present) -> step fails, message names files found
3. artifact missing / download error -> step fails with clear message
Edges: skipped=true but agent output present; telemetry absent or attribute missing; w.WorkflowRun nil

    Next steps:

    • /fs-code — agent creates a PR to implement this issue
    • /fs-code <your instruction> — agent implements with your specific guidance
  9. github-project-automation
    moved this from Todo to In progress in fullsend-ai Workstreamon Oct 24, 2026

  10. fullsend-ai-coder commented on Oct 24, 2026

    @fullsend-ai-coder
    fullsend-ai-coderBot
    on Oct 24, 2026 – with fullsend-ai-coder
    Contributor

    🤖 Finished Code · ✅ Success · Started 7:44 PM UTC · Completed 7:53 PM UTC

    Commit: 6603bc5 · View workflow run →

    Runtime: claude · Model: opus → claude-opus-5-5 · Effort: high · Cost: $1.75

  11. waynesun09
    added 2 commits that reference this issue on Oct 24, 2026

    fix(#8792): read the skip telemetry from the extracted artifact root

    593476a

    fix(#8792): find skip telemetry in the GitLab artifact layout too

    6c1652b
  12. waynesun09
    closed this as completedin #8793on Oct 24, 2026
  13. github-project-automation
    moved this from In progress to Done in fullsend-ai Workstreamon Oct 24, 2026
  14. jflowers
    added 3 commits that reference this issue on Oct 25, 2026

    test(fullsend-ai#8792): add step asserting a stage was skipped by its…

    2c2b549

    Merge pull request fullsend-ai#8793 from fullsend-ai/agent/8792-pre-s…

    3ea9dd1

    test(fullsend-ai#1784): assert the closed-PR review skip stops the stage

    7171358
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    featureFeature-category issue awaiting human prioritizationtriagedTriaged but awaiting human prioritization

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions