fix(#22): enforce structured output in code agent by waynesun09 · Pull Request #24 · fullsend-ai/agents

waynesun09 · 2026-07-02T22:48:28Z

Summary

Add "Structured output" section to agents/code.md documenting the requirement to produce code-result.json and validate with fullsend-check-output
Add mandatory step 11 to skills/code-implementation/SKILL.md for final schema validation
Update partial work and failure handling sections to require structured output in all exit paths

Aligns the code agent with the fix and review agents, which both have explicit structured output enforcement. Without this, the validation_loop in harness/code.yaml rejects runs because the agent never produces the expected output file.

Closes #22

fullsend-ai-review · 2026-07-02T22:49:14Z

🤖 Review · ⚠️ Cancelled · Started 10:49 PM UTC · Ended 10:53 PM UTC
Commit: 0a95cac · View workflow run →

qodo-code-review · 2026-07-02T22:49:19Z

PR Summary by Qodo

Enforce structured output validation for the code agent

🐞 Bug fix 📝 Documentation 🕐 10-20 Minutes

AI Description

• Document mandatory code-result.json production and validation for the code agent.
• Add a required final step to schema-validate structured output before exit.
• Require structured output even on partial-work and failure exit paths to satisfy harness.

Diagram

graph TD
  A["Code agent"] --> B["Write code-result.json"] --> C["fullsend-check-output"] --> D["Harness validation_loop"] --> E["Post-script: create PR"]
  C --> F[("code-result schema")]

High-Level Assessment

The following are alternative approaches to this PR:

1. Enforce output generation in the harness/post-script

➕ Reduces reliance on agent instruction adherence
➕ Can fail fast with clearer harness-owned error messages
➖ Harder for the agent to choose/confirm the correct target branch early
➖ Still requires agent context to decide target branch; harness may guess wrong

2. Auto-generate structured output from git context (default branch) unless overridden

➕ Eliminates missing-file failures for common cases
➕ Simplifies agent steps when default branch is correct
➖ Can silently pick an incorrect target branch when issues specify otherwise
➖ Makes branch-selection logic implicit and harder to audit

Recommendation: Keep the PR’s approach (explicit agent-side contract + mandatory final validation). It matches the existing fix/review agent pattern, makes the output requirement unambiguous, and directly addresses validation_loop retries by ensuring the file exists and is schema-compliant across all exit paths (success, partial, failure).

Files changed (2) +69 / -8

Bug fix (2) +69 / -8

code.mdDocument mandatory structured output + validation for code agent +24/-3

Document mandatory structured output + validation for code agent

• Adds a new "Structured output" section requiring the code agent to write '$FULLSEND_OUTPUT_DIR/$FULLSEND_OUTPUT_FILE' (typically 'code-result.json') and validate it via 'fullsend-check-output'. Updates failure-handling guidance to treat valid structured output as part of the handoff contract, including best-effort output for non-commit exits.

agents/code.md

SKILL.mdAdd mandatory final step to schema-validate code agent output +45/-5

Add mandatory final step to schema-validate code agent output

• Extends the step checklist to include step 11 and updates earlier guidance to write structured output early and re-validate at the end. Introduces a dedicated validation step describing the strict schema constraints (including 'additionalProperties: false') and requires validation even for partial-work exits.

skills/code-implementation/SKILL.md

qodo-code-review · 2026-07-02T22:52:09Z

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 📎 Requirement gaps (1) 📜 Skill insights (1)

Context used

✅ Tickets: 🎫 fix: code agent missing structured output instructions and self-validation

✅ Compliance rules (platform): 55 rules

✅ Skills: 4 invoked

code-review

code-implementation

pr-review

docs-review

1. ~~Sandbox output filename missing~~ ✓ Resolved 🐞 Bug ☼ Reliability

Description

Step 11 (and the new agent instructions) require reading/validating
${FULLSEND_OUTPUT_DIR}/${FULLSEND_OUTPUT_FILE}, but harness/code.yaml documents
FULLSEND_OUTPUT_FILE as runner-only and env/code.env does not export it for the sandbox. If the
sandbox env lacks FULLSEND_OUTPUT_FILE, the agent may write to an invalid path and the harness
validation loop will keep failing with missing/invalid code-result.json.

Code

skills/code-implementation/SKILL.md[R741-772]

+### 11. Validate structured output
+
+**This step is MANDATORY.** The harness runs a validation loop that
+checks `$FULLSEND_OUTPUT_DIR/$FULLSEND_OUTPUT_FILE` against
+`schemas/code-result.schema.json`. If validation fails, the harness
+retries the agent. Producing a valid output file is not optional.
+
+You wrote the initial output file in step 3. Confirm it still exists
+and contains the correct target branch:
+
+```bash
+echo "::notice::STEP 11: Validate structured output"
+cat "${FULLSEND_OUTPUT_DIR}/${FULLSEND_OUTPUT_FILE}"
+```
+
+The file must be valid JSON with exactly one field:
+
+```json
+{
+  "target_branch": "main"
+}
+```
+
+**Schema compliance:** The schema uses `additionalProperties: false`.
+Only the `target_branch` field is allowed. Any extra fields will cause
+validation to fail.
+
+Validate the output against the schema:
+
+```bash
+fullsend-check-output "${FULLSEND_OUTPUT_DIR}/${FULLSEND_OUTPUT_FILE}"
+```

Evidence

The new mandatory step 11 and the new agent Structured output section depend on
$FULLSEND_OUTPUT_FILE being set when running inside the sandbox. However, harness/code.yaml
explicitly documents that runner_env vars (including FULLSEND_OUTPUT_FILE) never enter the
sandbox, and the sandbox env file env/code.env does not export FULLSEND_OUTPUT_FILE. Separately,
post-code.sh is hardcoded to look for code-result.json, so a mismatch/unset variable can still
lead to the harness reporting the output file as missing and retrying/failing the run.

skills/code-implementation/SKILL.md[741-772]
agents/code.md[95-108]
harness/code.yaml[47-52]
env/code.env[1-30]
scripts/post-code.sh[64-80]
skills/code-implementation/SKILL.md[189-199]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
The skill and agent docs now mandate using `$FULLSEND_OUTPUT_FILE` in the sandbox, but this repo only sets `FULLSEND_OUTPUT_FILE` for runner-side scripts (validation_loop) and explicitly says runner env vars never enter the sandbox. This can still prevent the agent from reliably writing the required `code-result.json`.

### Issue Context
- `scripts/post-code.sh` searches specifically for `iteration-*/output/code-result.json`.
- `skills/code-implementation/SKILL.md` writes/validates `${FULLSEND_OUTPUT_DIR}/${FULLSEND_OUTPUT_FILE}`.
- `harness/code.yaml` sets `FULLSEND_OUTPUT_FILE` under `runner_env` and documents it does not enter the sandbox.

### Fix options (pick one, ensure consistency end-to-end)
1) **Export `FULLSEND_OUTPUT_FILE=code-result.json` into the sandbox** (preferred minimal change):
  - Add `export FULLSEND_OUTPUT_FILE=code-result.json` to `env/code.env` (or introduce `env.sandbox` in `harness/code.yaml` like `harness/fix.yaml`).
2) **Hardcode `code-result.json` in the skill/docs**:
  - Replace `${FULLSEND_OUTPUT_FILE}` usage in `agents/code.md` and `skills/code-implementation/SKILL.md` with `code-result.json` to match `post-code.sh`.
3) If configurability is desired, **teach `post-code.sh` to honor `FULLSEND_OUTPUT_FILE`** (runner env) and search for that filename instead of hardcoding `code-result.json`.

### Fix Focus Areas
- harness/code.yaml[47-52]
- env/code.env[1-30]
- scripts/post-code.sh[64-80]
- agents/code.md[95-108]
- skills/code-implementation/SKILL.md[189-199]
- skills/code-implementation/SKILL.md[741-772]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

2. Protected paths modified 📜 Skill insight § Compliance

Description

This PR modifies protected governance/infrastructure paths (agents/ and skills/), which require
human review and must not be auto-approved. Even with justification, this needs an explicit
protected-path compliance flag in review.

Code

agents/code.md[R95-102]

+## Structured output
+
+You MUST produce a JSON file at `$FULLSEND_OUTPUT_DIR/$FULLSEND_OUTPUT_FILE`
+(typically `code-result.json`) that documents the target branch for PR
+creation. The `code-implementation` skill describes the schema and the exact
+step where you write it. The post-script reads this file to determine which
+branch to target the PR against. Without this file, the validation loop
+rejects the run and retries.

Evidence
The compliance checklist marks changes under agents/ and skills/ as protected and requires
raising a finding to prevent auto-approval. The diff shows new/modified content in both protected
files.
agents/code.md[95-128]
skills/code-implementation/SKILL.md[741-790]
Skill: pr-review

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
Protected paths were modified (`agents/`, `skills/`), so the PR must not be auto-approved and needs explicit human review acknowledgment.

## Issue Context
The compliance rule requires a finding whenever protected paths are touched; justification lowers severity but does not remove the requirement.

## Fix Focus Areas
- agents/code.md[95-102]
- skills/code-implementation/SKILL.md[741-776]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

3. Structured output section incomplete 📎 Requirement gap ☼ Reliability

Description

agents/code.md adds a "Structured output" section but it does not include the required schema
fields/example or explicit instruction to write the output early, and it is not placed between `##
Failure handling and ## Detailed implementation procedure` as required. This can leave the code
agent without the mandated, self-contained contract guidance and risks repeated harness validation
failures.

Code

agents/code.md[R95-113]

+## Structured output
+
+You MUST produce a JSON file at `$FULLSEND_OUTPUT_DIR/$FULLSEND_OUTPUT_FILE`
+(typically `code-result.json`) that documents the target branch for PR
+creation. The `code-implementation` skill describes the schema and the exact
+step where you write it. The post-script reads this file to determine which
+branch to target the PR against. Without this file, the validation loop
+rejects the run and retries.
+
+After writing the file, validate it before exiting:
+
+```bash
+fullsend-check-output "${FULLSEND_OUTPUT_DIR}/${FULLSEND_OUTPUT_FILE}"
+```
+
+If validation fails, read the error output, fix the JSON file, and
+re-run the check. If it still fails after 3 attempts, write the best
+JSON you have and exit.
+

Evidence

Rule 1540707 specifies both content requirements (schema/example + early-write instruction) and
placement (between Failure handling and Detailed implementation procedure). The added section
appears before ## Failure handling and contains validation instructions but not schema/example nor
early-write guidance.

Code agent definition includes mandatory structured output instructions
agents/code.md[95-130]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The new `## Structured output` section in `agents/code.md` does not meet the documented compliance requirements: placement is wrong (it appears before `## Failure handling`), and the section lacks schema fields + an example and explicit early-write guidance.

## Issue Context
Rule 1540707 requires a dedicated section positioned between `Failure handling` and `Detailed implementation procedure`, with MUST/MANDATORY language, early-write instruction, schema fields, an example, and `fullsend-check-output` instructions.

## Fix Focus Areas
- agents/code.md[95-130]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

View more (1)

4. ~~Step 3 lacks output validation~~ ✓ Resolved 📎 Requirement gap ☼ Reliability

Description

In skills/code-implementation/SKILL.md, step 3 still does not mark output writing as
MUST/MANDATORY and does not run fullsend-check-output immediately after writing the output file.
This violates the skill requirement to make structured output reliably produced and validated early,
not only at the end.

Code

skills/code-implementation/SKILL.md[R189-193]

+Write your chosen branch to the structured output file **early** so
+it is available even if the agent hits a timeout or error later:

```bash
mkdir -p "${FULLSEND_OUTPUT_DIR}"

Evidence
Rule 1540708 requires step 3 to be explicitly mandatory and to run fullsend-check-output after
writing the output file; the updated step 3 text emphasizes "early" but does not add MUST/MANDATORY
language nor the validation command in-step.
Code implementation skill makes structured output mandatory and self-validates it
skills/code-implementation/SKILL.md[189-203]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
Step 3 writes the structured output file but does not (1) label it MUST/MANDATORY and (2) invoke `fullsend-check-output` right after writing, as required.

## Issue Context
Rule 1540708 requires early, mandatory output writing and early validation in step 3, plus a final step 11 verification (which is present).

## Fix Focus Areas
- skills/code-implementation/SKILL.md[189-203]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

qodo-code-review · 2026-07-02T22:52:10Z

+## Structured output
+
+You MUST produce a JSON file at `$FULLSEND_OUTPUT_DIR/$FULLSEND_OUTPUT_FILE`
+(typically `code-result.json`) that documents the target branch for PR
+creation. The `code-implementation` skill describes the schema and the exact
+step where you write it. The post-script reads this file to determine which
+branch to target the PR against. Without this file, the validation loop
+rejects the run and retries.


1. Protected paths modified 📜 Skill insight § Compliance

This PR modifies protected governance/infrastructure paths (agents/ and skills/), which require human review and must not be auto-approved. Even with justification, this needs an explicit protected-path compliance flag in review.

Agent Prompt

## Issue description Protected paths were modified (`agents/`, `skills/`), so the PR must not be auto-approved and needs explicit human review acknowledgment. ## Issue Context The compliance rule requires a finding whenever protected paths are touched; justification lowers severity but does not remove the requirement. ## Fix Focus Areas - agents/code.md[95-102] - skills/code-implementation/SKILL.md[741-776]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

qodo-code-review · 2026-07-02T22:52:10Z

+## Structured output
+
+You MUST produce a JSON file at `$FULLSEND_OUTPUT_DIR/$FULLSEND_OUTPUT_FILE`
+(typically `code-result.json`) that documents the target branch for PR
+creation. The `code-implementation` skill describes the schema and the exact
+step where you write it. The post-script reads this file to determine which
+branch to target the PR against. Without this file, the validation loop
+rejects the run and retries.
+
+After writing the file, validate it before exiting:
+
+```bash
+fullsend-check-output "${FULLSEND_OUTPUT_DIR}/${FULLSEND_OUTPUT_FILE}"
+```
+
+If validation fails, read the error output, fix the JSON file, and
+re-run the check. If it still fails after 3 attempts, write the best
+JSON you have and exit.
+


2. Structured output section incomplete 📎 Requirement gap ☼ Reliability

agents/code.md adds a "Structured output" section but it does not include the required schema fields/example or explicit instruction to write the output early, and it is not placed between `## Failure handling and ## Detailed implementation procedure` as required. This can leave the code agent without the mandated, self-contained contract guidance and risks repeated harness validation failures.

Agent Prompt

## Issue description The new `## Structured output` section in `agents/code.md` does not meet the documented compliance requirements: placement is wrong (it appears before `## Failure handling`), and the section lacks schema fields + an example and explicit early-write guidance. ## Issue Context Rule 1540707 requires a dedicated section positioned between `Failure handling` and `Detailed implementation procedure`, with MUST/MANDATORY language, early-write instruction, schema fields, an example, and `fullsend-check-output` instructions. ## Fix Focus Areas - agents/code.md[95-130]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

The code agent was missing structured output enforcement, causing validation_loop failures. The review and fix agents both have explicit structured output sections and mandatory final validation steps, but the code agent had neither. Changes to agents/code.md: - Add "Structured output" section documenting the requirement to produce code-result.json and validate with fullsend-check-output - Use hardcoded filename (not FULLSEND_OUTPUT_FILE env var, which is only in runner_env and unavailable inside the sandbox) - Update failure handling to reference structured output in the handoff contract Changes to skills/code-implementation/SKILL.md: - Add step 11 as a mandatory final step to validate structured output against schemas/code-result.schema.json - Hardcode "code-result.json" in step 3 output write (replacing FULLSEND_OUTPUT_FILE env var reference) - Add fullsend-check-output at every post-step-3 exit point: step 4 (open PR), step 6 (label-gated), step 7 (already fixed), step 9c (retry limit exhausted) - Add early-exit guidance in step 3 for steps 4/6/7 - Update progress markers list to include step 11 - Require structured output even for partial work exits Closes #22 Assisted-by: Claude Signed-off-by: Wayne Sun <gsun@redhat.com>

fullsend-ai-review · 2026-07-02T22:54:42Z

🤖 Finished Review · ✅ Success · Started 10:54 PM UTC · Completed 11:07 PM UTC
Commit: c509828 · View workflow run →

fullsend-ai-review · 2026-07-02T23:07:48Z

Review

Findings

Medium

[protected-path] agents/code.md, skills/code-implementation/SKILL.md — Both modified files are under protected paths (agents/, skills/). The PR links to issue fix: code agent missing structured output instructions and self-validation #22 and the description explains the rationale for the changes. Human approval is always required for protected-path changes, regardless of context.

Low

[internal-consistency] skills/code-implementation/SKILL.md — Step 6 "Already-fixed" and "Label-gated" early-exit paths say "validate structured output and stop cleanly" but do not include the fullsend-check-output bash block inline, unlike steps 4, 7, and 9c which all include the explicit command. The general instruction after step 3 ("If you stop early (steps 4, 6, or 7), run fullsend-check-output before exiting") covers this, but adding inline bash blocks would improve consistency and reduce the risk of an LLM agent missing the validation call.

Labels: PR fixes a bug where the code agent failed validation due to missing structured output instructions.

qodo-code-review Bot reviewed Jul 2, 2026

View reviewed changes

waynesun09 force-pushed the fix-code-output-enforcement branch from b46b157 to c509828 Compare July 2, 2026 22:53

fullsend-ai-review Bot added requires-manual-review Review requires human judgment bug Something isn't working labels Jul 2, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix(#22): enforce structured output in code agent#24

fix(#22): enforce structured output in code agent#24
waynesun09 wants to merge 1 commit into
mainfrom
fix-code-output-enforcement

waynesun09 commented Jul 2, 2026

Uh oh!

fullsend-ai-review Bot commented Jul 2, 2026 •

edited

Loading

Uh oh!

qodo-code-review Bot commented Jul 2, 2026

Uh oh!

qodo-code-review Bot commented Jul 2, 2026 •

edited

Loading

Uh oh!

qodo-code-review Bot Jul 2, 2026

Uh oh!

qodo-code-review Bot Jul 2, 2026

Uh oh!

Uh oh!

Uh oh!

fullsend-ai-review Bot commented Jul 2, 2026 •

edited

Loading

Uh oh!

fullsend-ai-review Bot commented Jul 2, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

waynesun09 commented Jul 2, 2026

Summary

Uh oh!

fullsend-ai-review Bot commented Jul 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

qodo-code-review Bot commented Jul 2, 2026

PR Summary by Qodo

Uh oh!

qodo-code-review Bot commented Jul 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code Review by Qodo

Uh oh!

qodo-code-review Bot Jul 2, 2026

Choose a reason for hiding this comment

Uh oh!

qodo-code-review Bot Jul 2, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

fullsend-ai-review Bot commented Jul 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fullsend-ai-review Bot commented Jul 2, 2026

Review

Findings

Medium

Low

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

fullsend-ai-review Bot commented Jul 2, 2026 •

edited

Loading

qodo-code-review Bot commented Jul 2, 2026 •

edited

Loading

fullsend-ai-review Bot commented Jul 2, 2026 •

edited

Loading