evaluate: working_dir undocumented, brownfield default not used as fallback

[thisisjun786]

Bug: evaluate cannot access project source files without explicit working_dir

Summary

OOO evaluate's Stage 2 sandbox cannot access project source files unless working_dir is explicitly set. Without it, .ouroboros_eval_artifact.md only contains a file path pointer, and the evaluator can only see ~/.hermes/ config files — causing Stage 2 to reject all ACs with "source code not present."

Root Causes (3 issues)

1. evaluate SKILL.md missing working_dir parameter — The documented MCP tool signature in skills/evaluate/SKILL.md omits working_dir entirely, even though it's a required parameter for correct sandbox file visibility.

2. working_dir described as "Stage 1 only" — In the tool definition, working_dir is documented as affecting only Stage 1 (mechanical verification), but in practice it controls the entire sandbox's file visibility, including what Stage 2 can see.

3. Brownfield default / seed metadata not used as fallback — A brownfield-registered project with a default and a seed with project metadata should automatically set working_dir for evaluate, but this fallback path is missing.

Reproduction

# Greenfield project, brownfield-registered
ouroboros_start_evaluate(
  session_id="...",
  artifact="eval_output.txt",
  seed_content="...",
  acceptance_criteria=[...]
  # working_dir NOT set
)
# Result: 0/6 ACs — evaluator can't find project files
# Fix: add working_dir="/path/to/project"

Impact

• Greenfield projects: evaluate returns REJECTED (0/6) regardless of actual code quality
• Users resort to trial-and-error (9 attempts in our case) to discover working_dir
• The timeout/retry loop burns significant LLM tokens

Suggested Fix

1. Add working_dir to evaluate SKILL.md documentation with clear description
2. Correct the tool definition to reflect that working_dir affects ALL stages, not just Stage 1
3. Implement fallback: if working_dir is unset, use brownfield default → seed metadata project_dir → session cwd

Environment

• OOO version: v0.42.4
• Runtime: GJC
• Project type: greenfield, Python 3.12, FastMCP

[Q00]

This is a duplicate of #1490. I opened #1492 to address the shared evaluate working_dir documentation and fallback behavior, and referenced this issue there.

Posted by agentos-roadmap-warden — bot. Reply with /warden ignore to suppress further comments on this thread.

[ouroboros-agent]

Attempted automated fix via the Ouroboros pipeline/direct fallback, but could not produce a fully verified draft PR in this environment.

What was attempted:

• Implemented the evaluate working_dir fix in /tmp/issue_repo on branch issue-1491/evaluate-workingdir-undocumented-brownfi.
• Documented working_dir in the evaluate skill docs.
• Updated evaluate/start-evaluate schema text to state that working_dir affects all evaluation stages, including Stage 2 source collection.
• Added fallback resolution: explicit working_dir -> brownfield default repo -> seed metadata project_dir/working_directory -> MCP server cwd.
• Wired the shared BrownfieldStore into both evaluate handlers and added focused regression tests.

Verification:

• uv run ruff format . passed.
• Targeted regression tests passed: 201 passed.
• Full suite command uv run --with pytest-timeout pytest tests/ -x --timeout=60 progressed to 7767 passed / 7 skipped, then failed in tests/unit/orchestrator/test_profile_loader.py::TestWheelPackaging::test_wheel_contains_profile_yamls because uv build could not fetch PyPI build dependencies (hatchling, hatch-vcs) after DNS/tunnel errors.

Where it stalled:
The requested full verification did not pass due an environment/network package-fetch failure, so no commit, push, or draft PR was created. This needs human review to either rerun verification in a network-capable environment or decide whether to proceed with the uncommitted patch.

ouroboros-agent[bot]