Summary

The openhands harness fails to load skills during evoskill run / evoskill eval, so every agent attempt errors out and accuracy is always 0. The goose harness works end-to-end on the same project, so this looks specific to the OpenHands integration with the currently-resolved openhands-sdk.

Environment

• EvoSkill: installed from main via uv tool install git+https://github.com/sentient-agi/EvoSkill.git
• openhands-sdk: 1.24.0 (resolved from the >=1.16.1 constraint in pyproject.toml)
• OS: macOS (Apple Silicon), Python 3.12
• Model: openrouter/minimax/minimax-m2.7 (OpenRouter)

Repro

With an OpenRouter/OpenHands config (e.g. the bundled examples/officeqa/.evoskill/config.openrouter.toml, which uses name = "openhands"):

cd examples/officeqa && bash setup.sh
evoskill eval --config .evoskill/config.openrouter.toml --verbose

Error

WARNING  Attempt 1/2 failed: module 'openhands.sdk.context.skills' has no attribute 'load_skills_from_dir'. Retrying in 30s...
WARNING  Attempt 2/2 failed: module 'openhands.sdk.context.skills' has no attribute 'load_skills_from_dir'. Retrying in 60s...
Failed on question: ...  Error: module 'openhands.sdk.context.skills' has no attribute 'load_skills_from_dir'
  Accuracy: 0.0%  (0/2 correct)

Likely cause

openhands.sdk.context.skills.load_skills_from_dir appears to have been removed/renamed in a newer openhands-sdk than the integration targets. The >=1.16.1 lower bound resolves to 1.24.0, which no longer exposes that symbol.

Workarounds

• Use the goose harness (name = "goose" with the same OpenRouter model) — validated working end-to-end (full propose → generate → evaluate → frontier loop, real skill edits).
• Or pin openhands-sdk to the last version that exposes load_skills_from_dir.

Suggested fix

Update the OpenHands skill-loading call to the current openhands-sdk API, or add a compatible upper bound on openhands-sdk.

Happy to send a PR if useful — really enjoying EvoSkill; building a Sentient Arena Challenge-1 community quickstart on top of it. 🙏