v0.7.0

Changelog

• 3c013b1: Remove early evolution warning from README (@roark47)
• 916256b: Update README.zh.md to remove early stage warning (@roark47)
• a366f2f: feat(config): add per-case mocked MCP override schema and merge (@roark47)
• 939c20a: feat(evaluator): provision MCP config per case with override isolation (@roark47)
• 967125c: fix(config): validate per-case mcp overrides (@roark47)
• cb3a9d6: test(e2e): add per-case mocked MCP override eval fixture and test (@roark47)

v0.6.0

Changelog

• fb41d22: Merge remote-tracking branch 'origin/main' into codex/implement-proposal-0002 (@roark47)
• dc801c2: Revert "docs: add judge agent skill usage guide" (@roark47)
• 9f8776c: ci: bump the github-actions group with 5 updates (@dependabot[bot])
• bdbd453: feat(agent): add SessionResumer interface for multi-turn conversation (@roark47)
• 8a37fd4: feat(agent): implement SessionResumer for Codex engine (@roark47)
• a93ce15: feat(config): add multi-turn conversation schema fields and validation (@roark47)
• 5037fb9: feat(evaluator): add multi-turn conversation execution engine (@roark47)
• 009979c: feat(evaluator): strengthen post_condition and capture semantics (@roark47)
• de841a9: feat(judge): add per-turn assertion rules for multi-turn evaluation (@roark47)
• a7a9196: feat(judge): support judge-specific skills (@roark47)
• 96a3422: feat(report): add per-turn results to report output formats (@roark47)
• 96a3c98: feat(report): surface judge skill metadata (@roark47)
• ab3dfa5: fix(agent): use codex exec resume for non-interactive session resumption (@roark47)
• caacdeb: fix(ci): adapt cleanup to runtime shell descriptor (@roark47)
• 93c6907: fix(evaluator): correct multi-turn execution accounting (@roark47)
• 1028b09: fix(evaluator): proceed to judge on skip_remaining post-condition failure (@roark47)
• 09f3684: fix(judge): address judge skill review feedback (@roark47)
• ca01982: fix(judge): include rule keywords in assertion evidence (@roark47)
• 488f0e9: fix(judge): require explicit judge skill invocation (@roark47)
• 5605b45: fix(report): render multi-turn conversation and failure reasons in HTML (@roark47)
• f972d95: fix: address copilot multi-turn review feedback (@roark47)
• dfaa69f: refactor(runtime): replace TargetGOOS with shell descriptor (@zpzjzj)

v0.5.0

Changelog

• 023ccdf: Merge origin/main into feat/0004-agent-judge-context-delivery (@roark47)
• 271acf1: Merge remote-tracking branch 'origin/codex/ci-lean-gates' into feat/0004-agent-judge-context-delivery (@roark47)
• d0a9df9: Modify CodeQL workflow schedule and runner type (@roark47)
• bbcda43: Set up Python 3.12 in CodeQL workflow (@roark47)
• caa8df5: ci: add automatic e2e smoke gate (@roark47)
• 2ffde96: ci: centralize action cleanup trap (@roark47)
• 93d6c73: ci: clarify runner image workflows (@roark47)
• 5067970: ci: isolate self-eval runner images (@roark47)
• 0d613a6: ci: remove custom CodeQL workflow (@roark47)
• 8d0e4f0: ci: remove scheduled extended checks (@roark47)
• cef1f37: ci: retrigger PR checks (@roark47)
• cba360a: ci: run extended checks in merge queue (@roark47)
• 5a3966e: ci: run workflows on self-hosted runners (@roark47)
• 3e266b2: ci: slim default workflow gates (@roark47)
• 6070fc2: feat(judge): materialize agent judge context (@roark47)
• 4a767b9: fix(judge): address context review feedback (@roark47)

v0.4.0

What's Changed

• refactor(custom-engine): collapse finishLocal's 11-parameter signature (#55) by @zpzjzj in #86
• refactor(agent): decouple nvm bootstrap from agent Run/InstallMCP commands by @zpzjzj in #76
• feat: first-class Windows support across CLI, runtime, and judges by @zpzjzj in #33
• chore: bump go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc from 1.43.0 to 1.44.0 by @dependabot[bot] in #82
• refactor(custom-engine): extract customTransport interface with localTransport (#51) by @zpzjzj in #88
• refactor(custom-engine): share ${...} token parsing between config and agent resolvers (#50) by @zpzjzj in #89
• fix: pin qoder.com installer with sha256 verification by @zpzjzj in #92
• docs: document SKILL_UP_BASH env var for all platforms by @zpzjzj in #91
• test: assert custom engine report content in e2e tests by @zpzjzj in #93
• fix: pin GitHub Actions to commit SHAs by @zpzjzj in #90
• chore: bump go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc from 1.43.0 to 1.44.0 by @dependabot[bot] in #94
• chore: bump go.opentelemetry.io/otel/exporters/stdout/stdoutmetric from 1.43.0 to 1.44.0 by @dependabot[bot] in #95
• ci: bump astral-sh/setup-uv from 8.1.0 to 8.2.0 by @dependabot[bot] in #97
• chore: bump go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp from 1.43.0 to 1.44.0 by @dependabot[bot] in #96
• refactor(runtime): move case metadata into AgentMetadata off generic ExecOptions (#49) by @zpzjzj in #98
• chore(agent): remove unused codexStatusSuccess/codexStatusError consts by @zpzjzj in #101
• feat(custom-engine): HTTP transport JSON request/response core by @zpzjzj in #99
• feat(custom-engine): download artifacts.files[].url artifacts into the report by @zpzjzj in #103
• feat(custom-engine): HTTP transport multipart file upload (http.files) by @zpzjzj in #102
• chore: bump github.com/alibaba/OpenSandbox/sdks/sandbox/go from 1.0.1 to 1.0.2 by @dependabot[bot] in #105
• feat: add Agent Skill Eval GitHub Action by @zpzjzj in #100
• ci: bump docker/setup-buildx-action from 3.12.0 to 4.1.0 by @dependabot[bot] in #111
• ci: bump actions/checkout from 4.3.1 to 7.0.0 by @dependabot[bot] in #109
• ci: bump actions/upload-artifact from 4.6.2 to 7.0.1 by @dependabot[bot] in #108
• docs+test(custom-engine): finish http transport rollout by @zpzjzj in #107
• docs: correct the action's uses: ref (v0.1.0 was a nonexistent tag) by @zpzjzj in #112
• feat(engine): add built-in qwen_code engine (#116) by @zpzjzj in #117
• ci: bump docker/login-action from 3.7.0 to 4.2.0 by @dependabot[bot] in #114
• ci: bump actions/setup-go from 6.4.0 to 6.5.0 by @dependabot[bot] in #113
• ci: bump docker/build-push-action from 6.19.2 to 7.2.0 by @dependabot[bot] in #115
• docs: add multi-turn conversation evaluation support proposal (SUP-0001) by @roark47 in #40
• chore: bump github.com/alibaba/OpenSandbox/sdks/sandbox/go from 1.0.2 to 1.0.3 by @dependabot[bot] in #110
• fix: restore internal/runtime build after OpenSandbox SDK v1.0.3 bump by @zpzjzj in #125
• ci: cut shared-runner queueing on PRs by @zpzjzj in #124
• docs: cut CHANGELOG for v0.3.0 by @zpzjzj in #126
• fix: preserve exec perms on skill install; validate duplicate case IDs (#118, #123) by @zpzjzj in #127
• feat(cli): validate only selected cases in run (#121) by @zpzjzj in #131
• ci: bump docker/build-push-action from 7.2.0 to 7.3.0 by @dependabot[bot] in #128
• ci: bump docker/login-action from 4.2.0 to 4.4.0 by @dependabot[bot] in #129
• ci: bump docker/setup-buildx-action from 4.1.0 to 4.2.0 by @dependabot[bot] in #130
• fix: gracefully terminate none-runtime process group on cancel by @zpzjzj in #132
• docs(proposals): add per-case mocked MCP proposal by @roark47 in #139
• Implement multi-turn conversation features with validation and reporting by @roark47 in #136

Full Changelog: v0.2.4...v0.4.0

v0.3.0

Added

• Built-in qwen_code engine (aliases qwen-code, qwen), backed by the
  Qwen Code CLI (@qwen-code/qwen-code).
  Installed on demand via npm (Node.js bootstrapped automatically) and run
  non-interactively with qwen --yolo -p <instruction>. Authenticates against
  any OpenAI-compatible endpoint through OPENAI_API_KEY / OPENAI_BASE_URL,
  with engine.model.name / --model forwarded as both the -m flag and
  OPENAI_MODEL; MCP servers install via qwen mcp add.
• Custom Engine http transport (engine.custom.transport: http): calls a
  remote (or local) HTTP agent service, POSTing the SessionInput and parsing
  the SessionResult from the response. Renders ${...} references in
  http.url / http.headers / http.request_body; a request_body field
  whose value is exactly ${session_input}, ${messages}, or ${kwargs} is
  injected as a JSON structure rather than a string. A non-2xx status is treated
  as an engine execution error, and the API key is rejected in the URL (use a
  header or the request body instead) and masked in responses.
• Custom Engine http transport multipart file upload (engine.custom.http.files):
  when files are declared the request becomes multipart/form-data — the JSON
  body is sent as the payload field and each matched workspace file as a
  files part (the part filename is the workspace-relative path). Paths may be
  exact or doublestar globs (*, **), are confined to the workspace, and only
  regular files are uploaded; required: true (the default) errors when an
  entry matches nothing, required: false skips it.
• Custom Engine: download artifacts.files[].url artifacts into the report. A
  result that declares a downloadable url is fetched (http/https only) and
  written into the case artifact directory under name. The download is
  best-effort and bounded (256 MB cap, request timeout): a non-2xx status,
  transport error, non-http(s) scheme, or over-cap body is logged and skipped
  without failing the run. A URL embedding the configured API key is refused, and
  logged errors are scrubbed of the configured api_key / engine.custom.env
  secrets.

v0.2.4

Changelog

• f81609a: ci: bump actions/upload-artifact from 5 to 7 (@dependabot[bot])
• 57e5715: ci: bump golangci/golangci-lint-action from 8 to 9 (@dependabot[bot])
• 5fa437a: feat(evaluator): collect_artifacts glob to download workspace files on any outcome (#85) (@zpzjzj)
• 653ae83: feat: add Custom Engine with local transport (#24) (@zpzjzj)
• a54597b: fix(runner): report skill name from SKILL.md frontmatter, not dir basename (@zpzjzj)
• 03bce5e: refactor(agent,runtime): unify PATH/env handling via command prelude (#63) (@zpzjzj)

v0.2.3

Changelog

• af7a2ca: ci: bump actions/deploy-pages from 4 to 5 (@dependabot[bot])
• cd41d6d: ci: make revive violations actually fail the Lint job (@zpzjzj)
• 11d5ca4: feat(agent): DEBUG-log unknown engine.kwargs keys to catch typos (@zpzjzj)
• e237098: feat(engine): add engine.kwargs (CLI --engine-kwarg/--ek); codex honors bypass_sandbox (@zpzjzj)
• bf519ad: test: improve unit coverage (#71) (@lbfsc)

v0.2.2

Changelog

• c80f11e: ci: bump actions/download-artifact from 5 to 8 (@dependabot[bot])
• 449215f: ci: bump actions/setup-go from 5 to 6 (@dependabot[bot])
• cdcb0af: feat(runtime): add docker runtime for local container-isolated evals (#56) (@zpzjzj)
• a6e284b: fix(ci): publish coverage badge from badges branch (#64) (@lbfsc)
• dcdb5d8: fix(codex): synthesize provider override when ModelProvider is empty but BaseURL is set (@zpzjzj)
• ba1b06d: fix(config): apply documented defaults when loading eval.yaml (@zpzjzj)
• d04fe13: fix(evaluator): exclude parent-supplied deadlines from case-timeout label (@zpzjzj)
• c7bfb5a: fix(evaluator): gate case-timeout annotation on case ctx actually expiring (@zpzjzj)
• 5eb811e: fix(runtime): surface stdout when runtime.exec exits non-zero (@zpzjzj)
• 9fc85b5: fix(timeout): strict case-timeout budget, judge knob for agent_judge, named errors (@zpzjzj)
• 87fa0b2: refactor(timeout): hoist annotate-gate decisions to caller (fix revive lint) (@zpzjzj)
• b9f6d9f: test(judge): check type assertion in agent_judge timeout test (@zpzjzj)

v0.2.1

Changelog

• b4f8f57: Revert "fix(claude_code): preserve provider prefix in --model for custom proxies" (@zpzjzj)
• 3d020a3: fix(claude_code): also write ANTHROPIC_AUTH_TOKEN for Bearer-auth proxies (@zpzjzj)
• 632abc3: fix(claude_code): preserve provider prefix in --model for custom proxies (@zpzjzj)
• c670ff9: fix(cli): gate --model provider/name split on provider being configured (@zpzjzj)
• 26239e0: fix(cli): honour --api-key, eval.yaml base_url, and framework-default providers (@zpzjzj)
• 1836d12: fix(cli): let opaque slashed model ids pass through with --api-key (@zpzjzj)
• 724ae51: fix(cli): scope collapse to CLI splits, honour --api-key and Qoder PAT (@zpzjzj)
• a046966: fix(credential): qoder PAT probe is qoder-specific, not a generic suffix (@zpzjzj)
• 1e9d9ed: fix(runner): generate report.json when --format json is specified for run subcommand (#58) (@lbfsc)
• f684a9f: fix: gofmt credential.go after adding EnvAnthropicAuthToken (@zpzjzj)
• 7f7a3c7: refactor(credential): extract ResolveModelRef + Resolver.HasProvider (@zpzjzj)
• 2a9c7ce: refactor(credential): pass CLI hints into ResolveModelRef predicate (@zpzjzj)
• b3cb00f: test(cli): consolidate preserves-split cases into table-driven test (@zpzjzj)

v0.2.0

Changelog