Skip to content

Add pluggable verifier interface for skill promotion#74

Open
nutstrut wants to merge 1 commit into
aiming-lab:mainfrom
nutstrut:codex/add-pluggable-verifier-interface-for-skill-promotion
Open

Add pluggable verifier interface for skill promotion#74
nutstrut wants to merge 1 commit into
aiming-lab:mainfrom
nutstrut:codex/add-pluggable-verifier-interface-for-skill-promotion

Conversation

@nutstrut

Copy link
Copy Markdown

Motivation

  • Introduce a new verified_skills operating mode to gate automatic skill promotion with pluggable verifiers.
  • Provide a safe verification fallback so skill evolution can run even if verifier configuration fails.

Mode comparison

Mode GPU Required Cloud Required Verification Auditability
skills_only No No None Existing skill metadata
verified_skills No No by default Pluggable verifier Verification metadata + optional receipts
rl No Yes Existing RL evaluation Existing training artifacts
madmax No Yes Existing training path Existing training artifacts

Description

  • Add a verification subsystem under metaclaw/verification with a VerifierInterface, VerificationResult, and implementations: LocalVerifier, NullVerifier, RemoteHTTPVerifier, and SettlementWitnessVerifier, plus a factory with build_verifier/build_verifier_safe helpers.
  • Add VerificationConfig and wire a verification field into MetaClawConfig and _DEFAULTS in config_store, and expose config parsing in ConfigStore.to_metaclaw_config and describe.
  • Integrate verifier into the FastAPI proxy (metaclaw/api_server.py) by building a safe verifier, constructing verification packets for evolved skills, attaching verification metadata, auditing rejected/indeterminate promotions, and skipping promotion unless verifier rules permit it.
  • Add verified_skills to the CLI and interactive SetupWizard, treat it similar to skills_only in launcher and environment setup, and update README.md with mode docs and config examples.
  • Extend SkillManager to persist and load optional metadata.json alongside SKILL.md so verifier metadata is stored with skills.
  • Add unit tests in tests/test_verification.py covering NullVerifier, LocalVerifier behavior, and factory/fallback selection.

Testing

  • Ran the new unit tests with pytest -q tests/test_verification.py and all tests passed.

  • Verified the verification factory fallback by creating a MetaClawConfig with a bad verifier and asserting build_verifier_safe returns a NullVerifier (test passed).

  • Exercised LocalVerifier cases in unit tests covering fail/duplicate/indeterminate/pass verdicts (all assertions succeeded).

  • Focused verifier/runtime tests passed:

  • pytest -q tests/test_verification.py tests/test_memory_system.py tests/test_runtime_state.py

  • Result: 537 passed


Codex Task

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant