Open-World Alignment

Evaluation infrastructure for AI systems beyond direct human supervision.

Open-world alignment starts from a practical observation: some AI systems make consequential claims that exceed unaided human ability to generate, rank, verify, or forecast. In those settings, human approval remains necessary, but it should not be the only instance-level target. Human authority needs to move up a level: mandate formation, verifier design, hard constraints, uncertainty-aware escalation, and rollback-capable monitoring.

This repository turns that idea into reusable research artifacts.

Who This Is For

• Researchers writing or reviewing papers about frontier AI evaluation.
• Benchmark builders who need to state what their benchmark can and cannot support.
• Developers of agents, foundation models, scientific ML systems, and forecasting systems.
• Governance, audit, and safety teams that need explicit monitoring and rollback criteria.

What You Can Use Today

• Open-World Evaluation Card template: a reporting artifact for beyond-human or ASI-relevant claims.
• Machine-readable card template: a JSON-compatible YAML card that can be validated without extra dependencies.
• JSON Schema: the current schema for automated checks.
• Specification: the canonical definition of the card, its scope, validation levels, and compatibility policy.
• Filled example cards: coding agents, foundation models, scientific ML, and forecasting.
• Verifier-stack templates: domain checklists for designing stronger evidence channels.
• Reviewer checklist: a compact checklist for papers, benchmarks, and deployment reports.
• Reviewer package: 5-minute, 15-minute, and full artifact-review checklists.
• Failure-mode library: common ways benchmark and evaluation claims become overextended.
• Integrations: lightweight mappings for existing eval frameworks and governance vocabularies.
• Adoption guides: audience-specific workflows for researchers, reviewers, builders, developers, and governance teams.
• Curated resources: a taxonomy of resources for places where direct human judgment stops being enough.

3-Minute Workflow

Copy the template:

cp cards/open_world_evaluation_card.yaml my_card.yaml

Fill in the eight required dimensions:

1. claim_type
2. target_world_state
3. human_signal_role
4. verifier_stack
5. hard_constraints
6. evaluator_frontier
7. absent_stakeholders
8. monitoring_and_rollback

Validate the card:

python3 scripts/validate_card.py my_card.yaml

Or validate all example cards:

find cards -name '*.yaml' -print0 | xargs -0 -n1 python3 scripts/validate_card.py

Open-World Evaluation Card

The card asks each system or benchmark to separate human-facing evidence from world-facing evidence. A good card should make clear:

• what capability is being claimed;
• where unaided human supervision becomes weak;
• which human signals are used and what they cannot verify;
• which hard and soft verifiers test the target claim;
• which constraints cannot be traded away for approval or benchmark score;
• when uncertainty triggers abstention or escalation;
• which affected stakeholders are absent from the rating loop;
• which deployment signals trigger revision, suspension, rollback, or withdrawal.

See cards/README.md for adaptation guidance and quality criteria. The current schema is stable enough for research use, benchmark documentation, and review checklists, but it is not yet a v1.0 standard.

Selected Related Work

• Open Problems and Fundamental Limitations of RLHF: motivates explicit reporting of human-signal limits.
• Weak-to-Strong Generalization: makes the evaluator frontier concrete.
• SWE-bench: shows how coding-agent claims can be tied to executable repository-level checks.
• HELM: demonstrates transparent multi-metric evaluation rather than a single leaderboard score.
• Datasheets for Datasets: a precedent for standardized reporting artifacts.

For a fuller map, see docs/related_work.md and awesome/README.md.

Repository Map

• cards/: card templates, schema, and filled examples.
• scripts/: dependency-light validation tools.
• tests/: validator behavior tests and invalid fixtures.
• rubrics/: maturity rubrics for scoring card quality.
• failure-modes/: common overclaiming and verifier-stack failure modes.
• integrations/: lightweight mappings from existing evaluation frameworks.
• guides/: audience-specific adoption guides.
• verifier-stacks/: domain-specific verifier-stack templates.
• checklists/: reviewer and deployment checklists.
• reviewer-package/: time-boxed checklists and red flags for artifact review.
• awesome/: curated literature and tooling resources.
• docs/: glossary and related-work notes.
• ROADMAP.md: staged path toward v1.0.
• CHANGELOG.md: project history.

Citation

If you use this repository, cite the toolkit:

@misc{he2026openworldalignment,
  title  = {Open-World Alignment: Evaluation Infrastructure for AI Systems Beyond Direct Human Supervision},
  author = {He, Chaoyue},
  year   = {2026}
}

License

Code in this repository is released under the MIT License. Documentation, card templates, checklists, and examples are released under Creative Commons Attribution 4.0 International. See LICENSE and LICENSE-docs.