[WG] Evaluation & Quality — charter and Lead/Member self-nomination

[Xunzhuo]

Direction

Make every supported capability measurable and every change verifiable.

Goal

Own the common evaluation framework, quality standards, CI, E2E, compatibility, and regression gates used across all vLLM Semantic Router Workgroups.

In scope

Evaluation contracts

• Common benchmark structure, dataset provenance, metrics, comparison rules, reproducibility, and publication.
• Decision-level routing evaluation and complete Mixture-of-Models evaluation.
• Shared evaluation contracts for Router Models, agents, context/session behavior, serving, platforms, and developer workflows without replacing domain objectives.

Performance, CI, E2E, and regression

• Performance coverage, baselines, reports, and hardware qualification.
• Required PR gates, affected-test selection, CI reliability, and failure classification.
• Behavioral E2E coverage across Router, Dashboard, CLI, deployments, recipes, and supported environments.
• Functional, quality, performance, compatibility, security, reliability, and operational regression.

Readiness evaluation

• Cross-Workgroup definitions of done and evidence required for milestone readiness.
• Upgrade, rollback, partial-deployment, and compatibility validation.
• Explicit blockers, gaps, and reviewed exceptions.

Release admission, milestone commitment, and the final release decision remain Maintainer Team responsibilities.

Owned Epics

View all current Evaluation & Quality Epics

Out of scope

• Choosing another Workgroup's product direction or target.
• Treating every desirable issue as a release commitment.
• Replacing Maintainer review or release authority with automation.
• Owning implementation fixes or model-quality research that belong to another Workgroup.

Shared interfaces

Each technical Workgroup defines and fixes the behavior it owns. Evaluation & Quality supplies shared measurement, comparison, execution, regression, and publication contracts.

Participation and self-nomination

This Workgroup follows the Lead/Member policy in #2978. Reply with your GitHub handle, requested role, merged contribution, intended focus, expected availability, and Sponsor when required.

Operating cadence

• Review active Epics, coverage gaps, unassigned work, and blocked dependencies at least every two weeks.
• Keep every open task under one accountable Epic and one Workgroup owner.
• Close an Epic when its bounded children are complete instead of using it as a permanent inbox.