Context
Found by adversarial review of PR #154 (rubric-review). Blocking. Not covered by the existing follow-ups (#155–#160).
Problem
PR #154 replaces rubric20's hard-coded Applies to: lists with a conditional gate: "Always report results… but only score if <X> is identified elsewhere." For several questions the only place <X> can be identified is the question's own scoring fields, creating a circular gate that can suppress exactly the items it is meant to score.
Examples in .claude/agents/d4d-rubric20-semantic.md:
- Q8 Ethical and Privacy Declarations — gate (line 209): "only score if human subjects or governance restrictions are identified elsewhere." Q8's own fields (line ~199):
ethical_reviews, human_subject_research, is_deidentified, participant_privacy, … i.e. the "elsewhere" evidence is Q8's evidence.
- Q9 Access Requirements and Governance (line 225) — same governance gate over its own governance fields.
- Q11 Tool and Software Transparency (line 259) — "only score if software tools were identified elsewhere" over its own software fields.
The v2.1 source rubric instead uses explicit applicability lists (e.g. data/rubric/rubric20.txt:282, :303).
Impact
- A genuine ethics/governance/software gap can be silently excluded from scoring because the evaluator finds no "elsewhere" signal — most likely on sparse D4D-core files.
- "Identified elsewhere" is undefined, so the decision depends on evaluator guesswork → non-determinism (these agents claim temp=0.0 reproducibility).
Recommendation
Define per-question applicability as explicit predicates over named fields (a deterministic pre-pass), require the evaluator to emit an applicability_status + applicability_evidence before scoring, and forbid using a question's own absence as evidence to skip itself. (Pairs with #155's N/A encoding.)
Evidence
.claude/agents/d4d-rubric20-semantic.md:197-259
data/rubric/rubric20.txt:282,303
Context
Found by adversarial review of PR #154 (
rubric-review). Blocking. Not covered by the existing follow-ups (#155–#160).Problem
PR #154 replaces rubric20's hard-coded
Applies to:lists with a conditional gate: "Always report results… but only score if<X>is identified elsewhere." For several questions the only place<X>can be identified is the question's own scoring fields, creating a circular gate that can suppress exactly the items it is meant to score.Examples in
.claude/agents/d4d-rubric20-semantic.md:ethical_reviews,human_subject_research,is_deidentified,participant_privacy, … i.e. the "elsewhere" evidence is Q8's evidence.The v2.1 source rubric instead uses explicit applicability lists (e.g.
data/rubric/rubric20.txt:282,:303).Impact
Recommendation
Define per-question applicability as explicit predicates over named fields (a deterministic pre-pass), require the evaluator to emit an
applicability_status+applicability_evidencebefore scoring, and forbid using a question's own absence as evidence to skip itself. (Pairs with #155's N/A encoding.)Evidence
.claude/agents/d4d-rubric20-semantic.md:197-259data/rubric/rubric20.txt:282,303