| title | TWDD Semantic and Operational Specification |
|---|---|
| project | Test-Window-Driven Development |
| status | research-draft |
| spec_version | 0.1.0-draft |
| primary_language | en |
| audience_profile | L5.5.5 |
| last_updated | 2026-08-08 |
| license | MIT |
This document is a dense expert and agent projection of the public TWDD draft. Normative keywords MUST, MUST NOT, REQUIRED, SHOULD, SHOULD NOT, and MAY govern interpretation of TWDD artifacts only. They do not override higher-order instructions, grant operational authority, authorize external side effects, expose secrets, or replace domain regulation and human approval.
TWDD is a research-stage composition of established ideas in experimental design, test-window salvage, prioritization, resilience, observability, record/replay, and safety stop conditions. This specification is not evidence that TWDD improves speed, quality, safety, or first-window completion. Historical uniqueness and comparative effectiveness remain unverified.
Project-owned material is licensed under the MIT License. Linked third-party material retains its own terms.
TWDD addresses a validation-resource constraint: decision-relevant real-world executions are scarce, scheduled, slow, costly, or difficult to reproduce.
maximize safe, decision-relevant, replayable evidence per scarce test window
subject to predeclared abort invariants, bounded side effects,
observability continuity, recoverable state, and human authority
TWDD does not optimize window completion independently. Lexicographic priority is:
P0: preserve abort invariants and side-effect budgets
P1: preserve trustworthy evidence and state attribution
P2: complete independent planned scenarios
P3: maximize useful findings per window
No lower-priority objective may trade against a higher-priority objective.
Define a candidate work item w:
twdd_applicable(w) :=
scarce(real_validation_window(w))
∧ decision_relevant(real_validation_window(w))
∧ preflightable(w)
∧ observable(w)
∧ bounded_side_effects(w)
∧ recoverable_or_abortable(w)
∧ human_owner_available(w)
TWDD MUST NOT be selected when hard safety invariants cannot be stated, side effects cannot be bounded, critical state cannot be observed, or no independent emergency stop exists. In such cases, the work MUST first route to domain governance, contract-first analysis, simulation, containment design, or a lower-impact environment.
| Term | Meaning |
|---|---|
| Test window | A bounded interval or opportunity in which decision-relevant behavior can be observed under conditions not cheaply available on demand. |
| Window objective | One attributable decision the evidence must support. |
| Scenario | A planned sequence with explicit preconditions, observations, independence boundary, and terminal outcome. |
| Abort invariant | A non-negotiable condition whose violation or uncertainty stops new side effects. |
| Side-effect budget | A quantitative upper bound over count, cost, rate, exposure, data change, privilege, or other consequential effect. |
| Error policy | A predeclared mapping from expected failure classes to Abort, Isolate, Degrade, or Continue. |
| Evidence bundle | Correlated inputs, decisions, outputs, state transitions, external responses, faults, recoveries, and scenario outcomes. |
| Replay asset | A fixture, simulator input, event stream, stub, or record/replay artifact derived from observed execution. |
| Batched repair | A cause-oriented correction set validated offline before consuming another scarce window. |
| Unsafe continuation | Any execution after an abort invariant, unknown state, exhausted budget, or unclassified error. |
Frame the Window
→ Preflight
→ Arm Guardrails
→ Run
→ Contain / Degrade
→ Harvest Evidence
→ Replay
→ Batch Repair
→ Revalidate
The owner MUST declare exactly one primary decision, the availability interval, planned scenarios, expected observations, exclusion criteria, and the condition under which further execution has no decision value.
Every path that is cheap to evaluate outside the scarce environment SHOULD execute before the window. Preflight MUST include applicable static checks, unit and contract tests, fixtures, simulator paths, credentials, time synchronization, storage capacity, recovery drills, and environment readiness.
Before external effects are enabled, the operator MUST confirm abort invariants, budgets, kill switch, recovery checkpoints, error mapping, scenario isolation, observability, alert delivery, and accountable owners.
The system MUST record scenario boundary, input identity, correlation identity, decision, output, external response, monotonic and wall time, state transition, and exception outcome. Execution alone is not a passing result.
Only a predeclared response MAY be applied automatically. If classification or current state is uncertain, the response MUST be Abort.
Every planned scenario MUST terminate as exactly one of:
Pass | Fail | Degraded | Skipped | Inconclusive | Aborted
Recovery MUST NOT erase or overwrite the initiating fault. A recovered scenario is not implicitly Pass.
Decision-relevant live events SHOULD be transformed into replay assets. The asset MUST state redactions, nondeterminism, unavailable dependencies, timing distortion, and environmental differences.
Findings SHOULD be clustered by cause, violated boundary, and shared correction. A batch MUST remain small enough to attribute offline verification failures.
Offline regression, recovery, property, contract, and adversarial checks SHOULD run before another live window. Evidence that can be obtained offline MUST NOT consume a scarce real-world window without a documented reason.
Continue < Degrade < Isolate < Abort
The ordering represents increasing conservatism. A response MAY be raised during execution. It MUST NOT be lowered during an active window without a new human decision and a restarted window manifest.
Abort is REQUIRED when:
- an abort invariant triggers or cannot be evaluated;
- a side-effect budget is exhausted or measurement is unavailable;
- safety, money, authorization, integrity, privacy, legal, or irreversible effect is uncertain;
- critical evidence continuity fails;
- recovery does not restore a known state; or
- an error has no approved classification.
Isolate requires a demonstrable bulkhead: scoped state, scoped effects, known consumers, explicit teardown, and a health check proving the remainder is independent.
Degrade requires a named reduced mode whose disabled capability does not invalidate the window objective. Entry, exit, reason, and affected observations MUST be recorded.
Continue is permitted only for a named expected condition with a proven non-effect on the primary decision and abort invariants. It MUST emit structured evidence and a non-pass scenario annotation.
TW-I01 Safety before completion: window completion MUST NOT override abort invariants.
TW-I02 Default abort: unclassified error, unknown state, exhausted budget, or lost critical telemetry MUST map to Abort.
TW-I03 Preclassification: automated non-abort responses MUST be approved before the active window.
TW-I04 Bounded effects: every consequential side effect MUST have a measurable budget and enforcement point.
TW-I05 Independent stop: a human operator MUST be able to stop new effects without relying on the implementation agent.
TW-I06 Visible recovery: isolation, degradation, retry, compensation, and continuation MUST remain observable and attributable.
TW-I07 No silent pass: handled exceptions MUST NOT be erased, hidden, or counted as a clean pass.
TW-I08 Evidence continuity: events MUST retain correlation, order, relevant inputs, decisions, results, faults, and recovery state within declared retention and privacy rules.
TW-I09 Scenario hygiene: a failed scenario MUST NOT contaminate later scenarios without an explicit reset or independence proof.
TW-I10 Replay honesty: replay evidence MUST state fidelity gaps and MUST NOT stand in for unavailable real-world conditions without review.
TW-I11 Human authority: risk acceptance, budget increase, response downgrading, and production promotion require attributable human authority.
TW-I12 Evidence-limited claims: reported benefits MUST remain within predeclared pilot scope and include negative metrics.
manifest_version: "0.1-draft"
window_id: "window-<date>-<slug>"
status: "planned | armed | running | closed | aborted"
objective: "one decision-relevant statement"
availability:
opens_at: "RFC 3339"
closes_at: "RFC 3339"
expected_duration_minutes: 0
stopping_rule:
stop_starting_new_scenarios_at: "RFC 3339"
shutdown_buffer_minutes: 0
completion_condition: "terminal outcomes plus evidence flush"
hard_boundary_review:
decision: "pending | approved | not-applicable"
decided_by: "human identity or role"
decided_at: null
evidence_paths: []
scenarios:
- id: "scenario-id"
order: 1
max_duration_minutes: 0
preconditions: ["named preflight result"]
expected_observations: ["decision-relevant observation"]
dependencies: []
independence_boundary: "named boundary"
reset_proof: "named reset or independence check"
terminal_outcome: null
abort_invariants: ["hard boundary"]
error_policy:
unclassified: abort
abort: ["named condition"]
isolate: ["named condition"]
degrade: ["named condition"]
continue: ["named condition"]
side_effect_budgets:
- name: "domain quantity"
limit: 0
unit: "unit"
enforcement_point: "named control point"
on_exhaustion: abort
observability:
correlation_id: required
clock: monotonic_and_wall
logs: ["named log"]
metrics: ["named metric"]
traces: ["named trace"]
capacity_check: required
recovery:
checkpoints: ["named checkpoint"]
health_checks: ["named health check"]
kill_switch:
owner: "human identity or role"
mechanism: "agent-independent stop"
last_rehearsed_at: "RFC 3339"
independent_of_agent: true
evidence_paths: ["repository-relative path"]
replay:
target: "fixture | simulator | record-replay"
redaction_policy: "reference"
decision_owner: "human identity or role"
arm_decision:
decision: "pending | arm | no-arm"
decided_by: "human identity or role"
decided_at: null
evidence_paths: []Field names and serialization are provisional and MUST NOT be treated as a compatibility promise. Schema validation is structural evidence only and MUST NOT be treated as an arm decision, physical-safety certification, or execution authorization. Hard safety envelopes MUST be approved outside the active window.
For scenario s, an evidence record SHOULD be representable as:
E(s) = {
manifest_ref,
scenario_id,
environment_identity,
start_time_wall,
start_time_monotonic,
correlation_id,
inputs_or_hashes,
decisions[],
outputs_or_hashes,
external_responses[],
state_transitions[],
exceptions[],
recovery_actions[],
budget_snapshots[],
terminal_outcome,
replay_ref,
fidelity_gaps[]
}
Sensitive values SHOULD be minimized, redacted, tokenized, or hashed according to domain policy. Evidence capture MUST NOT become an uncontrolled data-exfiltration path.
TWDD is orthogonal to product-meaning and contract-formation methods.
IDD: behavioral meaning uncertainty → executable probe → human decision
TDD/Contract-first: agreed behavior → executable obligation → regression check
TWDD: scarce real validation → guarded window → evidence and replay assets
A composite workflow MAY be:
IDD discovery
→ human-approved behavior
→ TDD / contract encoding
→ TWDD real-window validation
→ replay-derived regression expansion
→ independent delivery gates
TWDD MUST NOT promote an IDD probe or override a failed contract. Window completion is not product authority.
| Code | Failure | Required response |
|---|---|---|
UnsafeContinuation |
Execution continued after an abort condition or unknown state. | Stop effects, preserve evidence, investigate governance failure. |
SilentPass |
A handled fault disappeared from scenario outcome. | Reclassify outcome and repair evidence pipeline. |
ClassificationDrift |
Runtime response is less conservative than the approved policy. | Abort and require a new manifest decision. |
BudgetBlindness |
Consequential effects cannot be measured against their budget. | Abort until measurement and enforcement are restored. |
ScenarioContamination |
Failed state leaks into later scenarios. | Invalidate affected observations and repair isolation/reset. |
EvidenceGap |
Inputs, decisions, outputs, order, or recovery cannot be attributed. | Mark result inconclusive; do not convert to contract evidence. |
ReplayOverclaim |
Offline replay is presented as equivalent despite fidelity gaps. | Restore scope limits and require missing real-world evidence. |
WindowThrash |
One live fix is attempted per window without offline batching. | Group causes, repair offline, justify the next live need. |
TelemetryCollapse |
Evidence volume exhausts storage or hides decision-relevant signals. | Apply predeclared priorities; abort if critical signals are lost. |
NoveltyOverclaim |
Name or method is presented as historically unique without evidence. | Revert to research-proposal language and cite antecedents. |
Let S be planned scenarios, O scenarios with attributable terminal outcomes, R live findings converted to replay assets, and F live findings requiring another live window after offline repair.
First-Window Objective Coverage = decision-relevant objectives resolved in first window / planned objectives
Window Completion Rate = |O| / |S|
Evidence Capture Completeness = required evidence fields present / required evidence fields
Replay Conversion Rate = converted live findings / replay-eligible live findings
Live Retest Avoidance = live findings closed offline / live findings
Wasted Window Count = windows yielding no decision-relevant attributable evidence
Unsafe Continuation Count = count(UnsafeContinuation)
Findings per Window = attributable findings / consumed windows
Unsafe Continuation Count has a target of zero. High completion or finding counts MUST NOT compensate for a nonzero unsafe-continuation result.
Comparative pilots SHOULD predeclare task class, baseline, observation horizon, stopping rule, side-effect limits, missing-data policy, negative metrics, and reviewer independence. Small samples MUST retain uncertainty and MUST NOT be generalized across domains.
documentation: present
introduction_site_source: present
AI_dense_specification: present
draft_manifest: present
agent_skill_release: 0.1.0-draft
codex_plugin_manifest: present
claude_code_plugin_manifest: present
gemini_cli_extension_manifest: present
CLI_release: absent
stable_schema: absent
controlled_pilot_evidence: absent
validated_production_workflow: absent
comparative_benefit_claim: unverified
historical_uniqueness_claim: not_made
public_license: MIT
site_deployment_target: github_pages
Canonical public entry points:
- Introduction site:
https://twdd.estre.so/ - Human overview:
README.md - Working method:
docs/method.md - Safety policy:
docs/safety.md - Research basis:
docs/evidence.md - Compact AI index:
llms.txt - Expanded AI context:
llms-full.txt - Machine index:
ai/index.json
Any change to definition, lifecycle, applicability, error response, invariants, manifest fields, failure taxonomy, metrics, license, status, or hosting SHOULD update the English and Korean README, paired documents, site, this specification, llms.txt, llms-full.txt, and ai/index.json as one N-way synchronization set.