Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 43 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,50 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

## [Unreleased]

### Fixed
- Experiment resume no longer skips materially different runs. `run_id` was
derived only from model alias, regime, condition and seed, so changing the
case count (or the resolved model, the distributions config, or provider
parameters) produced the same identifier and `skip_completed=True` treated the
new experiment as already completed.

### Added
- `mech_gov.experiment.identity` — deterministic, versioned run identity. A
readable `run_label` is now separate from an `identity_sha256` digest computed
over a canonical, allowlisted payload (case count, regime, condition, seed,
hashed model ID, semantic distributions fingerprint, provider parameters,
package version). Only the full digest decides resume.
- `LLMInterface.resume_identity_parameters()` — optional hook for providers to
declare adapter/deployment identity plus generation-affecting parameters.
The default returns `None`, so existing custom providers remain
API-compatible and are treated as non-resumable. Implemented for the bundled
mock, OpenAI-compatible and Bedrock providers. SageMaker remains
non-resumable because endpoint name and region do not establish an
account-scoped deployment identity.
- Results now include `run_label`, `identity_schema`, `identity_sha256` and
`identity` alongside the existing fields.
- Stable identity warning codes `MGI001`–`MGI008` on the
`mech_gov.experiment.identity` logger. Diagnostics name the fields that
differ, use a pseudonymous reference for correlation, distinguish unreadable
completion state, and never print raw labels, paths, or parameter values.
- README section documenting resume semantics, non-resumable runs and legacy
result files.

### Changed
- `run_id` keeps the previous readable format for backward compatibility.
`identity_sha256`, not `run_id`, decides whether a run may be skipped.
- Runs whose identity cannot be established (no stable model ID, undeclared
provider/deployment parameters, unreadable completion state, missing package
version) always execute rather than risking a wrong skip.
- Legacy records without identity metadata are read, never rewritten, and never
treated as equivalent to a versioned identity.
- The distributions configuration is loaded before the completion check, since
its content participates in run identity.
- OpenAI-compatible endpoint targets and Bedrock regions are represented only
by SHA-256 fingerprints. Raw endpoint URLs, credentials and private paths are
excluded. These fingerprints provide stable pseudonymous identity, not
confidentiality.

- Open-source readiness scaffolding:
- Apache 2.0 `NOTICE`, expanded `CONTRIBUTING.md` (CLA + issue/PR flow),
`CODE_OF_CONDUCT.md`, `SECURITY.md`, `CODEOWNERS`
Expand Down
109 changes: 109 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -138,6 +138,115 @@ mech_gov_framework/
└── tests/ # offline tests (mock provider)
```

## Experiment resume and run identity

`ExperimentRunner` is resumable: re-running an experiment skips work that is
already on disk. Each result therefore carries two different things.

| Field | Purpose |
|---|---|
| `run_label` | Readable name, e.g. `mock-R2-S0-seed42`. Never decides resume. |
| `identity_sha256` | Full SHA-256 over the canonical identity payload. **Decides resume.** |
| `identity` | The canonical payload itself, so the digest is reproducible. |
| `identity_schema` | Version of the canonical identity contract. |
| `run_id` | The previous readable identifier, unchanged for compatibility. |

A run is skipped **only** when a stored record carries the exact same verified
`identity_sha256`. The identity covers the case count, regime, condition, seed,
a digest of the resolved model ID, a digest of the *loaded* distributions
configuration, the provider's declared adapter/deployment fingerprint and
generation parameters, and the package version. Changing any of them re-runs
the experiment instead of silently reusing an older result.

`run_label` is present on every new result. `identity_schema`,
`identity_sha256`, and `identity` are present only when all required identity
producers are available; a non-resumable result retains the legacy `run_id`
without claiming a machine identity.

The distributions fingerprint is taken from the parsed configuration, not the
file bytes, so reformatting or editing comments does not invalidate previous
runs, while a changed parameter does. Equivalent content at a different path
fingerprints identically.

Canonical numbers are type-tagged and derived exactly from integers or the
round-trip representation of finite floats. Decimal normalization uses explicit
precision equal to the operand's digit count, so a caller's mutable
`decimal` context cannot round two configurations into one identity.
Non-finite floats are rejected.

### Non-resumable runs

Some runs cannot prove equivalence. They always execute, and log why:

- the provider cannot expose a stable, non-empty `model_id`;
- the provider cannot declare its adapter/deployment identity and generation
parameters (see below);
- the package version is unavailable.

Failing open is deliberate: recomputing a run costs time, whereas wrongly
skipping one silently corrupts a result set.

Code provenance is bounded to the package version. Source edits made without a
version change are intentionally outside this identity contract; use
`skip_completed=False` while iterating on an editable checkout.

### Declaring provider parameters

`LLMInterface.resume_identity_parameters()` returns an allowlisted adapter
identifier, privacy-safe deployment fingerprint, and generation-affecting
values, or `None`. The base implementation returns `None`, so existing custom
providers remain API-compatible but always execute until they declare a stable
identity. The bundled callable and SageMaker providers are non-resumable:
arbitrary callables are opaque, and an endpoint name plus region does not prove
which account-scoped SageMaker deployment is selected.

The bundled OpenAI-compatible and Bedrock adapters currently expose exactly one
constructor-configured generation parameter, `top_p`; both declare it alongside
their privacy-safe target fingerprint. Per-call `temperature` and `max_tokens`
come from governance-regime code and are therefore covered by the package
version. Connection, credential/profile, timeout, and retry settings affect
transport behavior rather than the successful response contract and are
deliberately excluded.

Return only public values and privacy-safe fingerprints — never an API key,
credential, raw private URL, profile name, endpoint, timeout, or path.

```python
class MyProvider(LLMInterface):
def resume_identity_parameters(self):
return {
"provider": "my_provider",
"deployment_sha256": self._safe_deployment_fingerprint,
"top_p": self._top_p,
}
```

SHA-256 fingerprints prevent raw identifiers from appearing in the identity
payload; they are pseudonymous identifiers, not encryption. Use non-secret,
stable model and deployment identifiers. For backward compatibility, the
existing raw `model_id` fields in results and manifests are unchanged.
The readable model alias remains in `run_id`, `run_label`, and `identity` as
public metadata; do not put credentials or private identifiers in aliases.

### Legacy result files

Records written before versioned identity carry a `run_id` and nothing else.
They are never treated as equivalent to a versioned identity: a matching
readable label logs `MGI001` and the run executes again. Existing files are read
but never rewritten or deleted. The first post-upgrade sweep can therefore
repeat legacy provider calls and incur their normal cost.

Identity and resume warnings use the `mech_gov.experiment.identity` logger with
stable codes `MGI001`–`MGI008`. They name the configuration fields that differ
and include a 12-hex SHA-256 reference for correlation, but never print the raw
run label, result path, or configuration values. `MGI008` distinguishes
unreadable completion state from a stored payload that failed validation.

Distribution configuration is loaded before any resume decision because its
parsed content is part of identity. A missing or invalid distributions file
therefore raises even when an older readable label exists; silently skipping
without validating an identity input would be unsafe.

## CLI

```bash
Expand Down
Loading
Loading