MO-000: Add retrospective architecture decision records

docs/adr/0001-keep-presentation-and-organisation-data-at-service-boundaries.md

@@ -0,0 +1,27 @@
+# ADR-0001: Keep presentation and organisation data at service boundaries
+
+**Status:** Accepted (retrospective)
+**Confidence:** High
+**Source PRs:** [#3](https://github.com/DEFRA/waste-obligations/pull/3), [#9](https://github.com/DEFRA/waste-obligations/pull/9), [#43](https://github.com/DEFRA/waste-obligations/pull/43)
+
+## Decision
+
+The obligations API owns an obligations contract, not an organisation or presentation contract. It may use Waste Organisations to validate an organisation exists, and PRN Common Backend to obtain obligation data, but returns only obligations in `GET /organisations/{organisationId}/obligations`.
+
+Translation and presentation decisions remain with the frontend. Integration-specific models are adapted to public DTOs at the endpoint/service boundary rather than being returned directly.
+
+## Current definition
+
+The organisation ID in the route scopes the request; it is not a request to embed an organisation view. The API maps PRN Common Backend fields into its own obligation DTO, including normalising missing downstream tonnages to zero where the public contract requires a non-null number.
+
+## Consequences
+
+- The API avoids duplicating frontend business-country/presentation logic and avoids coupling its response to the Waste Organisations shape.
+- Consumers needing organisation detail obtain or derive it through the appropriate organisation boundary.
+- Downstream model changes require an adapter review rather than silently changing the public contract.
+
+## Evidence
+
+PR #3 introduced an optional organisation inclusion. PR #9 introduced Waste Organisations enrichment. PR #43 explicitly removed organisation data from the obligations response because it increased service coupling and returned presentation text that the frontend would need to remap. `ReadObligations` now uses Waste Organisations only to decide whether to return `404`, then maps only the obligations collection.
+
+`AGENTS.md`, added later, codifies the same boundary rule: integration clients return integration response models and the consuming endpoint or service maps them to public DTOs. See [ADR-0009](0009-adapt-prn-common-backend-behind-an-organisation-api.md) for its PRN-specific application.

docs/adr/0002-persist-compliance-declarations-in-mongo.md

@@ -0,0 +1,25 @@
+# ADR-0002: Persist compliance declarations in MongoDB
+
+**Status:** Accepted (retrospective)
+**Confidence:** High
+**Source PRs:** [#12](https://github.com/DEFRA/waste-obligations/pull/12), [#18](https://github.com/DEFRA/waste-obligations/pull/18), [#20](https://github.com/DEFRA/waste-obligations/pull/20), [#25](https://github.com/DEFRA/waste-obligations/pull/25), [#35](https://github.com/DEFRA/waste-obligations/pull/35), [#39](https://github.com/DEFRA/waste-obligations/pull/39), [#56](https://github.com/DEFRA/waste-obligations/pull/56)
+
+## Decision
+
+Compliance declarations are persisted as MongoDB documents. Their internal identifier is a Mongo `ObjectId`, their stored timestamps are UTC, and their API timestamp representation is `DateTimeOffset`.
+
+The identifier choice supports stable ordering for paged results. Query patterns are backed by explicit Mongo indexes rather than relying on in-memory ordering.
+
+## Current definition
+
+`ComplianceDeclaration.Id` is an `ObjectId`, exposed as its hexadecimal string form. `Created` and `Updated` are UTC BSON dates; values are generated without microsecond precision so they round-trip consistently through Mongo. The document has indexes/migrations for organisation/year reads and the search paths introduced later.
+
+## Consequences
+
+- Existing clients must treat a declaration ID as an opaque string, not a UUID.
+- Stable paging includes an identifier tie-breaker where appropriate.
+- Persistence-related changes are schema changes when they alter BSON or analytics payload shape; [ADR-0006](0006-version-persisted-contracts-and-migrate-forward.md) governs that evolution.
+
+## Evidence
+
+PR #18 added Mongo persistence following the creation endpoint scaffold. PR #20 chose `DateTimeOffset` at the API boundary while retaining UTC Mongo storage. PR #25 introduced the organisation/year index. PR #56 explicitly replaced GUIDs with `ObjectId`s so pages could be ordered stably, and PR #39 records the decision to truncate before save rather than alter values on read. The present `ComplianceDeclaration` entity, Mongo context and read queries retain those choices.

docs/adr/0003-store-submission-snapshots.md

@@ -0,0 +1,26 @@
+# ADR-0003: Store submission snapshots with the declaration
+
+**Status:** Accepted (retrospective)
+**Confidence:** High
+**Source PRs:** [#33](https://github.com/DEFRA/waste-obligations/pull/33), [#70](https://github.com/DEFRA/waste-obligations/pull/70), [#93](https://github.com/DEFRA/waste-obligations/pull/93), [#97](https://github.com/DEFRA/waste-obligations/pull/97)
+
+## Decision
+
+A compliance declaration retains the organisation and user facts submitted with it. Those facts are a snapshot of the declaration context, not a live projection of the current organisation/account record.
+
+## Current definition
+
+The persisted organisation contains the organisation ID and declaration-relevant values, including registration type, regulator details and the appropriate name variants. The declaration also stores a submitter name and the business audit entries retain user ID, email, name and later locale. Declaration text is deliberately not stored.
+
+The API can still read the current organisation for route validation and notification language/recipient resolution. That does not rewrite the declaration snapshot.
+
+## Consequences
+
+- A declaration remains interpretable if account or organisation data subsequently changes.
+- Search can filter on the declaration-time registration type and name/reference fields.
+- Snapshot additions or changed semantics are persisted-contract changes and must follow ADR-0006.
+- Consumers must not assume a declaration's organisation fields are a current account profile.
+
+## Evidence
+
+PR #33 made organisation data part of creation. PR #70 added registration type specifically for recipient selection and search, and made the organisation name optional because it is not applicable to every registration flow. PR #93 removed declaration text from the backend model. PR #97 added the user name expected from the account-facing frontend. The retained entity/DTO mapping and audit model express this history.

docs/adr/0004-model-declaration-lifecycle-and-concurrency.md

@@ -0,0 +1,25 @@
+# ADR-0004: Model declaration lifecycle and concurrency explicitly
+
+**Status:** Accepted (retrospective)
+**Confidence:** High
+**Source PRs:** [#34](https://github.com/DEFRA/waste-obligations/pull/34), [#50](https://github.com/DEFRA/waste-obligations/pull/50), [#59](https://github.com/DEFRA/waste-obligations/pull/59), [#63](https://github.com/DEFRA/waste-obligations/pull/63), [#69](https://github.com/DEFRA/waste-obligations/pull/69), [#154](https://github.com/DEFRA/waste-obligations/pull/154)
+
+## Decision
+
+Compliance declarations have a controlled status lifecycle, embedded business audit history, and optimistic concurrency. The service, rather than the public API client, owns the stored version used for concurrency checks.
+
+## Current definition
+
+Creation submits a declaration, sets version `1`, and appends a `Submitted` audit entry. Valid transitions are `Submitted → Accepted`, `Submitted → Cancelled`, and `Accepted → Cancelled`. A transition appends a timestamped audit entry with the actor and an optional reason.
+
+Update and delete filters include the current internal `Version`; a successful update increments it. A version conflict returns `409 Conflict`, which a client may retry. A requested but invalid state transition returns `422 Unprocessable Entity`, which retrying cannot correct. `IsRegulation43Compliant` and `ObligationStatus` are declaration facts, not alternative lifecycle mechanisms.
+
+## Consequences
+
+- The public request does not carry an entity version, so stale writes are detected after the service reads the current document.
+- `Version` is distinct from `SchemaVersion`; the latter governs payload shape, not concurrency.
+- Status changes create business history and, through ADR-0007, publishable change history.
+
+## Evidence
+
+PR #50 introduced the embedded audit structure. PR #59 introduced PATCH status updates and internal optimistic concurrency. PR #63 explicitly corrected the `409`/`422` distinction. PR #154 added the `Accepted → Cancelled` transition. The entity's `CanTransition` and service update filters are the current executable definition.

docs/adr/0005-page-and-search-compliance-declarations.md

@@ -0,0 +1,25 @@
+# ADR-0005: Page and search compliance declarations with a bounded public contract
+
+**Status:** Accepted (retrospective)
+**Confidence:** High
+**Source PRs:** [#65](https://github.com/DEFRA/waste-obligations/pull/65), [#68](https://github.com/DEFRA/waste-obligations/pull/68), [#119](https://github.com/DEFRA/waste-obligations/pull/119), [#170](https://github.com/DEFRA/waste-obligations/pull/170), [#171](https://github.com/DEFRA/waste-obligations/pull/171)
+
+## Decision
+
+Declaration list endpoints use page-number paging with a total, bounded page sizes, explicit Mongo ordering and validated filter/sort contracts. Regulator search is a bounded, case-insensitive partial match over declaration-time organisation fields.
+
+## Current definition
+
+`page` defaults to `1`; `pageSize` defaults to `20` and is constrained to `1`–`100`. Organisation/year reads order by `updated` descending then ID; sorting happens in Mongo before paging. The regulator route accepts optional obligation year, comma-separated status and registration type filters, `search`, and a priority-ordered `sort` list.
+
+`search` is limited to 100 characters, regex-escaped, case-insensitive, and matches organisation name, compliance-scheme name, scheme-operator name, or reference number. Supported sort fields and directions are explicitly parsed, not passed through to Mongo.
+
+## Consequences
+
+- Page boundaries and totals have database, rather than process-memory, semantics.
+- Query values outside the advertised JSON enum vocabulary are rejected instead of accepting numeric enum values.
+- New public filters/sort fields require a deliberate contract, index and source-data review.
+
+## Evidence
+
+PRs #65 and #68 established the search endpoint. PR #119 moved ordering and paging into the Mongo query to avoid an unstable/incomplete page. PR #170 added the controlled sort grammar; PR #171 added the escaped organisation identifier/name search. `AGENTS.md` from PR #151 corroborates the string-bind, exact-JSON-enum validation pattern used by public query parameters.

docs/adr/0006-version-persisted-contracts-and-migrate-forward.md

@@ -0,0 +1,25 @@
+# ADR-0006: Version persisted contracts and migrate forward
+
+**Status:** Accepted (retrospective)
+**Confidence:** High
+**Source PRs:** [#102](https://github.com/DEFRA/waste-obligations/pull/102), [#134](https://github.com/DEFRA/waste-obligations/pull/134), [#141](https://github.com/DEFRA/waste-obligations/pull/141), [#142](https://github.com/DEFRA/waste-obligations/pull/142), [#149](https://github.com/DEFRA/waste-obligations/pull/149), [#150](https://github.com/DEFRA/waste-obligations/pull/150), [#152](https://github.com/DEFRA/waste-obligations/pull/152)
+
+## Decision
+
+The compliance declaration's persisted BSON and analytics payload are versioned contracts. Each published JSON schema is immutable. Shape changes create a new major/minor schema version and, where necessary, a new idempotent Mongo migration; they do not rewrite historical audit-event snapshots.
+
+## Current definition
+
+`Version` is an optimistic-concurrency counter. `SchemaVersion` identifies the BSON/analytics shape and is currently `v1.2`. Version `v1.0`, `v1.1` and `v1.2` schema files remain embedded. A stored audit event records the schema version that was current for its immutable `before`/`after` snapshot, allowing later hosts to serialise undispatched historic events with the correct schema.
+
+Compatible additions use a minor version; breaking representation/semantic changes require a major version or an expand/backfill/contract rollout. Migrations act on raw BSON, target precise source versions, preserve already-migrated values, and advance data and schema version together.
+
+## Consequences
+
+- A declaration/entity change is assessed across persistence, DTOs, schemas, audit snapshots, analytics, fixtures, migrations and documentation.
+- Historic audit events are never backfilled merely to make them look current.
+- The schema changelog is part of the contract record.
+
+## Evidence
+
+PR #102 established embedded schema resolution and schema-aware analytics serialisation. PR #134 added locale as `v1.1`; PR #141 added the optional coverage percentage as `v1.2`. PR #142 introduced the explicit schema-change workflow, and PR #149 added the required changelog step. `AGENTS.md` is unusually strong corroboration here: it specifies version classification, immutable schema files, raw-BSON idempotent migrations, retained historical audit events, and compatibility testing.

docs/adr/0007-publish-analytics-through-a-transactional-outbox.md

@@ -0,0 +1,25 @@
+# ADR-0007: Publish analytics through a transactional outbox
+
+**Status:** Accepted (retrospective)
+**Confidence:** High
+**Source PRs:** [#99](https://github.com/DEFRA/waste-obligations/pull/99), [#100](https://github.com/DEFRA/waste-obligations/pull/100), [#102](https://github.com/DEFRA/waste-obligations/pull/102), [#104](https://github.com/DEFRA/waste-obligations/pull/104), [#106](https://github.com/DEFRA/waste-obligations/pull/106), [#113](https://github.com/DEFRA/waste-obligations/pull/113)
+
+## Decision
+
+Record declaration mutations and their publishable analytics events in the same Mongo transaction. Publish the resulting outbox records asynchronously to SNS from a lease-protected background processor.
+
+## Current definition
+
+Create, update and delete write an `AuditEvent` alongside the declaration mutation. The event has a globally increasing sequence, an event ID, entity identity, operation, event type, version, schema version, actor, trace ID and immutable `before`/`after` snapshots. Create, update and delete map respectively to `submission.created`, `submission.amended` and `submission.removed`.
+
+The analytics processor claims the `analytics` process lease, reads oldest undispatched/due failed events, serialises against the event's embedded schema version, then publishes to SNS. It records `Dispatched`, `Failed` or `DeadLettered` per process with attempt and retry metadata. Oversized SNS bodies are gzip/base64 encoded when that fits; delivery is at least once, so consumers must de-duplicate by event ID or sequence.
+
+## Consequences
+
+- A successful API mutation cannot lose its outbox record; failed publication does not roll back the committed declaration.
+- Consumers receive an integration envelope, not an internal Mongo document.
+- Background process coordination and migrations use private Mongo lease collections, not request-path state.
+
+## Evidence
+
+PR #99 explicitly introduced the generic event envelope, transactional recording, asynchronous dispatch, retries, leases and SNS handling. PRs #100, #102 and #104 were merged into that analytics branch before it reached `main`; they refined the dispatcher, schema serialisation, traceability and failure isolation. The event-flow documentation and `AGENTS.md` retain the operational invariants.

docs/adr/0008-send-best-effort-notifications.md

@@ -0,0 +1,25 @@
+# ADR-0008: Send best-effort, recipient-specific notifications
+
+**Status:** Accepted (retrospective)
+**Confidence:** High
+**Source PRs:** [#52](https://github.com/DEFRA/waste-obligations/pull/52), [#98](https://github.com/DEFRA/waste-obligations/pull/98), [#108](https://github.com/DEFRA/waste-obligations/pull/108), [#112](https://github.com/DEFRA/waste-obligations/pull/112), [#144](https://github.com/DEFRA/waste-obligations/pull/144), [#155](https://github.com/DEFRA/waste-obligations/pull/155), [#159](https://github.com/DEFRA/waste-obligations/pull/159), [#178](https://github.com/DEFRA/waste-obligations/pull/178), [#179](https://github.com/DEFRA/waste-obligations/pull/179)
+
+## Decision
+
+Use Gov.uk Notify for submission and cancellation notifications, after the declaration mutation has committed. Notification delivery is best effort: a notification failure is logged and measured but does not undo or fail the declaration state change.
+
+## Current definition
+
+Submission sends one message to the user recorded in the declaration's `Submitted` audit entry. The selected template depends on declaration registration type; language derives from the current organisation business country. Regulator leading/inline values are prepared as notification personalisation rather than being general presentation data.
+
+Cancellation sends only for a recognised cancellation reason. Recipients are resolved from Account Backend as the original submitter (when matched to a complete person record) and the organisation's `Approved Person`; duplicate addresses are removed and the output is deterministically ordered. Missing notification parameters do not block cancellation. No resolvable recipient means no cancellation message, not a failed cancellation.
+
+## Consequences
+
+- Declaration persistence and email delivery have separate reliability semantics.
+- Notification recipient data is current account data, while the declaration remains an immutable submission snapshot.
+- Template, localisation and personalisation changes need end-to-end Notify coverage.
+
+## Evidence
+
+PR #52 introduced Notify submission messages; #98 deliberately narrowed their recipient from all organisation users to the actual submitter. PR #108 selected a compliance-scheme variant and language from business country. Account Backend was removed in #155 when no longer needed, then reintroduced for cancellation in #159; PR #179 corrected the cancellation rule to submitter plus Approved Person. PR #178 removed notification-parameter validation as a precondition for cancellation. The current `EmailService` and `CancellationEmailRecipientResolver` encode these definitions.

docs/adr/0009-adapt-prn-common-backend-behind-an-organisation-api.md

@@ -0,0 +1,25 @@
+# ADR-0009: Adapt PRN Common Backend behind an organisation-scoped API
+
+**Status:** Accepted (retrospective)
+**Confidence:** High
+**Source PRs:** [#5](https://github.com/DEFRA/waste-obligations/pull/5), [#140](https://github.com/DEFRA/waste-obligations/pull/140), [#148](https://github.com/DEFRA/waste-obligations/pull/148), [#151](https://github.com/DEFRA/waste-obligations/pull/151), [#158](https://github.com/DEFRA/waste-obligations/pull/158)
+
+## Decision
+
+Expose PRN data through Waste Obligations' organisation-scoped contract while using PRN Common Backend as the operational source. Keep source response types and source-specific values behind an adapter, and validate that a returned PRN belongs to the route organisation.
+
+## Current definition
+
+Read and list routes first confirm the organisation exists, call PRN Common Backend with `X-EPR-ORGANISATION`, map the source response to public `Prn`/`PrnsPaged`, and return `404` when the returned recipient organisation does not match the route. Required source fields are validated during mapping; blank optional recycling process is normalised to `null`.
+
+The organisation list has bounded page-number paging, a narrow search over PRN number/issuer name, and explicit public-to-source status/sort mappings. It makes one source search call per public page. Detail read is separate from list. A public PRN status update is deliberately singular even though the downstream API accepts a collection: the frontend determines the required individual changes and each call is an isolated downstream transaction.
+
+## Consequences
+
+- Public vocabulary and nullability are controlled by this API rather than source defaults.
+- A source identity mismatch fails closed instead of exposing a PRN to the wrong organisation.
+- New source fields, status/sort options or filters require an explicit adapter/contract mapping.
+
+## Evidence
+
+PR #5 introduced the PRN Common Backend integration. PR #140 established the organisation-scoped detail route and recipient verification. PR #148 documents the singular update choice. PR #151 added the paged search adapter and its extensive source-capability analysis; #158 made `recyclingProcess` optional when the source cannot reliably provide it. `AGENTS.md` changes in #140 and #151 formalise the integration-model mapping and exact public enum-validation patterns.