BE-06 - API-owned work-level incremental distribution jobs

[ja573]

BE-06 - API-owned work-level incremental distribution jobs

Status

PROGRAMME ARCHITECTURE DECISION RECORDED - detailed CRITICAL-risk specification required; implementation unauthorized

This issue records the CTO decision of 2026-08-28 that incremental AutomaticPush distribution must become API-owned durable work rather than depend on a GitHub Actions time-window watermark.

Nothing in this issue creation authorizes source edits, branch creation, commit/push, PR mutation, CI dispatch, migration execution, release, deployment, production configuration, durable-job mutation, provider access or external dissemination.

Task identity

• Programme: Publisher Services and Distribution Configuration
• Parent programme issue: Publisher Services: packages, distribution configuration and controlled rollout #765
• Task ID: BE-06
• Repository: thoth-pub/thoth
• Stage: incremental distribution durability / Crossref reliability
• Risk: CRITICAL
• Workflow: to be fixed by the reviewed implementation/release specification; production-line hotfix isolation is the preferred candidate below
• Upstream released baseline: v1.7.0 / master @ 40e9c06d4ab76217c3ef277dd539d3b5580e2bb8
• Released tree: 8797558959fcf1f418eb33ca080b469e97783cbd
• Current develop at decision recording: 91dd607d674fcb9a75c8df31891eea0355c9ca84
• Current Metrics integration branch: feature/metrics @ b96372a3ae20728abd412a7b974ab1e6731684e4
• Exact implementation base: NONE - not authorized
• Task branch: NONE - not authorized

Incident motivation

The legacy thoth-dissemination Crossref selector uses a fixed 75-minute updatedAtWithRelations lookback. GitHub Actions scheduled-run delays/drops on 2026-08-27 created substantial uncovered intervals, and customer DOI 10.11647/OBP.0542 was not registered as expected. The legacy selector therefore makes scheduler reliability part of correctness.

The incident reconciliation and any recovery deposits are owned separately in thoth-pub/thoth-dissemination; they must not wait for BE-06 source delivery.

Approved programme decision

The following architecture direction is approved at programme level and is the premise for the detailed BE-06 specification:

1. Thoth owns durable outstanding incremental distribution work. A missed/delayed GitHub Actions invocation may increase latency but must not lose distribution work.
2. GitHub Actions becomes a wake-up/execution mechanism only. GitHub workflow history is not the canonical watermark or job ledger.
3. Use Publisher Services' existing durable-job boundary. The released BE-04 distribution_job* model remains the durable authority for job, target, attempt, lease, retry and claim-token state; thoth-dissemination remains the external-write execution engine.
4. Add work-level semantics rather than reinterpret PUBLISHER_BACK_CATALOGUE. The existing back-catalogue kind retains its current publisher-level activation meaning. Incremental distribution is a new Publisher-Services-local job kind/contract.
5. Crossref is the first target for this correction. No other destination becomes work-level durable-job executable by analogy. Each later target still needs its own replay/reconciliation readiness evidence.
6. Do not create a generic cross-programme job abstraction. ADR-0008's Publisher-Services-specific boundary remains intact.
7. API lifecycle evidence is authoritative. Job/attempt creation, claim/start, completion/failure and retry/reconciliation evidence are stored in the API model rather than inferred from GitHub run timing.

Existing released seam

BE-04 v1.7.0 already deliberately leaves room for this work:

• distribution_job.work_id is nullable and documented as reserved for future work-level jobs; PUBLISHER_BACK_CATALOGUE is constrained to work_id IS NULL;
• distribution_job_kind is closed and currently contains only PUBLISHER_BACK_CATALOGUE, but the Rust domain documentation explicitly allows later Publisher Services kinds to reuse the same relation;
• job state already records claimed_at and completed_at;
• attempt state already records started_at, finished_at, result and bounded failure diagnostics;
• claim token, lease, retry and attempt-budget semantics already exist.

BE-06 should extend these programme-local primitives rather than add a second cursor/queue store unless detailed review proves that impossible.

Required detailed design questions

The implementation specification must resolve these before source authorization:

1. Creation trigger / completeness: audit every way that updated_at_with_relations can move and decide how durable work is created without missing relation-only or indirect updates. Do not assume one GraphQL mutation path sees every relevant change.
2. Job kind and target granularity: define the exact new closed kind and whether one work-level job has exactly one target. The preferred safety shape is one work + one executable target so failure/retry semantics do not couple destinations with different replay properties.
3. Deduplication/revision identity: define what happens when the same work changes repeatedly while a job is PENDING, RUNNING or already SUCCEEDED. The current globally unique deduplication_key cannot simply be kind:work:target forever if later updates must create new work.
4. Coalescing versus source revisions: choose a correctness-first rule for rapid edits. Do not introduce a complex observed-delivery subsystem merely to optimize duplicate Crossref redeposits unless required.
5. Eligibility: derive work-level job creation only for currently enabled/eligible AutomaticPush configuration and initially only the explicitly approved Crossref pathway. Unknown/inactive/manual/pull-feed targets fail closed.
6. Claim API compatibility: define how the worker claims the new kind without accidentally widening the existing PUBLISHER_BACK_CATALOGUE pilot.
7. Cancellation/config changes: define what disabling a destination does to PENDING/RUNNING work-level jobs without erasing evidence or assuming an in-flight provider write did not occur.
8. Migration/rollback: specify the additive PostgreSQL enum/constraint/index changes and an exact rollback strategy. Production migration execution remains a separate authorization.
9. Asynchronous Crossref validation: preserve the distinction between synchronous deposit-endpoint receipt and eventual DOI registration. Do not silently redefine provider acceptance as observed registration.
10. Operational reporting: ensure staff/operator surfaces can reconstruct queued/running/succeeded/failed work-level activity without using GitHub as the source of truth.

Metrics and release isolation

Metrics work must not be accidentally released with this customer-impact correction.

Verified at decision recording:

• production/release master remains exactly v1.7.0 at 40e9c06d4ab76217c3ef277dd539d3b5580e2bb8;
• Metrics is isolated on feature/metrics @ b96372a3ae20728abd412a7b974ab1e6731684e4 and its child branches/PRs target that integration branch, not develop;
• no Metrics implementation is currently on master or develop;
• develop is nevertheless 18 commits ahead of master and therefore contains other unreleased state. A blanket develop -> master release for this incident must not be assumed safe merely because Metrics is isolated.

Preferred release candidate

Because this correction is customer-impacting and BE-04 already exists in production v1.7.0, the preferred release shape to review is an isolated production-line hotfix based on the exact production master, followed by exact forward-integration into develop.

That is a candidate release strategy, not active authorization. Before implementation it must receive an exact-base decision that explicitly permits the hotfix exception to the normal develop-based task flow and defines:

1. exact production base SHA;
2. task/hotfix branch and PR target;
3. migration/version/release identity;
4. exact source write budget;
5. CI/container-publication side effects;
6. independent CRITICAL source and migration review;
7. separate merge/release/migration/deployment authorizations;
8. post-release forward integration into develop;
9. no automatic refresh or merge of feature/metrics - any Metrics integration-branch refresh remains owned and authorized by the Metrics programme separately.

If detailed review chooses normal develop delivery instead, release preparation must first prove every master...develop delta intended for that release is production-ready. Do not use this incident as implicit authorization to ship unrelated develop work.

Cross-repository order

The required dependency shape is:

BE-06 approved API contract/source -> backend release deployed initially inert -> thoth-dissemination incremental worker consumes the exact released contract -> dissemination source/release -> separately authorized production activation -> observation/reconciliation -> legacy Crossref schedule retirement

Compatibility must permit the old legacy Crossref flow to remain available until the new API-owned path is independently operationally approved. Do not remove the legacy path merely because BE-06 merges or deploys.

Incident and rollout gates

The current Crossref customer incident is independent of implementation timing:

• reconciliation starts immediately and read-only;
• exact missing DOI/work candidates are reviewed before any recovery write;
• manual recovery deposits require separate authorization;
• Crossref api cutover and the DIS-02A production pilot remain HOLD while the incident blast radius is unresolved;
• broader activation waits for work-level job creation/execution observation and reconciliation.

Current authorization

Authorized by the CTO decision recorded here:

• record this programme architecture decision and create the bounded planning/reconciliation ledger;
• read-only repository/task/incident inspection.

Not authorized:

• detailed specification approval;
• source/new-file edits;
• branch creation;
• commit/push;
• PR creation/mutation;
• manual CI dispatch/rerun;
• migration execution;
• release/version/tag mutation;
• merge;
• deployment;
• production API/configuration mutation;
• durable-job creation/claim/complete/fail/cancel;
• provider reads requiring credentials;
• Crossref submission/recovery;
• THOTH_DISTRIBUTION_JOB_CREATION mutation;
• any Metrics branch mutation.

Next action

Produce a detailed CRITICAL-risk BE-06 specification against released BE-04 and all live work-freshness mutation paths, including the exact deduplication/revision model and the isolated-release strategy. Then obtain a genuinely independent specification review and explicit CTO specification approval before any implementation authorization.