| title | Document processing architecture | ||||||
|---|---|---|---|---|---|---|---|
| summary | Current architecture for durable jobs, immutable inputs, capability workers, artifacts, quotas, and the frontend task center. | ||||||
| status | current | ||||||
| type | architecture | ||||||
| scope | community | ||||||
| audience |
|
||||||
| topics |
|
||||||
| related |
|
||||||
| code_paths |
|
This page describes the implemented Document Processing foundation. Community
owns the generic LaTeX build and PPTX import/export contracts, public API,
finalizers, task center, and worker SDK. Its default distribution enables only
latex.compile.pdf/v1 and ships only the optional TeX Live processor; a
deployment may bind reviewed processors to the generic PPTX operations.
Document Processing handles explicit operations that have a durable outcome:
- build a project into an artifact under a controlled runtime;
- export an immutable project snapshot into another file format;
- import an uploaded file into a validated new Workspace project;
- retain bounded diagnostics and provenance after the initiating tab closes;
- support future automation without coupling it to an editor session.
It does not accelerate live preview, retain resident project compilers, choose between browser and server compilers, or provide an arbitrary remote-command platform.
browser or API client
contextual submit action
|
v
core API: Document Processing
authorize -> reserve -> capture/validate -> enqueue
| |
| +---- PostgreSQL job and attempt state
+---- immutable input ---------- PostgreSQL processing blob
|
| authenticated pull, lease, and transfer tickets
v
worker agent ---- per-job sandbox ---- pinned processor runtime
|
+---- staged artifact ---------- PostgreSQL processing blob
|
v
core API: validate -> finalize -> publish artifact or provision project
|
v
global user task center
The core application may remain a single replica for realtime and Git reasons. Worker replicas are stateless execution capacity and may scale independently.
| Capability | Owner |
|---|---|
| Job, attempt, lease, retry, cancellation, capability status | Document Processing |
| Immutable processing input and result artifact metadata | Document Processing |
| Project authorization and current principal | Access |
| Accepted Yjs update projection | Collaboration |
| Authoritative files, settings, project creation, and replacement | Workspace |
| Distribution allowlist and user-facing operation policy | Distribution |
| Bounded processing blobs and transfer tickets | Document Processing; PostgreSQL in the current implementation |
| Processor implementation and toolchain | Independently deployed worker |
Document Processing calls narrow owner-named facades. It does not import Workspace, Collaboration, or Access persistence modules. A file-import worker returns a candidate bundle; only Workspace may turn that bundle into an authoritative project.
The worker protocol is storage-agnostic, but the current implementation stores bounded input, staging, and artifact bytes in PostgreSQL. Project assets may still come from the Workspace object-storage adapter while Core captures an input bundle. Moving processing blobs to object storage is a future adapter; workers already use transfer tickets and therefore need no protocol change.
External repository jobs remain owned by External Repositories. Similar lease
patterns do not justify a global business-level background_jobs module. A
narrow technical helper may be extracted later only after identical invariants
exist in multiple contexts.
Execution extensibility is narrower than product extensibility. Core knows each operation's input, options, permissions, result schema, and finalizer. A worker advertises an implementation of that known operation.
The registry recognizes these versioned identifiers:
| Operation | Class | Input | Result | Community status |
|---|---|---|---|---|
latex.compile.pdf/v1 |
project build | LaTeX project-bundle/v1 |
PDF and optional log | Implemented and distribution-enabled |
typst.export.pptx/v1 |
project export | typst-project-bundle/v1 |
PPTX and conversion report | Contract and endpoint implemented; disabled by the default distribution; no Community processor yet |
pptx.import.typst/v1 |
file import | pptx-input/v1 |
Typst Workspace bundle and report | Contract and endpoint implemented; disabled by the default distribution; no Community processor yet |
Reserving a public operation does not enable it. Distribution policy first allows an operation; a worker identity in the deployment TOML then enables it; live compatible sessions determine whether that enabled operation is waiting or available. Operations absent from deployment configuration are omitted from the public capability list. A new semantic operation requires a Community protocol change; a new implementation of an existing operation does not.
Options are operation-specific structures, not an unrestricted JSON object. The operation version changes when input, result, option, or finalization semantics change incompatibly.
Processors are bounded transformations from an immutable input to a candidate result. They do not receive application or provider credentials and cannot publish project mutations themselves. Any future operation with an external or authoritative side effect requires an idempotent Core-owned finalizer rather than granting that side effect to processor code.
ProcessingJob is the user-visible aggregate. ProcessingAttempt records each
execution lease separately so retries do not overwrite operational evidence.
ProcessingJob
id, operation, version, requester, optional project
input, normalized options, state, phase
attempt count and retry policy
result artifact/project, public failure, timestamps, expiry
|
+-- ProcessingAttempt
attempt number, claim id, worker session
processor contract, lease, outcome, timestamps
|
+-- ProcessingArtifact
role, media type, filename, size, digest, blob key
The forward migration adds dedicated processing tables after the immutable Community baseline; it does not edit or repurpose the published baseline. Existing browser-uploaded PDF artifacts remain Workspace-owned; a processing artifact becomes a Workspace artifact only through an explicit future facade.
Submission checks an operation-specific permission before reserving input or capacity. Job list, detail, diagnostics, cancellation, retry, and artifact download each authorize the current principal; possession of a job identifier is never sufficient.
Project-derived jobs remain subject to current project access. If a requester's access is revoked, task queries may retain only the minimum audit-safe status allowed by policy and artifact download stops immediately. Any download URL is short-lived and minted only after that check. Import jobs are requester-owned until their result is provisioned through Workspace. Worker identities have no user-facing job-list permission and learn no principal identity beyond fields strictly required by an operation contract.
The public lifecycle is:
preparing -> queued -> running -> finalizing -> succeeded
failed is reachable from preparation, execution, or finalization.
cancelled is reachable only before finalization, and expired is for
abandoned preparation or queue entries under retention policy.
preparingreserves idempotency and quota while the API projects accepted collaboration updates, captures a project, or validates an uploaded file.queuedmeans the immutable input is complete and the job may be claimed.runningmeans exactly one unexpired claim fences the current attempt.finalizingmeans a worker has delivered staged output and Core owns validation and publication. Worker lease loss cannot abandon this phase, and the job is no longer cancellable after Core accepts delivery. A separate Core-owned finalization fence prevents concurrent recovery attempts.- terminal state is committed only after result publication or project provisioning is durable.
Phases are a small closed vocabulary such as capturing_input,
waiting_for_worker, processing, uploading_result, validating_result, and
publishing_result. A processor must not report invented percentages when its
toolchain has no trustworthy measure of progress.
Creation commands are semantic endpoints while queries use one common job resource. The current public surface is:
POST /v1/projects/{project_id}/builds
POST /v1/projects/{project_id}/exports/pptx
POST /v1/imports/pptx?filename=...&input_profile=...
GET /v1/processing/capabilities
GET /v1/projects/{project_id}/processing/capabilities
GET /v1/processing/jobs
GET /v1/processing/jobs/{job_id}
POST /v1/processing/jobs/{job_id}/cancel
GET /v1/processing/jobs/{job_id}/artifacts/{artifact_id}
PPTX import accepts the presentation media type as a bounded raw request body and may select a distribution-declared input profile. The profile values and labels belong to the distribution, not Core. PPTX export has no request body or user-selectable mode. These remain semantic commands, not a generic operation-submission route. An explicit retry command, if added, must create a new linked job rather than reopen a terminal job.
Creation accepts an Idempotency-Key. Its uniqueness scope includes the actor
and command endpoint. A concurrent duplicate returns the original job rather
than constructing another snapshot. The reservation stores a normalized
command digest; reusing a key with different operation, project, input, or
options returns a conflict. Keys expire under deployment policy and are not
permanent business identifiers.
The current PPTX import path buffers within the deployment request and processing-input limits, validates the OPC package before reservation, and stores the immutable bytes with their digest before queueing. Successful finalization atomically publishes a new project; it never mutates an existing project. A streaming upload path is not currently implemented.
The API, not the browser or worker, authors project input:
- authorize the operation against the current principal and project;
- reserve idempotency and admission capacity;
- enter a short project-mutation fence shared by accepted Collaboration updates and Workspace mutations;
- in one capture transaction, ask Collaboration to project every update accepted before the fence and ask Workspace to freeze the generation, tree, settings, documents, asset digests, and retention pins;
- commit the snapshot descriptor and release the fence so editing continues;
- resolve the pinned asset bytes through the owning storage facade;
- build a deterministic
project-bundle/v1archive; - persist its digest and immutable blob before making the job claimable.
The capture process has one documented lock order and never performs blob or compiler I/O while holding the project fence. A failed preparation releases its pins through idempotent cleanup. This gives a job one coherent accepted point without holding the editor still for the duration of a build.
The manifest includes schema, project type, entry path, operation-relevant
settings, Workspace version, content generation, and sorted file records with
kind, length, and SHA-256. It contains only project-owned documents and assets;
compiler packages and other runtime dependencies belong to the processor
contract and its package store. The input_digest covers the canonical
manifest and all project content. It is distinct from the browser's local
CompileWorld and from a user-created named snapshot.
Job identifiers, queue timestamps, and other volatile submission metadata do not enter the content digest. If a processor consumes a source epoch, that stable value is part of the manifest and digest; it is never derived from the worker start time.
An import result uses workspace-bundle/v1. The bundle may suggest a project
name and entry path, but Core revalidates path uniqueness, file count, expanded
size, media types, project type, and entry existence before Workspace provisions
a project. A worker never receives project-write credentials.
PostgreSQL is the durable queue. Claim selection uses row locking with
SKIP LOCKED, but scheduling policy belongs above the SQL primitive.
Claim, queued cancellation, and queue expiry all serialize on the same job row:
whichever commits first determines whether a running attempt exists.
Current deployment-local limits cover:
- queued jobs globally;
- active jobs per requester and per project;
- input bundle, diagnostic, and artifact sizes;
- attempts, wall time, queue wait, and retention;
- worker slots per capability.
FIFO creation time is the baseline among jobs eligible for a compatible processor, with per-requester and per-project active limits preventing one account or project from occupying every slot. Admission is not partitioned by operation; a deployment should not combine materially different resource classes behind one quota without first adding an explicit partition. Priority is not user-controlled.
Capacity has no replicated free-slot field. Session registration owns each processor's slot ceiling, Core owns the count of unexpired running claims, and the agent owns its local semaphore. An acquire request carries a short-lived offer only after the agent has local permits. Core intersects that offer with the registered ceiling and active claims in the same transaction that creates the next attempt. Session heartbeat reports liveness and processor health, not capacity.
Operation exposure has four independent inputs:
- Core recognizes the operation and schema version.
- The selected distribution allows the operation.
- Deployment configuration expects a scoped worker identity for it.
- A compatible worker session is healthy and has or can regain capacity.
Only operations that satisfy the first three inputs appear in the public capability projection. Their live state is:
| State | Meaning |
|---|---|
available |
The operation is allowed and at least one compatible worker session is healthy; jobs may still queue behind occupied slots |
waiting |
The operation is configured and admission still permits bounded work, but compatible capacity is temporarily offline |
An absent operation is not enabled by this deployment. It is not represented
as a third unavailable worker-health state.
Project capability queries filter the same projection by project type. The query is guidance rather than authorization; submission checks the project type again and captures a fresh coherent snapshot.
A waiting operation may accept a bounded, expiring job while workers are temporarily offline. The frontend presents that condition before submission rather than implying that execution has started.
Queue exhaustion returns an explicit rate or service error. A local browser preview or export remains usable, but the server does not report that separate action as successful completion of the requested durable job.
Every claim creates a random claim_id and a new attempt row. Heartbeat, phase,
artifact staging, failure, release, and completion all compare that identifier
with the current unexpired database claim. Server/database time is authoritative
for lease decisions.
At-least-once execution is safe because publication is claim-fenced:
- an expired worker may finish locally but cannot create a visible artifact;
- retry creates a new claim rather than extending an ambiguous old attempt;
- staged blobs are private until Core finalizes them;
- orphan staging data is removed asynchronously;
- an infrastructure attempt may use another compatible processor contract, and the winning contract is recorded on the result.
Cancellation and worker-delivery acceptance serialize on the job row. If
cancellation commits first, completion is rejected and staged output remains
unpublished. If delivery acceptance commits first, the job enters the
non-cancellable finalizing phase; a later cancel returns that current state
and Core finishes or recovers publication. Queued work cancels immediately;
running work sets a request observed by heartbeat and the sandbox cancellation
channel. Lease expiry is the final fence when a process does not cooperate.
The worker claim closes when delivery is accepted. Finalization uses a separate
random Core ownership token and lease because blob validation cannot hold a
database row lock. Every publication step compares that token, and terminal
commit is unique per job. Import provisioning creates the Workspace project
and records result_project_id in the same transaction; a crash therefore
commits both or neither, and recovery cannot create a second visible project.
Workers return a closed failure classification; Core owns retry policy.
| Class | Example | Initial policy |
|---|---|---|
| invalid input | unsafe bundle, missing entry | terminal |
| processor rejected | LaTeX compile error | terminal with bounded diagnostic |
| unsupported dependency | package absent from runtime | terminal |
| resource limit | output too large, deterministic timeout | terminal |
| transient infrastructure | transfer or temporary storage outage | bounded backoff |
| worker interrupted | pod or sandbox loss | bounded retry after lease expiry |
| internal contract violation | invalid processor output | terminal and operator alert |
Retries never change an invalid source into success by selecting an unrelated runtime. The current API performs only policy-owned retries for retryable attempt failures; it does not expose a manual retry command. A future manual retry must create a new linked job rather than reopening a terminal aggregate.
Workers upload only to claim-scoped staging roles with declared media types and limits. Core streams or delegates an expiring upload, verifies size and digest, validates the operation result, rechecks the current project and requester authorization, and then atomically records visible artifact metadata. A deleted project or revoked grant cannot turn staged bytes into a downloadable result.
An operation may declare deterministic artifact reuse. The cache identity is:
reuse scope
+ operation and schema version
+ input digest
+ processor contract
+ normalized options digest
Initially, project operations reuse only within the same source project and file imports reuse only for the same requester. Cross-tenant semantic cache hits are disabled because their latency can disclose that another principal submitted matching content. Physical content-addressed blob deduplication may remain a storage detail, but it never grants visibility or skips authorization.
A cache hit creates a new authorized job and artifact reference. It does not reuse another job's visibility. Mutable compiler work directories are not part of this cache. Blob reference tracking and retention prevent one job's deletion from removing content still used by another result.
The worker agent is operator-controlled and may authenticate to the internal protocol only for configured operations. The untrusted processor sandbox sees an input directory, output directory, normalized request, resource limits, and cancellation signal. It must not see the worker token, database URL, object-storage credentials, user cookies, or general project API.
Production execution uses a per-job OS isolation boundary with a read-only root, non-root identity, bounded CPU, memory, processes, temporary storage, and time, and no network unless an operation contract explicitly requires a restricted destination. A local subprocess executor may exist for trusted development but is not a multi-tenant security profile.
Submission remains contextual:
- project build actions live with build/export controls;
- project export actions live in the export menu;
- file import begins from project creation/import surfaces.
State is centralized in a global task center opened from the application header. Desktop uses a right-side drawer; small screens use a full-screen route or sheet. The badge counts active jobs and surfaces failures completed since this user last opened the center. That browser-local acknowledgement only clears the indicator; it does not remove durable job history or artifacts.
“Global” is a presentation location, not a global Job bounded context. The first view reads Document Processing summaries. Other contexts may later contribute namespaced task summaries and contextual actions through a composed read model while retaining their own aggregates, permissions, and retry rules.
TanStack Query owns job lists, details, mutations, and adaptive polling. Polling runs while a nonterminal job is relevant and stops at terminal state. Upload orchestration may use a feature-scoped actor, but server job state does not enter the Workspace session, compilation actor, preview reducer, or Yjs state. A full job-history page is deferred until volume or automation makes the drawer insufficient.
Logs and metrics correlate job, attempt, operation, worker session, claim, and processor contract without including source content or service credentials. Required metrics include queue age/depth, active attempts, lease loss, duration, failure class, cache hits, staged bytes, and artifact cleanup failures.
Reapers handle abandoned preparation, expired queue entries, expired leases, unreferenced uploads, orphan staged results, and retention. Cleanup is idempotent and uses the existing object-deletion discipline rather than deleting blobs inside a user request transaction.
Repository CI currently proves:
- database-backed worker-request replay, concurrent duplicate fencing, retry
after transactional
5xx, and changed-payload conflicts; - an end-to-end fake-worker flow against a freshly migrated Core, including submission idempotency, offline/online capability state, claim and ticket replay, transfer integrity, duplicate upload, finalization, artifact download, exact cache reuse without an attempt, cancellation, and session drain;
- deterministic project-bundle construction and SDK verification of the exact Core manifest shape, file kinds, paths, sizes, and digests;
- frontend account-scoped query keys, cancellation rules, adaptive polling, semantic build submission, and fresh idempotency keys;
- worker formatting, linting, unit tests, processor-contract drift, and both generated OpenAPI documents.
Release validation additionally runs the real worker image for pdfLaTeX and XeLaTeX in the deployment sandbox. Broader race injection, lease-expiry fault tests, archive-bomb fixtures, and a compatibility corpus remain hardening work; this page does not claim those are already automated.
The implementation keeps these explicit boundaries:
backend/src/document_processing/ owning context and public HTTP edge
protocol/ generated public API plus separate worker contract
web/src/pages/processing/ task center and contextual actions
workers/processing-sdk/ protocol client and lease guard
workers/latex/ Community processor and image
Domain behavior remains in its owning context; distribution configuration and generic command runners must not replace it.