Knowhere API turns authenticated requests into document ingestion, document lifecycle, retrieval, billing, and webhook workflows.
Within this repository, apps/api is the coordination layer between HTTP
adapters and the shared implementations in packages/shared-python/shared.
The authenticated owner of jobs, documents, credits, API keys, and webhooks.
The isolation scope for retrieval-visible data. The default namespace is
default.
The API-side intake and execution handle for a workflow such as file parsing, URL ingestion, or demo source materialization.
The terminal artifact record attached to a Job. It stores delivery metadata, result bundle references, and the revision that publication uses.
The typed result of a Job state-machine transition. It preserves whether the transition succeeded, the target state, previous state when known, attempt count, and rejection reason while keeping older boolean facades available.
A post-transaction side effect planned during terminal Job finalization and run only after the database commit succeeds. Current effects include retrieval cache invalidation and outbound webhook publication.
The workflow that lists a User's Jobs and projects one Job into the public Job Result response shape.
The retrieval-visible knowledge object produced from a Job Result after publication.
The hierarchical navigation node derived from parsed headings and section paths.
The retrieval-visible text, image, or table row attached to a Document Section.
The workflow that creates a Job, accepts a file or URL source, confirms upload state, and starts parsing work.
The API-side handoff that advances an uploaded file Job to pending state and enqueues the worker parse task with the user-aware Celery queue policy.
The worker-side workflow that turns a source file into parsed DataFrame rows, parsed assets, and parser debug artifacts before chunk conversion and result packaging.
The worker-side Document Ingestion coordinator. It prepares task-local source files, bills the workload estimate, invokes Worker Document Parsing, builds the result package, uploads artifacts, and finalizes the Job.
The task-local folder set used by Worker Document Processing: input files, parser output files, and generated result packages. It is temporary worker storage, not a retrieval Namespace or durable document scope.
The stable parser adapter result with an output directory and optional parsed DataFrame.
The ingestion-side parsed content artifact. It validates parser output before chunk conversion and result packaging.
The generated ZIP bundle metadata used by terminal Job finalization, including ZIP path, checksum, statistics, and byte size.
The storage-side module set that turns parsed chunks and parser artifacts into a
Generated Result Package. ZipResultService is the orchestration interface;
resource discovery, schema projection, and physical ZIP writing live in
separate modules.
The worker-side estimate used for billing and processing metadata. It records page count, estimation method, and any fallback reason.
The typed worker-side parse request assembled from Job metadata, parser options, source-file identity, output naming, and storage transform keys.
The Worker Document Parsing module that selects one concrete parser adapter for the source document format while keeping format-specific conversion details out of the stable parser entrypoint.
The Worker Document Parsing module that reuses or creates rendered PDF artifacts for PDF-backed parsing paths, including PPTX-to-PDF fallback handling, image-only PDF rendering, temporary PDF materialization, MinerU handoff, and cleanup.
The Worker Document Parsing module that predicts section levels from Markdown lines, DOCX blocks, TOC context, layout metadata, heuristics, and optional LLM inference.
The policy checks that must pass before a new Job is created: authentication, guest scope, system limits, billing RPM, concurrent job limits, and daily quota.
The route-aware part of Job Admission that enforces guest API key scope and system limits from plain route-admission context built by HTTP dependency adapters.
The quota-aware part of Job Admission that enforces billing RPM, concurrent jobs, and daily quota.
The shared workflow that turns parsed chunks into Documents, Document Sections, Document Chunks, and document graph state.
The Publication module that replaces a single Document revision's Document Sections and Document Chunks from parsed chunk rows.
The query workflow that returns cited evidence from published documents.
The typed retrieval request that owns cache-shaping fields and route policy: scope, filters, data type, channels, ranking options, and agentic toggle.
The agentic Retrieval request passed through planning and step execution. It preserves user scope, filters, channel policy, internal recall, and explicit ranking policy fields for the workflow path.
The per-step projection of a Workflow Run Request. It applies step-level query, top-k, and data-type overrides while preserving the request policy.
An API-owned canonical document shipped with the repository for demo and guest flows.
The repeatable script workflow that validates Demo Source catalog metadata, canonical chunks, citation projection, original file size, and regenerated doc_nav.json output.
The workflow that copies a Demo Source into a user's Namespace as normal Job, Job Result, Document, and Document Chunk records.
The credits purchase, checkout, webhook handling, refund reconciliation, and tier refresh flows.
The auth-time workflow that validates API keys, reads and writes the API-key cache, and schedules best-effort last-used updates.
The user-facing workflow that creates, lists, reads, revokes, and toggles API keys.
The Billing Workflow adapter that creates Stripe payment intents and checkout sessions for credits purchases.
The Billing Workflow adapter that settles successful Stripe checkout and payment-intent events into credits, payment records, and tier refreshes.
The Billing Workflow adapter that verifies Stripe events and reconciles credits, payment records, and refunds.
A guest-tier API key with a restricted route surface.
The user-facing workflow for storing outbound webhook configuration and reading delivery logs.
The verified async callback used to continue background work after external delivery.
The shared URL safety workflow used before Knowhere reaches user-provided or third-party HTTP targets. It validates public HTTP/HTTPS URLs, pins resolved addresses for outbound requests, blocks unsafe redirects, and detects URL file types for Document Ingestion.
The shared Redis-backed runtime state used by background work, rate limits, state-machine progress, distributed locks, and job metadata. It owns Redis key language and Redis retry policy.
The shared Redis-backed token leasing workflow used by provider-specific quota managers such as Ali, iLoveAPI, and MinerU.
apps/api/app/api/v1/routes/*
apps/api/app/api/dependencies/*
These modules translate HTTP requests and dependency context into application workflow calls.
apps/api/app/services/*
These modules coordinate Job Admission, Document Ingestion, document lifecycle, Billing Workflow, Demo Source Materialization, webhook handling, and internal callbacks.
apps/api/app/repositories/*
These modules own database reads and writes for API-side workflows.
packages/shared-python/shared/*
These modules own the lower-level implementations for publication, retrieval,
state machines, storage, Redis-backed metadata, billing primitives, and core
exceptions. Shared Job lifecycle finalization lives under
packages/shared-python/shared/services/jobs/lifecycle/*.
app/api/v1/routes/jobs.pyapp/services/document_ingestion/service.pyapp/services/document_ingestion/creation_service.pyapp/services/document_ingestion/confirmation_service.pyapp/services/document_ingestion/handoff_service.pyapp/services/document_ingestion/scope_service.pyapp/services/document_ingestion/worker_dispatcher.pyapp/repositories/job_repository.py
app/api/v1/routes/jobs.pyapp/services/jobs/read_service.pyapp/services/jobs/result_projection.pyapp/repositories/job_repository.py
app/api/dependencies/auth.pyapp/api/dependencies/current_user.pyapp/api/dependencies/route_admission.pyapp/api/dependencies/job_admission.pyapp/services/auth/*app/services/rate_limit/*
auth.py, current_user.py, and route_admission.py are HTTP dependency
adapters. job_admission.py owns only the route-level billing and system-limit
admission dependencies.
app/api/v1/routes/documents.pyapp/services/documents/lifecycle_service.pyapp/repositories/document_repository.py
app/api/v1/routes/retrieval.pypackages/shared-python/shared/services/retrieval/app_service.pypackages/shared-python/shared/services/retrieval/publication_service.pypackages/shared-python/shared/services/retrieval/publication_content.pypackages/shared-python/shared/services/retrieval/publication_models.pypackages/shared-python/shared/services/retrieval/execution/*packages/shared-python/shared/services/retrieval/search/*packages/shared-python/shared/services/retrieval/hydration/*packages/shared-python/shared/services/retrieval/graph/*packages/shared-python/shared/services/retrieval/stats/*packages/shared-python/shared/services/retrieval/workflow/*packages/shared-python/shared/services/retrieval/agentic/core/*packages/shared-python/shared/services/retrieval/agentic/discovery/*packages/shared-python/shared/services/retrieval/agentic/navigation/*packages/shared-python/shared/services/retrieval/agentic/evidence/*
app/api/v1/routes/demo.pyapp/services/demo/*apps/api/scripts/validate_demo_documents.py
app/api/v1/routes/billing.pyapp/services/billing/*app/repositories/payment_record_repository.py- shared billing modules in
packages/shared-python/shared/services/billing/*
app/api/v1/routes/api_key.pyapp/services/auth/*app/repositories/api_key_repository.py
app/api/v1/routes/webhook.pyapp/api/v1/routes/webhook_secrets.pyapp/services/webhook/*app/repositories/webhook_repository.py
app/api/v1/routes/s3_events.pyapp/services/s3_events/*
The internal workflow that decodes S3-compatible storage events, sanitizes headers, acknowledges malformed or unsafe events, and triggers upload handoff.
app/api/v1/routes/qstash_callbacks.pyapp/services/webhook/qstash_callback_service.py
The route owns QStash HTTP signature verification and HTTP response projection. The workflow owns callback parsing, event status resolution, and webhook log side effects.
shared/services/jobs/lifecycle/service.pyshared/services/jobs/lifecycle/success_finalizer.pyshared/services/jobs/lifecycle/failure_finalizer.pyshared/services/jobs/lifecycle/result_writer.pyshared/services/jobs/lifecycle/publication.pyshared/services/jobs/lifecycle/post_commit_effects.pyshared/services/jobs/lifecycle/webhook_outbox.py
app/services/document_ingestion/service.pyapp/services/document_ingestion/processing_run.pyapp/services/document_ingestion/source_preparation.pyapp/services/document_ingestion/parse_execution.pyapp/services/document_ingestion/success_finalization.pyapp/services/document_ingestion/workspace.pyapp/services/document_ingestion/parse_result_package.pyapp/services/document_ingestion/processing_billing.py
app/services/document_parser/parse_service.pyapp/services/document_parser/orchestration/parse_input.pyapp/services/document_parser/orchestration/parse_session.pyapp/services/document_parser/orchestration/route_parse.pyapp/services/document_parser/orchestration/format_router.pyapp/services/document_parser/orchestration/format_adapters.pyapp/services/document_parser/formats/*app/services/document_parser/providers/*app/services/document_parser/structure/*app/services/document_parser/tables/*app/services/document_parser/assets/*app/services/document_parser/support/*
app/services/document_parser/formats/pdf/rendered_transform.pyapp/services/document_parser/formats/pdf/pptx_rendering.pyapp/services/document_parser/formats/pdf/parser.pyapp/services/document_parser/formats/pptx/parser.py
app/services/document_parser/structure/heading_hierarchy.pyapp/services/document_parser/structure/layout_parser.pyapp/services/document_parser/formats/markdown/parser.pyapp/services/document_parser/formats/docx/parser.py
apps/apicoordinates workflows. Parsing, publication, retrieval internals, storage mechanics, and state-machine implementation mostly live outside the route modules.- Worker Document Parsing exposes
checkerboard_parse_outputas the stable parser entrypoint; parser option shaping, format routing, rendered PDF transforms, typed Parse Output, and heading inference stay behind that entrypoint. - A Job and a Document are not the same thing. Jobs track intake and processing; Documents track retrieval-visible knowledge state.
- Terminal Job finalization should plan post-commit effects with primitive identifiers and run them after the database transaction commits.
- State-machine callers that need diagnostics should consume Job Transition Outcome; boolean state-machine methods remain compatibility facades.
current_job_result_idselects the active revision of a Document.- Namespace is part of the retrieval contract, not a UI-only label.
- Demo Sources should behave like normal Documents after materialization.
- Billing Workflow and Job Admission shape whether work is allowed to start; they are not worker-only concerns.