Citation Machine

Citation Machine is a small, provenance-preserving client built on NEXUS. It uses NEXUS's public CitationIngestionPort and CitationSearchPort; document parsing, persistence, lexical retrieval, and provenance validation remain NEXUS responsibilities.

Citation Machine v0.3 supports PDF, EPUB, Markdown, and plain-text ingestion, lexical and independent semantic retrieval, support/contradiction inference, exact evidence verification, and MLA, APA, Chicago, IEEE, and BibTeX formatting. It is a Python library and local CLI.

Development setup

Keep this repository next to the NEXUS checkout, then install both editable projects:

python3 -m venv .venv
source .venv/bin/activate
python3 -m pip install -r requirements-dev.txt

requirements-dev.txt installs ../Nexus[citation] as the local development dependency and Citation Machine's dev extra for tests and linting.

Python API

from citation_machine import CitationMachine
from nexus_knowledge.ingestion import CitationIngestionService
from nexus_knowledge.persistence import InMemoryKnowledgeRepository
from nexus_knowledge.retrieval import CitationLexicalSearch

repository = InMemoryKnowledgeRepository()
machine = CitationMachine(
    ingestion_port=CitationIngestionService(repository),
    search_port=CitationLexicalSearch(repository),
)

document_id = machine.ingest("paper.pdf")
results = machine.search("known passage", filters={"document_id": document_id})

Every result contains the exact segment text, source and document identity, page/chapter/section locators when available, line and character offsets, source metadata, segment metadata, and the NEXUS retrieval score.

Durable evidence identity

The additive evidence-identity kernel gives an exact citation a permanent identity independent of search queries, ranking, embeddings, timestamps, provider identifiers, and NEXUS's compatibility IDs. It records four immutable, full-SHA-256 layers:

1. a logical source artifact identified by its resolved reference and source kind;
2. a source version bound to the SHA-256 of the exact ingested bytes;
3. canonical UTF-8 content bound to a named, versioned, ordered transformation profile; and
4. a non-empty half-open character span bound to its exact reconstructed text.

from citation_machine.backend import LocalCitationMachine

machine = LocalCitationMachine.open("citations.json")
machine.ingest("paper.pdf")
result = machine.search("known passage")[0]

identity = machine.identity(result)
assert machine.reconstruct(identity.evidence_id) == result.text

Canonicalization is explicit and auditable through identity.canonical_content.profile.steps. Compatibility fields such as NEXUS source/document/chunk IDs and page, chapter, section, and line locators remain attached to the identity record for rendering, but do not participate in its authoritative hash.

Local persistence uses a Citation Machine-owned <store>.identity.json sidecar. The sidecar stores the canonical content needed for exact reconstruction and is SHA-256-bound to the corresponding NEXUS snapshot. Missing or stale sidecars are deterministically rebuilt from NEXUS's persisted canonical documents; malformed sidecars fail closed. The NEXUS snapshot schema is not modified.

Provider-neutral acquisition

CitationAcquisitionAdapter is an additive boundary for discovery systems. An adapter receives a generic DiscoveryCandidate and returns complete CapturedSource bytes. Citation Machine stores those bytes under a content-addressed local path and passes that path through the unchanged NEXUS ingestion port.

from citation_machine import (
    CandidateHighlight,
    CapturedSource,
    CitationAcquisitionService,
    DiscoveryCandidate,
    DiscoveryMetadata,
)
from citation_machine.backend import LocalCitationMachine

class MyAdapter:
    def capture(self, candidate):
        return CapturedSource(content=b"A complete source with exact evidence.", source_kind="text")

machine = LocalCitationMachine.open("citations.json")
acquisition = CitationAcquisitionService(machine)
candidate = DiscoveryCandidate(
    source_kind="text",
    discovery=DiscoveryMetadata(provider="example", candidate_id="candidate-1", score=0.9),
    highlight=CandidateHighlight(text="exact evidence"),
)
receipt = acquisition.acquire(MyAdapter(), candidate)
assert receipt.evidence is not None
assert machine.reconstruct(receipt.evidence) == "exact evidence"

Provider IDs, URLs, scores, ranks, queries, timestamps, and arbitrary JSON-like provider metadata remain on DiscoveryMetadata and AcquisitionReceipt; they are never sent to NEXUS or included in source-version, canonical-content, or evidence-span identity. A highlight is promoted to evidence only when it matches persisted canonical content exactly. Repeated text requires an exact character offset, and a highlight that crosses provenance segment boundaries fails closed. Once acquired, evidence reconstruction uses local canonical persistence and does not call the adapter.

Exa discovery and source capture

Local configuration and SDK quickstart

This checkout is ready to use with source .venv/bin/activate. Its local virtual environment reuses the Python installation's existing dependencies via --system-site-packages, with Citation Machine installed editable. For an isolated environment on another machine, follow the development setup above.

Install the optional SDK example dependencies after the development setup above:

python3 -m pip install -e '.[exa]'
cp .env.example .env  # only on a fresh checkout; preserve an existing .env
chmod 600 .env

Set EXA_API_KEY in .env. This file is ignored by Git; never put the key in source code. The example loads this project's .env explicitly and preserves existing environment variables:

python3 examples/exa_quickstart.py search "research on durable evidence identity" --limit 5
python3 examples/exa_quickstart.py answer "What is evidence provenance?"

search uses the existing adapter with type="auto" and highlights. answer uses the OpenAI SDK against https://api.exa.ai, model exa, and returns an answer with citations. Only an Exa key is required. These are discovery outputs; they are not verified local evidence. The example does not acquire sources or modify the citation store. Each invocation makes a billable Exa request; automatic SDK retries are disabled.

The library and main CLI read environment variables, without automatically loading .env. For acquisition from this project's terminal, export the settings first:

set -a
source .env
set +a
citation-machine --store citations.json acquire "your evidence query" --limit 5

If a macOS Python installation reports CERTIFICATE_VERIFY_FAILED, configure its trusted CA bundle (for example SSL_CERT_FILE pointing to python3 -m certifi's output). Keep certificate verification enabled.

Provider adapter

Exa networking is opt-in through the provider module; it adds no runtime dependencies:

import os
from citation_machine.exa import ExaAcquisitionAdapter

adapter = ExaAcquisitionAdapter(api_key=os.environ["EXA_API_KEY"])
candidates = adapter.search("research on durable evidence identity", limit=5)

Search returns a tuple of generic DiscoveryCandidate objects in provider order, with one-based ranks. Each candidate selects the first returned highlight as CandidateHighlight without a character offset. Additional highlights, full text, and summaries are discarded, including nested content payloads. Provider IDs, URLs, scores, queries, UTC discovery timestamps, and JSON-like metadata remain solely in DiscoveryMetadata; result and response metadata are separately named. Missing result IDs fall back to the URL. No discovery operation ingests into NEXUS or creates evidence.

Tests and callers can inject ExaAcquisitionAdapter(client=fake). The client must implement search(query, *, num_results, type, contents) and return a JSON-like mapping in Exa's HTTP response shape, such as {"results": [{"id": "result-1", "url": "https://example.org"}]}. SDK response objects must be converted to that shape by the caller's client wrapper. Opaque provider objects are rejected. The core package does not import the Exa module automatically.

The default client uses the Exa search endpoint, a 30-second socket timeout, a 4 MiB response limit, and no redirects or automatic retries. timeout and max_response_bytes are configurable; limit accepts integers from 1 to 100. Client failures and malformed responses raise ExaProviderError without exposing raw payloads or credentials. Malformed results fail the entire search instead of silently dropping rows.

capture(candidate) independently fetches the underlying URL and returns generic CapturedSource bytes. It never calls Exa contents or substitutes search text, summaries, or highlights. Pass a discovered candidate to the existing service:

from citation_machine import CitationAcquisitionService
from citation_machine.backend import LocalCitationMachine

machine = LocalCitationMachine.open("citations.json")
receipt = CitationAcquisitionService(machine).acquire(adapter, candidates[0])

PDF, EPUB, and Markdown URL suffixes supply candidate format hints, checked against the fetched Content-Type. Other URLs default to text. Extensionless binary downloads require an explicitly corrected candidate kind (using dataclasses.replace); format mismatches fail explicitly. Text accepts text/plain, text/markdown, text/html, and application/xhtml+xml. Markdown accepts text/markdown or text/plain; PDF and EPUB require their respective MIME types. Text must be UTF-8 or ASCII; unsupported charsets are rejected without transcoding. All accepted bytes remain exact.

HTML is stored as raw markup in a text artifact, not rendered or stripped. External assets and JavaScript-generated content are not captured. This preserves the HTTP document bytes but does not claim a complete browser-rendered page. Highlight alignment still uses exact persisted text, so markup or entity differences can cause HighlightAlignmentError; no approximate matching occurs.

Capture requires HTTP 200, a supported Content-Type, a non-empty bounded body, and matching Content-Length when present. Partial, encoded, oversized, malformed, and unsupported responses raise CaptureValidationError; network failures raise ExaProviderError. Redirects are rejected and no Exa credentials are sent to source hosts. Timeout and byte limits also apply to capture. PDF/EPUB receive initial format checks; NEXUS retains responsibility for full document parsing.

Inject fetcher= independently of the search client for offline tests. It implements fetch(url, *, timeout, max_response_bytes) and returns a mapping containing integer status, string-to-string headers, and immutable content bytes. The adapter validates these even for injected fetchers. Provider metadata never enters the captured bytes or NEXUS. Existing content-addressing, duplicate acquisition, and offline reconstruction semantics are unchanged.

Semantic and hybrid retrieval

search() remains lexical-only. semantic_search() enumerates all eligible persisted chunks through NEXUS's additive CitationCandidatePort; it does not require a lexical bridge. hybrid_search() combines independent lexical and semantic ranks using reciprocal-rank fusion (constant 60), deduplicates by persisted segment identity, and breaks ties by source, document, segment index, and chunk ID. Results contain .citation and separate .retrieval metadata.

from citation_machine.backend import LocalCitationMachine

machine = LocalCitationMachine.open("citations.json")
document_id = machine.ingest("story.txt")
semantic = machine.semantic_search("sad")  # can retrieve a passage containing “wept”
hybrid = machine.hybrid_search("Ulysses was sad")

InMemorySemanticIndex is replaceable and rebuilt from the filtered eligible candidate snapshot on each request. Indexes receive immutable ID/text projections and return only IDs and scores. Unknown IDs, invalid scores, or mismatched candidates fail closed. No embeddings or index data enter canonical identity or persistence; reconstruction requires neither.

The default LocalEmbeddingMatcher uses feature hashes and a bundled concept lexicon. It supports conceptual matches within that inspectable vocabulary, including grief and crying, but is not a learned, open-domain semantic model. For broader semantics, pass an index using a caller-loaded sentence-transformers encoder:

from citation_machine import InMemorySemanticIndex, SentenceTransformerMatcher

# encoder is a sentence-transformers model loaded and configured by the caller.
index = InMemorySemanticIndex(matcher=SentenceTransformerMatcher(encoder))
results = machine.hybrid_search("your proposition", index=index)

The library never downloads a model automatically. Implement SemanticIndex or SemanticMatcher for another backend. Legacy SemanticCitationSearch uses independent candidates when its client supports enumeration; an older lexical-only custom port retains its documented reranking behavior. The new semantic/hybrid APIs explicitly reject ports without enumeration.

Finding verified evidence

findings = machine.find_evidence("Find evidence that Ulysses was sad", limit=5)
for finding in findings:
    print(finding.text, finding.evidence_ids, finding.relation)
    assert machine.reconstruct(finding.evidence) == finding.text

The pipeline retrieves a configurable candidate pool (default max(50, limit * 10)), applies a pluggable InferenceBackend, and ranks supporting candidates. Every proposed quote and offset is independently matched against the correct canonical source version before identity creation. Punctuation changes, paraphrases, invented text, wrong versions, ambiguous matches without offsets, and cross-chunk spans fail closed. An invalid supporting proposal raises an error; it is never silently repaired or promoted.

LocalInferenceBackend is a conservative offline rule baseline. It handles supported simple facts, negation, and a bounded set of concepts. Questions, hypothetical/reported assertions, unsupported qualifiers, unknown relationships, and ambiguous subjects cause abstention. Unknown headings or standalone fragments also cause abstention; only recognized distress-topic labels are treated as harmless structure. Use a configured NLI backend for broader document context. A bare pronoun such as “He wept” cannot establish that the subject is Ulysses without contextual binding. Real-world entailment requires an appropriate configured inference backend and evaluation in your domain. LearnedNLIBackend accepts a caller-owned premise/hypothesis classifier and an explicit label map; it neither downloads a model nor treats retrieval similarity as proof.

Use inference=backend and index=index on find_evidence() to configure those layers. Relation scores and explanations remain fallible derived assessments; canonical verification proves exact source text and provenance, not the truth of the claim. diagnostics=True also returns contradictions, related, and insufficient candidates with evidence=None. include_context=True includes the original candidate passage. Filters match the lexical API.

A single supporting result holds an EvidenceSpanIdentity. Multiple non-contiguous passages hold an additive EvidenceSet of independent identities; .text and reconstruct() return a tuple of exact texts rather than a fabricated continuous quotation. Each member remains inside its original chunk. Inference evaluates each retrieved chunk independently and batches model calls when supported; cross-document evidence synthesis is not automated.

Bibliography and citation formatting

Bibliographic metadata is explicit, validated presentation data stored separately in <store>.bibliography.json. No author, title, date, publisher, DOI, URL, or page number is inferred from a capture filename or invented when absent.

from citation_machine import BibliographicAuthor, BibliographicDate, BibliographicMetadata

machine.set_bibliography(document_id, BibliographicMetadata(
    kind="book", title="The Voyage", authors=(BibliographicAuthor("Writer", "Jane"),),
    issued=BibliographicDate(2024), publisher="Example Press",
))
print(machine.format_citation(findings[0].evidence, style="mla"))
print(machine.format_citation(findings[0].evidence, style="apa", form="inline"))

Supported kinds are book, paper, webpage, and document. MLA, APA, Chicago, and IEEE use packaged CSL styles and Pandoc’s citation processor (pypandoc-binary); BibTeX uses a safely escaped bibliographic record serializer. Reference forms are supported for all styles. Inline/footnote availability follows the selected style; unsupported combinations raise an explicit error. Formatting a span uses canonical page locators only when available. Reference page ranges must be explicitly supplied bibliographic metadata. Each formatting call handles one item; document-wide numbering and same-author/year disambiguation require a citation processor with the full bibliography context. For EvidenceSet, format each independent member. Metadata corrections and style changes never change evidence IDs.

Exa acquisition and CLI

Acquire complete sources with an explicitly configured adapter; search highlights still undergo exact canonical alignment:

receipts = machine.acquire("your evidence query", adapter=adapter, limit=5)
findings = machine.find_evidence("your proposition")

With no adapter supplied, acquire() uses EXA_API_KEY. Previously captured evidence can be reconstructed offline. Captures complete sequentially; if a later capture or highlight fails, earlier successful captures remain durable. Discovery metadata is returned on receipts, outside identity; pass any verified bibliographic source details separately to set_bibliography().

citation-machine --store citations.json ingest story.txt
citation-machine --store citations.json search "known passage" --limit 5
citation-machine --store citations.json semantic-search "sad"
citation-machine --store citations.json hybrid-search "Ulysses was sad"
citation-machine --store citations.json find-evidence "Ulysses was sad" --context
citation-machine --store citations.json find-evidence "Ulysses was sad" --diagnostics
citation-machine --store citations.json acquire "your evidence query" --limit 5

All commands emit JSON. Retrieval accepts --source-id, --document-id, and --source-kind. Evidence search additionally accepts --candidate-limit. Operational errors go to stderr with exit status 2. Networking occurs only through explicit acquisition.

Optional learned backends (v0.3)

Install the lightweight core as usual, then opt into the model runtime:

python3 -m pip install '.[learned]'

The default retrieval and inference remain deterministic and work without Torch or Transformers. The optional implementations use revision-pinned BGE small English v1.5 (normalized CLS embeddings, with the author's query instruction) and DeBERTa v3 MNLI/FEVER/ANLI (a three-way entailment classifier). BGE keeps local indexing modest; the NLI checkpoint provides an independent support judgment learned across several inference datasets. Both are replaceable.

from citation_machine import (
    LearnedSemanticConfig, LearnedSemanticIndex,
    TransformerNLIConfig, TransformerNLIBackend,
)

# Explicit download permission is needed only to populate the model cache.
index = LearnedSemanticIndex(LearnedSemanticConfig(local_files_only=False))
inference = TransformerNLIBackend(TransformerNLIConfig(local_files_only=False))
findings = machine.find_evidence(
    "Find a passage showing that Elin was grieving",
    index=index, inference=inference, diagnostics=True,
)
# Once cached, use the default local_files_only=True for offline inference.

Every search rebuilds the eligible ID/text projection from persisted chunks; unchanged text embeddings are reused in memory. index.clear() discards those derived caches, and the next search rebuilds them. All chunks are indexed, including overlapping windows for long chunks. No embeddings are persisted in the authoritative store. Importing the package neither loads weights nor downloads anything. Downloads accept safetensors only and never execute model-repository code.

Frozen config objects expose model name, full revision, cache directory, device, batch size, token limit and NLI confidence/margin thresholds. The default NLI operating point is 0.85 confidence and 0.15 margin; these are uncalibrated scores. Entailment maps to SUPPORTS, contradiction to CONTRADICTS, neutral to RELATED, and uncertainty to INSUFFICIENT. This three-way model cannot perfectly distinguish all task-specific RELATED/INSUFFICIENT cases. It proposes the shortest sufficient contiguous sentence interval it can verify; exact canonical matching independently decides whether that text may become evidence. Context and model classification can still be wrong even when the quote is authentic.

By default, NLI abstains on a complete claim/passage exceeding its 512-token context. Explicit TransformerNLIConfig(long_passage_policy="windows") evaluates all overlapping windows, preserves the whole claim, translates proposals to exact source offsets, and abstains on support/contradiction conflicts or exceeded budgets. It cannot guarantee interpretation of dependencies beyond a window. Neither policy silently truncates a hypothesis. The semantic window maximum is a relevance score, not a support verdict. Reuse the backend objects across calls to reuse weights and embedding caches. All complete windows are classified before selecting one to shorten; conflicting windows cause immediate abstention. Optional shortening examines at most max_span_proposals=128 intervals, then retains the already-classified complete interval if no shorter one passes. This budget limits quote minimization, not source coverage. Reported confidence never exceeds either the complete passage/window or the chosen span's confidence.

For a separately formulated query and claim, pass retrieval_query="natural search query" to find_evidence("declarative claim", ...); retrieval receives the query, inference receives the claim. Neither string enters evidence identity. Common “Find a passage showing that …” requests are also normalized directly by the learned NLI backend.

Evidence benchmark and live validation

The checked-in, self-authored fixture has 60 passages across prose, technical writing and factual reports, 24 natural queries (18 answerable and six no-answer), 93 fixed classification pairs and exact gold spans. Gold was independently reviewed and frozen before running models. It includes shared entities, misleading lexical matches, contextual emotion, negation, uncertainty and explicit contradictions. This is a compact engineering regression set, not an external generalization test.

# Fast deterministic comparison, no model downloads or service calls:
citation-machine-evaluate benchmark
# First explicit model download and the complete seven-mode comparison:
citation-machine-evaluate benchmark --learned --allow-download --require-gain \
  --output /tmp/benchmark-report.json
# Subsequent runs can remain completely offline:
HF_HUB_OFFLINE=1 citation-machine-evaluate benchmark --learned --require-gain

Reports include Recall@1/3/5, Hit@k, MRR, fixed-pair accuracy and F1, exact-span success, strict end-to-end success, no-answer abstention, errors, per-domain metrics and paired retrieval gains. Retrieval-only modes have no invented classification or evidence metrics. One correct finding plus an incorrect supporting finding fails the complete case. Errors remain in denominators. The predeclared --require-gain gate requires learned semantic or hybrid Recall@3 to exceed lexical by at least 0.10 absolute, with MRR non-regression. It does not certify general semantic reliability.

# EXA_API_KEY must already be exported. This is deliberately outside the offline test suite.
citation-machine-evaluate live --learned --long-passages windows \
  --store /tmp/citation-live/store.json --output /tmp/live-report.json

The live command searches three RFC topics through the existing Python Exa HTTP adapter, captures complete source bytes independently, ingests through NEXUS, runs hybrid retrieval and inference, verifies exact spans and formats APA/BibTeX citations. It explicitly ignores discovery highlights; model evidence must come from the complete capture. It repeats ingestion of identical captured bytes with changed provider IDs, URLs, scores, ranks, queries, timestamps and metadata, then checks both canonical and evidence identities. No titles, authors or dates are inferred from search metadata. Each case needs verified, formatted supporting evidence to pass; failed captures and abstentions remain in the report. A missing key produces an explicit skip. Live success is an execution and provenance smoke test, not independently labeled evidence-quality evaluation.

Both commands accept --model-cache, --device, --embedding-batch-size, --nli-batch-size, --threshold and --margin. Downloads require --allow-download; validation defaults to local cached weights. Reports and model caches should be kept outside the repository. --max-span-proposals sets the optional quote-minimization budget.

Validation

PYTHONPATH=src:../Nexus/src python3 -m pytest --cov=citation_machine --cov-branch
python3 -m ruff check src tests
cd ../Nexus
PYTHONPATH=src python3 -m pytest tests/test_citation_ingestion.py tests/test_citation_search.py \
  tests/test_citation_candidates.py tests/test_persistence.py tests/test_package_contract.py

The regression fixture in tests/fixtures/semantic_regression.json covers conceptual retrieval, low lexical overlap, negation, contradiction, and abstention. Adversarial tests cover independent canonical verification and metadata/style/index independence from evidence identity.