Skip to content

CLAUDE.md

Latest commit

 

History

History
595 lines (466 loc) · 39.3 KB

CLAUDE.md

File metadata and controls

595 lines (466 loc) · 39.3 KB
Raw

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

Project Overview

OpenContracts is an MIT-licensed enterprise document analytics platform for PDFs and text-based formats. It features a Django/GraphQL backend with PostgreSQL + pgvector, a React/TypeScript frontend with Jotai state management, and pluggable document processing pipelines powered by machine learning models.

Baseline Commit Rules

  1. Always ensure all affected (or new) tests pass - backend tests suite should only be run in its entirety for good reason as it takes 30+ minutes.
  2. Always make sure typescript compiles and pre-commits pass before committing new code.
  3. Never credit Claude or Claude Code in commit messages, PR messages, comments, or any other artifacts. This includes Co-Authored-By lines, "Generated by Claude", etc.

Essential Commands

Backend (Django)

# Run backend tests (sequential, use --keepdb to speed up subsequent runs)
docker compose -f test.yml run django python manage.py test --keepdb

# Run backend tests in PARALLEL (recommended - ~4x faster)
# Uses pytest-xdist with 4 workers, --dist loadscope keeps class tests together
docker compose -f test.yml run django pytest -n 4 --dist loadscope

# Run parallel tests with auto-detected worker count (uses all CPU cores)
docker compose -f test.yml run django pytest -n auto --dist loadscope

# Run parallel tests with fresh databases (first run or after schema changes)
docker compose -f test.yml run django pytest -n 4 --dist loadscope --create-db

# Run specific test file
docker compose -f test.yml run django python manage.py test opencontractserver.tests.test_notifications --keepdb

# Run specific test file in parallel
docker compose -f test.yml run django pytest opencontractserver/tests/test_notifications.py -n 4 --dist loadscope

# Run specific test class/method
docker compose -f test.yml run django python manage.py test opencontractserver.tests.test_notifications.TestNotificationModel.test_create_notification --keepdb

# Apply database migrations
docker compose -f local.yml run django python manage.py migrate

# Create new migration
docker compose -f local.yml run django python manage.py makemigrations

# Django shell
docker compose -f local.yml run django python manage.py shell

# Code quality (runs automatically via pre-commit hooks)
pre-commit run --all-files

Frontend (React/TypeScript)

cd frontend

# Start development server (proxies to Django on :8000)
yarn start

# Run unit tests (Vitest) - watches by default
yarn test:unit

# Run component tests (Playwright) - CRITICAL: Use --reporter=list to prevent hanging
yarn test:ct --reporter=list

# Run component tests with grep filter
yarn test:ct --reporter=list -g "test name pattern"

# Run E2E tests
yarn test:e2e

# Coverage reports (unit tests via Vitest, component tests via Playwright + Istanbul)
yarn test:coverage:unit
yarn test:coverage:ct

# Linting and formatting
yarn lint
yarn fix-styles

# Build for production
yarn build

# Preview production build locally
yarn serve

Production Deployment

# CRITICAL: Always run migrations FIRST in production
docker compose -f production.yml --profile migrate up migrate

# Then start main services
docker compose -f production.yml up

High-Level Architecture

Backend Architecture

Stack: Django 4.x + GraphQL (strawberry-graphql) + PostgreSQL + pgvector + Celery

Key Patterns:

  1. GraphQL Schema Organization (strawberry; migrated from graphene — query shapes pinned):

    • config/graphql/core/ - shared runtime: relay global IDs/Node, connection factories + graphene-parity pagination, FilterSet arg mapping, scalars, permission resolvers, DRF mutation bases, auth decorators
    • config/graphql/*_types.py - GraphQL type definitions (organized by feature)
    • config/graphql/*_queries.py / *_mutations.py - Query/Mutation fields + resolvers (each module exports QUERY_FIELDS / MUTATION_FIELDS)
    • config/graphql/schema.py - Schema composition + validation rules
    • config/graphql/schema.graphql - golden SDL contract; opencontractserver/tests/test_schema_parity.py fails on ANY shape drift. Regenerate it deliberately when changing the API surface (command in that test's docstring).
    • config/graphql/views.py - HTTP view; per-request auth (JWT/Auth0/API-key) happens in get_context via AUTHENTICATION_BACKENDS (no per-resolver auth middleware)

  2. Permission System (CRITICAL - see docs/permissioning/consolidated_permissioning_guide.md):

    • Annotations & Relationships: NO individual permissions - inherited from document + corpus
    • Documents & Corpuses: Direct object-level permissions via django-guardian
    • Analyses & Extracts: Hybrid model (own permissions + corpus permissions + document filtering)
    • Formula: Effective Permission = MIN(document_permission, corpus_permission)
    • Structural items are ALWAYS read-only except for superusers
    • Use Model.objects.visible_to_user(user) pattern (NOT resolve_oc_model_queryset - DEPRECATED)
    • For corpus-scoped document access (the most common pattern), prefer CorpusDocumentService.get_corpus_documents(user, corpus) over composing visible_to_user filters by hand — the service is the canonical entry point and prevents IDOR-prone copy-paste fusions of corpus.get_documents() + Document.objects.visible_to_user(user).
    • Corpus document access — two deliberate semantics (issue #1682): CorpusDocumentService.get_corpus_documents(user, corpus) is corpus-as-gate — corpus READ unlocks every document with an active path in that corpus. It is the documented default for pipeline-facing callers (MCP, badge/analysis tasks) that legitimately operate over a whole readable corpus and never return or persist verbatim document content to the caller. CorpusDocumentService.get_corpus_documents_visible_to_user(user, corpus) enforces MIN(document_permission, corpus_permission) — a private document inside a public (or merely shared) corpus stays hidden from users who lack document-level READ. User-facing surfaces that must not leak private documents (e.g. the GraphQL CorpusType.documents resolver) MUST use the _visible_to_user variant. Authority enrichment is _visible_to_user, not corpus-as-gate (PR #2084): EnrichmentService.discover/scan/apply (opencontractserver/enrichment/services/enrichment_service.py::_load) return verbatim excerpts and persist Annotation/CorpusReference rows derived from document text, so they load through the MIN variant and surface a documents_excluded_by_visibility count (+ a WARNING) when the caller's creator_id lacks per-document READ on some documents. Choose the method by caller intent; never silently swap one semantic for the other.

  3. Permission-annotation fields (myPermissions, isPublished, objectSharedWith):

    • Resolved by config/graphql/core/permissions.py (port of the graphene-era AnnotatePermissionsForReadMixin); most GraphQL types expose them
    • Requires model to have guardian permission tables ({model}userobjectpermission_set)
    • Per-request model-permission maps are memoised on info.context.permission_annotations
    • Notifications use simple ownership model and DON'T expose these fields

  4. Django Signal Handlers:

    • Automatic notification creation on model changes (see opencontractserver/notifications/signals.py)
    • Must be imported in app's apps.py ready() method
    • Use _skip_signals attribute to prevent duplicate notifications in tests

  5. Pluggable Parser Pipeline:

    • Base classes in opencontractserver/pipeline/base/
    • Parsers, embedders, thumbnailers auto-discovered and registered
    • Multiple backends: Docling (ML-based), LlamaParse, Text
    • All convert to unified PAWLs format for frontend

  6. Agent Tool Architecture (see docs/architecture/llms/README.md):

    • CoreTool (framework-agnostic) → PydanticAIToolWrapper (pydantic-ai specific)
    • All production tools MUST be async (a-prefixed in core_tools.py). The wrapper supports sync functions for lightweight helpers/tests but does NOT wrap them in a thread pool — sync ORM calls will raise SynchronousOnlyOperation.
    • Tool fault tolerance (issue #820): operational exceptions are caught and returned as error strings to the LLM; security exceptions (PermissionError, ToolConfirmationRequired) propagate.
    • Pre-execution checks run on every call (not cached): permission validation, resource ID validation, approval gates.

Frontend Architecture

Stack: React 18 + TypeScript + Apollo Client + Jotai (atoms) + PDF.js + Vite

Key Patterns:

  1. State Management - Jotai Atoms:

    • Global state via atoms in frontend/src/atoms/ (NOT Redux/Context)
    • Key atoms: selectedFolderIdAtom, folderCorpusIdAtom, selectedMessageIdAtom
    • Derived atoms automatically update when dependencies change
    • Apollo reactive vars in frontend/src/graphql/cache.ts for UI state
    • AuthGate pattern ensures auth completes before rendering

  2. Central Routing System (see docs/frontend/routing_system.md):

    • Single source of truth: frontend/src/routing/CentralRouteManager.tsx
    • URL paths → Entity resolution via GraphQL slug queries
    • URL params ↔ Reactive vars (bidirectional sync)
    • Components consume state via reactive vars, never touch URLs directly
    • Deep linking and canonical redirects handled automatically

  3. PDF Annotation System (see .cursor/rules/pdf-viewer-and-annotator-architecture.mdc):

    • Virtualized rendering: Only visible pages (+overscan) rendered for performance
    • Binary search to find visible page range (O(log n))
    • Height caching per zoom level
    • Two-phase scroll-to-annotation system
    • Dual-layer architecture: Document layer (annotations) + Knowledge layer (summaries)

  4. Unified Filtering Architecture:

    • useVisibleAnnotations hook provides single-source-of-truth filtering for both annotations and relationship-connected annotations
    • Reads from Jotai atoms via useAnnotationDisplay, useAnnotationControls, and useAnnotationSelection (from UISettingsAtom)
    • Ensures consistency across all components
    • Forced visibility for selected items and their relationship connections

  5. Component Testing (see .cursor/rules/test-document-knowledge-base.mdc):

    • ALWAYS mount components through test wrappers (e.g., DocumentKnowledgeBaseTestWrapper)
    • Wrapper provides: MockedProvider + InMemoryCache + Jotai Provider + asset mocking
    • Use --reporter=list flag to prevent hanging
    • Increase timeouts (20s+) for PDF rendering in Chromium
    • GraphQL mocks must match variables EXACTLY (null vs undefined matters)
    • Mock same query multiple times for refetches
    • Use page.mouse for PDF canvas interactions (NOT locator.dragTo)
    • Add settle time after drag operations (500ms UI, 1000ms Apollo cache)

  6. Development Server Configuration:

    • Vite dev server on :3000 proxies to Django on :8000
    • WebSocket proxy for /ws → ws://localhost:8000
    • GraphQL proxy for /graphql → http://localhost:8000
    • REST API proxy for /api → http://localhost:8000
    • Auth0 optional via REACT_APP_USE_AUTH0 environment variable

Data Flow Architecture

Document Processing:

  1. Upload → Parser Selection (Docling/LlamaParse/Text)
  2. Parser generates PAWLs JSON (tokens with bounding boxes)
  3. Text layer extracted from PAWLs
  4. Annotations created for structure (headers, sections, etc.)
  5. Relationships detected between elements
  6. Vector embeddings generated for search

GraphQL Permission Flow:

  1. Query resolver filters objects with .visible_to_user(user)
  2. GraphQL types resolve my_permissions via AnnotatePermissionsForReadMixin
  3. Frontend uses permissions to enable/disable UI features
  4. Mutations check permissions and return consistent errors to prevent IDOR

Critical Security Patterns

  1. IDOR Prevention:

    • Query by both ID AND user-owned field: Model.objects.get(pk=pk, recipient=user)
    • Return same error message whether object doesn't exist or belongs to another user
    • Prevents enumeration via timing or different error messages

  2. Permission Checks:

    • NEVER trust frontend - always check server-side
    • Use Model.objects.visible_to_user(user) manager method for querysets (list/filter)
    • For a single-object check use Model.objects.user_can(user, obj, perm) / obj.user_can(user, perm) — the canonical API, paired with visible_to_user and pinned to agree by opencontractserver/tests/permissioning/test_authorization_invariants.py. In GraphQL resolvers/mutations pass request=info.context to engage the Tier-2 permission cache; omit it in Celery/agent/internal code.

  3. XSS Prevention:

    • User-generated content in JSON fields must be escaped on frontend
    • GraphQL's GenericScalar handles JSON serialization safely
    • Document this requirement in resolver comments

Critical Concepts

  1. No dead code - when deprecating or replacing code, always try to fully replace older code and, once it's no longer in use, delete it and related texts.
  2. DRY - please always architect code for maximal dryness and always see if you can consolidate related code and remove duplicative code.
  3. Single Responsibility Principle - Generally, ensure that each module / script has a single purpose or related purpose.
  4. No magic numbers - we have constants files in opencontractserver/constants/ (backend) and frontend/src/assets/configurations/constants.ts (frontend). Use them for any hardcoded values.
  5. Don't touch old tests without permission - if pre-existing tests fail after changes, try to identify why and present user with root cause analysis. If the test logic is correct but expectations need updating due to intentional behavior changes, document the change clearly.
  6. Utility functions belong in utility files:
    • Before writing new utilities: Check existing utility files first (frontend/src/utils/, opencontractserver/utils/)
    • When reviewing code: If you find utility functions defined inline in components/views, check if they already exist in utility files. If not, consider whether they should be extracted there for reuse.
    • Frontend utilities: frontend/src/utils/formatters.ts (formatting), frontend/src/utils/files.ts (file operations), etc.
    • Backend utilities: opencontractserver/utils/ contains permissioning, PDF processing, and other shared utilities
  7. Always go through the app's services/ package. Code with a user context (GraphQL resolvers, MCP tools, LLM tools, REST views, Celery tasks invoked with a user) must reach models through opencontractserver/<app>/services/ — never compose visible_to_user / user_can / user_has_permission_for_obj inline. The shared base (opencontractserver.shared.services.base.BaseService) exposes get_or_none, filter_visible, require_permission, and user_has for the cases where a dedicated per-app method is overkill. The invariant is enforced twice — by a pytest test (opencontractserver/tests/architecture/test_graphql_service_layer.py) and by a Django system check (opencontractserver/shared/checks.py, opencontracts.E001) that fails manage.py startup on any inline Tier-0 use. Scope of mechanical enforcement: both the test and the check scan config/graphql/ only (recursively). The rule above is policy for the other user-context surfaces — MCP tools (opencontractserver/mcp/), LLM tools (opencontractserver/llms/tools/), REST views, and user-context Celery tasks — but those are not scanned today and still contain correct-but-inline Tier-0 calls. Treat a green E001 as "no inline Tier-0 in config/graphql/," not "none anywhere." Failure messages carry the copy-pasteable recipe inline; see docs/development/architecture_invariants.md for the invariant catalogue and docs/architecture/query_permission_patterns.md for the per-app service catalogue + migration recipes.
  8. Docs: keep them current, concise, and pointer-based. When you change behavior, update the relevant doc (or add one if none fits) as part of the same change — stale docs are worse than none. But keep docs concise and prune as you go: a doc that only ever grows rots into noise, so trim superseded content rather than appending forever. Favor code pointers over pasted code — reference files + symbols (e.g. opencontractserver/enrichment/authorities.py::bootstrap_authority_corpus) instead of copying snippets in. Pasted code drifts out of sync the moment the source changes and becomes dead documentation; a pointer stays live. (See ## Documentation Locations for where things live, and ## Changelog Maintenance for change records.)

Testing Patterns

Manual Test Scripts

Location: docs/test_scripts/

When performing manual testing (e.g., testing migrations, verifying database state, testing API endpoints interactively), always document the test steps in a markdown file under docs/test_scripts/. These scripts will later be used to build automated integration tests.

Format:

# Test: [Brief description]

## Purpose
What this test verifies.

## Prerequisites
- Required state (e.g., "migration at 0058")
- Required data (e.g., "at least one document exists")

## Steps
1. Step one with exact command
   ```bash
   docker compose -f local.yml run --rm django python manage.py shell -c "..."
  1. Step two...

Expected Results

  • What should happen after each step
  • Success criteria

Cleanup

Commands to restore original state if needed.


**When to document**:
- Migration testing (rollback, create test data, migrate forward)
- Database constraint validation
- Race condition verification
- Any manual verification requested during code review

### Backend Tests

**Location**: `opencontractserver/tests/`

**Parallel Testing** (pytest-xdist):
- Run with `-n 4` (or `-n auto`) for parallel execution across workers
- Use `--dist loadscope` to keep tests from the same class on the same worker (respects `setUpClass`)
- Each worker gets its own database (test_db_gw0, test_db_gw1, etc.)
- Use `@pytest.mark.serial` to mark tests that cannot run in parallel
- First run or after DB schema changes: add `--create-db` flag

**Patterns**:
- Use `TransactionTestCase` for tests with signals/asynchronous behavior
- Use `TestCase` for faster tests without transaction isolation
- Clear auto-created notifications when testing moderation: `Notification.objects.filter(recipient=user).delete()`
- Use `_skip_signals` attribute on instances to prevent signal handlers during fixtures

### Frontend Component Tests

**Location**: `frontend/tests/`

**Critical Requirements**:
- Mount through test wrappers that provide all required context
- GraphQL mocks must match query variables exactly
- Include mocks for empty-string variants (unexpected boot calls)
- Wait for visible evidence, not just network-idle
- Use `page.mouse` for PDF canvas interactions (NOT `locator.dragTo`)
- Add settle time after drag operations (500ms UI, 1000ms Apollo cache)

**Test Wrapper Pattern**:
```typescript
// ALWAYS use wrappers, never mount components directly
const component = await mount(
  <DocumentKnowledgeBaseTestWrapper corpusId="corpus-1" documentId="doc-1">
    <DocumentKnowledgeBase />
  </DocumentKnowledgeBaseTestWrapper>
);

Automated Documentation Screenshots

Location: docs/assets/images/screenshots/auto/ (output) | frontend/tests/utils/docScreenshot.ts (utility)

Screenshots for documentation are captured during Playwright component tests and pushed back to a PR branch by the screenshots.yml workflow. The workflow is manual / on-demand only (workflow_dispatch) — it does not auto-trigger on PR push or merge. Run it from the Actions tab or via gh workflow run "Update Documentation Screenshots" -f pr_number=<N> when you want to refresh the screenshots on a PR.

How it works:

  1. Import docScreenshot from ./utils/docScreenshot in any .ct.tsx test file
  2. Call await docScreenshot(page, "area--component--state") after the component reaches the desired visual state
  3. Reference the image in markdown: ![Alt text](../assets/images/screenshots/auto/area--component--state.png)
  4. Trigger the screenshot workflow on demand for the PR; it captures screenshots and commits any changes back to the PR branch

Naming convention (-- separates segments, - within words):

Segment Purpose Examples
area Feature area landing, badges, corpus, versioning, annotations
component Specific view or component hero-section, celebration-modal, list-view
state Visual state captured anonymous, with-data, empty, auto-award

At least 2 segments required, 3 recommended. All lowercase alphanumeric with single hyphens.

Example:

import { docScreenshot } from "./utils/docScreenshot";

// After the component renders and assertions pass:
await docScreenshot(page, "badges--celebration-modal--auto-award");

Rules:

  • Place docScreenshot() calls AFTER assertions that confirm the desired visual state
  • The filename IS the contract between tests and docs — keep names stable
  • Never manually edit files in docs/assets/images/screenshots/auto/ — they are overwritten by CI
  • Manually curated screenshots stay in docs/assets/images/screenshots/ (parent directory)

Release Screenshots (Point-in-Time)

For release notes, use releaseScreenshot to capture screenshots that are locked in amber — they show the UI at a specific release and never change.

Location: docs/assets/images/screenshots/releases/{version}/ (output)

import { releaseScreenshot } from "./utils/docScreenshot";

await releaseScreenshot(page, "v3.0.0.b3", "landing-page", { fullPage: true });

Key differences from docScreenshot:

  • Output: docs/assets/images/screenshots/releases/{version}/{name}.png
  • Write-once: If the file already exists, the function is a no-op (won't overwrite)
  • CI never touches the releases/ directory
  • Name is a simple kebab-case string (no -- segment convention)
  • Version must match v{major}.{minor}.{patch} format (with optional suffix)

When to use which:

  • docScreenshot → README, quickstart, guides (always fresh)
  • releaseScreenshot → Release notes (frozen at release time)

Authenticated Playwright Testing (Live Frontend Debugging)

When you need to interact with the running frontend as an authenticated user (e.g., debugging why a query returns empty results), use Django admin session cookies to authenticate GraphQL requests.

Architecture context: The frontend uses Auth0 for authentication, but the Django backend also accepts session cookie auth. Apollo Client sends GraphQL requests directly to http://localhost:8000/graphql/ (cross-origin from the Vite dev server at localhost:5173). Since fetch defaults to credentials: 'same-origin', browser cookies aren't sent cross-origin. The workaround is to inject the session cookie into request headers via Playwright's route interception.

Step 1: Set a password for the superuser (one-time setup):

docker compose -f local.yml exec django python manage.py shell -c "
from django.contrib.auth import get_user_model
User = get_user_model()
u = User.objects.filter(is_superuser=True).first()
u.set_password('testpass123')
u.save()
print(f'Password set for {u.username}')
"

Step 2: Playwright script pattern:

const { chromium } = require('playwright');

(async () => {
  const browser = await chromium.launch({ headless: true });
  const context = await browser.newContext();
  const page = await context.newPage();

  // Collect console messages for debugging
  const consoleMsgs = [];
  page.on('console', msg => consoleMsgs.push('[' + msg.type() + '] ' + msg.text()));

  // 1. Login to Django admin to get session cookie
  await page.goto('http://localhost:8000/admin/login/');
  await page.fill('#id_username', '<superuser-username>');
  await page.fill('#id_password', 'testpass123');
  await page.click('input[type=submit]');
  await page.waitForTimeout(2000);

  // 2. Extract the session cookie
  const cookies = await context.cookies();
  const sessionCookie = cookies.find(c => c.name === 'sessionid');

  // 3. Intercept GraphQL requests to inject the session cookie
  //    (needed because Apollo sends cross-origin requests to :8000)
  await page.route('**/graphql/**', async (route) => {
    const headers = {
      ...route.request().headers(),
      'Cookie': 'sessionid=' + sessionCookie.value,
    };
    await route.continue({ headers });
  });

  // 4. Navigate to the frontend page under test
  await page.goto('http://localhost:5173/extracts');
  await page.waitForTimeout(5000);

  // 5. Inspect results
  const bodyText = await page.textContent('body');
  console.log(bodyText);

  await browser.close();
})();

Run from the frontend directory (where playwright is a dependency):

cd frontend && node /path/to/script.js

Key details:

  • The Django admin login sets a sessionid cookie for localhost
  • page.route('**/graphql/**') intercepts Apollo's requests to localhost:8000/graphql/ and injects the cookie header
  • The AuthGate will still show anonymous state (no Auth0 session), but GraphQL queries execute as the authenticated Django user
  • This is useful for verifying backend query results, permission filtering, and data flow through the real frontend
  • For verifying just the GraphQL response without the full frontend, curl with the session cookie also works: 
    curl -s -X POST http://localhost:8000/graphql/ \
  -H "Content-Type: application/json" \
  -H "Cookie: sessionid=<session-key>" \
  -d '{"query":"query { extracts { edges { node { id name } } } }"}' | python3 -m json.tool

Alternative — create a session programmatically (no admin login needed):

docker compose -f local.yml exec django python manage.py shell -c "
from django.contrib.sessions.backends.db import SessionStore
from django.contrib.auth import get_user_model
User = get_user_model()
user = User.objects.filter(is_superuser=True).first()
session = SessionStore()
session['_auth_user_id'] = str(user.pk)
session['_auth_user_backend'] = 'django.contrib.auth.backends.ModelBackend'
session['_auth_user_hash'] = user.get_session_auth_hash()
session.save()
print(f'Session key: {session.session_key}')
"

Then use the printed session key directly in curl or Playwright route interception.

Documentation Locations

  • Permissioning: docs/permissioning/consolidated_permissioning_guide.md
  • Frontend Routing: docs/frontend/routing_system.md
  • Frontend Auth Flow: docs/frontend/auth_flow.md
  • PDF Data Layer: docs/architecture/PDF-data-layer.md
  • PAWLs Format (v1/v2): docs/architecture/pawls-format.md
  • Parser Pipeline: docs/pipelines/pipeline_overview.md
  • LLM Framework: docs/architecture/llms/README.md
  • Collaboration System: docs/commenting_system/README.md
  • Auth Pattern (detailed): frontend/src/docs/AUTHENTICATION_PATTERN.md
  • Documentation Screenshots: docs/development/screenshots.md
  • Query Permission Patterns: docs/architecture/query_permission_patterns.md
  • OS-Legal-Style Migration Guide: docs/frontend/os-legal-style-migration-guide.md

Branch Strategy

This project follows trunk-based development:

  • Work directly on main branch or use short-lived feature branches
  • Feature branches: feature/description-issue-number
  • Merge feature branches quickly (within a day or two)
  • Commit message format: Descriptive with issue references (e.g., "Closes #562")

Changelog Maintenance

IMPORTANT: Always record significant changes — but do NOT edit CHANGELOG.md directly in a PR. Add a changelog fragment under changelog.d/ instead.

Why: every PR used to insert its entry at the top of CHANGELOG.md's single ## [Unreleased] section, so concurrent PRs collided on the same lines and produced perpetual merge conflicts (the file changed in ~hundreds of commits per month). Each PR now adds its own uniquely-named file, so two PRs can never touch the same lines — changelog merge conflicts become structurally impossible. The fragments are collated into CHANGELOG.md at release time. (A merge=union driver in .gitattributes is a safety net for any direct edit that slips through; it auto-keeps both sides instead of conflicting.)

How to add an entry — create one file per change:

changelog.d/<slug>.<type>.md
  • <slug> — anything unique; the PR number (1901) is ideal, or a short kebab-case description.
  • <type> — one of added, changed, deprecated, removed, fixed, security (Keep a Changelog groups).
  • Body = the markdown bullet(s) you'd have written under the section header, without the ### Fixed header itself (the type comes from the filename).
  • Keep the same quality bar: file paths, line numbers, issue/impact, rationale.
  • Multiple categories in one PR → multiple fragments (1908-search.added.md + 1908-search.fixed.md).

Tooling (scripts/collate_changelog.py): --check validates fragments (CI / pre-commit), --preview prints the collated markdown, --apply folds fragments into CHANGELOG.md's [Unreleased] section and deletes them (release step). See changelog.d/README.md for full details.

When to add a fragment:

  • New features or models added
  • Production code bugs fixed (document file location, line numbers, and impact)
  • Breaking changes to APIs or data models
  • Test suite fixes that reveal production issues
  • Database migrations
  • Architecture changes

What to include:

  • File paths and line numbers for code changes
  • Clear description of the issue and fix
  • Impact on system behavior
  • Migration notes if applicable

Pre-commit Hooks

Automatically run on commit:

  • black (Python formatting)
  • isort (import sorting)
  • flake8 (linting)
  • prettier (frontend formatting)
  • pyupgrade (Python syntax modernization)

Run manually: pre-commit run --all-files

Common Pitfalls

  1. Frontend tests hanging: Always use --reporter=list flag
  2. Permission N+1 queries: Use .visible_to_user() NOT individual permission checks. For Conversation list queries in GraphQL resolvers, use ConversationService from opencontractserver.conversations.services - it provides request-level caching to avoid repeated corpus/document visibility subqueries. For annotation/relationship source privacy, looping user_can over rows dereferences created_by_analysis / created_by_extract; use list/queryset surfaces or select_related("created_by_analysis", "created_by_extract").
  3. Missing GraphQL mocks: Check variables match exactly (null vs undefined matters), add duplicates for refetches
  4. Notification duplication in tests: Moderation methods auto-create ModerationAction records
  5. Structural annotation editing: Always read-only except for superusers
  6. Missing signal imports: Import signal handlers in apps.py ready() method
  7. PDF rendering slow in tests: Increase timeouts to 20s+ for Chromium
  8. Cache serialization crashes: Keep InMemoryCache definition inside wrapper, not test file
  9. Backend Tests Waiting > 10 seconds on Postgres to be Ready: Docker network issue. Fix with: docker compose -f test.yml down && docker kill $(docker ps -q) && docker compose -f test.yml down
  10. Empty lists on direct navigation: AuthGate pattern solves this (don't check auth status, it's always ready)
  11. URL desynchronization: Use CentralRouteManager, don't bypass routing system
  12. Jotai state not updating: Ensure atoms are properly imported and used with useAtom hook
  13. Writing sync agent tools: All agent tools must be async. The PydanticAIToolWrapper accepts sync functions but calls them directly (no thread pool) — sync Django ORM calls will raise SynchronousOnlyOperation. Use the a-prefixed async versions in core_tools.py.
  14. PydanticAI system_prompt silently dropped: When creating PydanticAIAgent, use instructions= NOT system_prompt=. The system_prompt parameter is only included when message_history is None, but OpenContracts' chat() persists a HUMAN message before calling pydantic-ai's run(), so history is always non-empty. This causes the system prompt to be silently dropped. See docs/architecture/llms/README.md for full details.
  15. Apollo cache keyArgs must use field argument names, not variable names: In cache.ts, relayStylePagination(["corpus", "name_Contains"]) uses the GraphQL field argument names (e.g., extracts(corpus: $id, name_Contains: $text)), NOT the variable names ($id, $text). Mismatched keyArgs silently fail to isolate cache entries, causing queries with different filters to share stale cached results.
  16. Playwright CT split-import rule: In frontend/tests/*.ct.tsx, keep JSX-component imports in their own import { Wrapper } from "./Wrapper" statement, separate from any helper/constant imports from the same file. Playwright CT's babel transform only rewrites component references into importRefs when every specifier in the statement is a JSX component; mixing a component with helpers leaves the component unrewritten and mount() throws. Do not let auto-import organisers merge them.
  17. pydantic-ai dataclass-shape assertion at import time: opencontractserver/llms/history_processors.py performs a module-level hasattr(_msg_type, "__dataclass_fields__") check on ToolReturnPart/ModelRequest/ModelResponse and raises RuntimeError if any of them stops being a stdlib @dataclass. The check is intentional — the in-run shrink helpers use dataclasses.replace() and the outer defensive try/except would otherwise swallow a silent shape change. The cost is that a routine pip install --upgrade pydantic-ai that flips one of these to a BaseModel will surface as a startup RuntimeError (manage.py runserver, Celery worker boot). The traceback names the offending message class — update the helpers in history_processors.py to use the new message API rather than disabling the assertion.
  18. Corrupted Docker iptables chains (RARE): If you see Chain 'DOCKER-ISOLATION-STAGE-2' does not exist errors, Docker's iptables chains have been corrupted during docker cycling. Run this nuclear fix: 
    sudo systemctl stop docker && sudo systemctl stop docker.socket && sudo ip link delete docker0 2>/dev/null || true && sudo iptables -t nat -F && sudo iptables -t nat -X && sudo iptables -t filter -F && sudo iptables -t filter -X 2>/dev/null || true && sudo iptables -t mangle -F && sudo iptables -t mangle -X && sudo iptables -t filter -N INPUT 2>/dev/null || true && sudo iptables -t filter -N FORWARD 2>/dev/null || true && sudo iptables -t filter -N OUTPUT 2>/dev/null || true && sudo iptables -P INPUT ACCEPT && sudo iptables -P FORWARD ACCEPT && sudo iptables -P OUTPUT ACCEPT && sudo systemctl start docker
    This completely resets Docker's networking and iptables state. Docker will recreate all required chains on startup.
  19. WebSocket 1011 reconnect churn (channels-redis ↔ redis-py socket_timeout): If every WebSocket consumer (/ws/notification-updates/, /ws/agent-chat/, …) drops with close code 1011 on a clean ~5-second beat and the browser reconnects forever, it is almost certainly the channel layer's idle blocking read timing out client-side, NOT a frontend re-render/remount (the connect hook frontend/src/hooks/useWebSocketAuth.ts is re-render-safe: url is a value-compared string, callbacks are ref-held). channels-redis' receive loop issues a 5s server-side blocking pop (bzpopmin, brpop_timeout=5); redis-py 8.0 changed the default socket_timeout from None to 5s, so the client read deadline fires before the server's nil reply returns → redis.exceptions.TimeoutError escapes the consumer → Daphne emits 1011. redis is unpinned in requirements/base.txt, which is how it drifted to 8.0. Fix (do NOT pin redis back below 8.0): the CHANNEL_LAYERS host must be a dict with socket_timeout: None (see config/settings/base.py and config/settings/test_integration.py) — a bare (host, port) tuple or URL string inherits redis-py's default and reintroduces the bug. Server-side symptom in the Daphne logs: Exception inside application: Timeout reading from redis:6379 immediately followed by WSDISCONNECT. Regression test: test_redis_integration.py::TestChannelsRedisLayer::test_idle_receive_survives_blocking_pop_window. See issue #1886.
  20. Reasoning models that require the OpenAI Responses API (gpt-5.6-sol / -terra / -luna, and likely successors): two separate failures, both invisible on /v1/chat/completions and both fatal to an agent run.
    • Function tools are rejected outright: Function tools with reasoning_effort are not supported for gpt-5.6-luna in /v1/chat/completions. To use function tools, use /v1/responses or set reasoning_effort to 'none'. Every OpenContracts agent carries function tools, and zeroing reasoning discards the capability the model is chosen for, so the endpoint moves instead — llms/model_factory.py::requires_responses_api routes the family to /v1/responses on BOTH the DB-credential and env-credential paths. Routed automatically because these names are selectable in the System Settings LLM picker; without it, choosing one 400s every agent in the install with the reason visible only in a worker log.
    • Orphaned function_call items: the Responses API rejects a tool call whose paired reasoning item is missing — Item 'fc_…' of type 'function_call' was provided without its required 'reasoning' item: 'rs_…'. In-run compaction (llms/history_processors.py) drops ThinkingParts from older messages, which Chat Completions tolerated for years. It now keeps reasoning on any response that also carries a ToolCallPart; reasoning-only responses still shed theirs. Expressed as an invariant, not a per-provider switch.
    • Also register the context window in constants/context_guardrails.py. There is no bare gpt-5 entry to inherit, so an unlisted name falls to DEFAULT_CONTEXT_WINDOW (128K) and sizes a 1M-window model at an eighth of its budget — which moves when a deep-research run compacts, not just a reported number. Same trap as the gpt-4.1 entries above it.
  21. tool_call_log does not see the whole toolset by default: research_tasks.py::_audited wraps only the closures that module builds. Everything from the corpus agent's default toolset (similarity_search, list_documents, ask_document, …) bypasses it unless _audit_default_toolset runs, and that helper reaches into pydantic-ai's Tool.function_schema.function seam — which a framework upgrade could move. Never read a tool's absence off this log without corroboration. Doing so once produced a confidently wrong conclusion ("ten runs never searched by meaning") that drove a prompt rewrite; the agent had been calling similarity_search the whole time. Independent check that costs nothing: a phrase lookup (find_citable_passages → AnnotationService.search_corpus_annotation_text) performs a TEXT query and embeds nothing, while a semantic search must embed its query — so Embedding text with MicroserviceEmbedder in the worker log during a run is proof of semantic retrieval regardless of what the audit log shows. Note also that closures passed to the factory are RE-WRAPPED by it, so marker attributes set on our wrapper are not reachable from the resolved toolset; _audit_default_toolset therefore skips by NAME.