docs: map existing codebase (#81)

Author: tazarov

.planning/codebase/ARCHITECTURE.md

@@ -0,0 +1,130 @@
+# Architecture
+
+**Analysis Date:** 2026-03-18
+
+## Pattern Overview
+
+**Overall:** Layered Go library for native ONNX Runtime interop, with a low-level FFI core in `ort/` and model-specific convenience layers in `embeddings/`.
+
+**Key Characteristics:**
+- No-CGO design: native symbols are loaded dynamically through `purego`
+- Global runtime lifecycle managed once per process in `ort/environment.go`
+- Explicit resource ownership for tensors, sessions, memory info, and embedders
+- Higher-level embedding packages build reusable batching and post-processing on top of the raw ORT session API
+
+## Layers
+
+**FFI Runtime Layer (`ort/`):**
+- Purpose: map the ONNX Runtime C API into Go and expose safe-ish wrappers for environment, tensor, memory, and session lifecycle
+- Contains: `InitializeEnvironment`, `DestroyEnvironment`, `AdvancedSession`, `Tensor[T]`, bootstrap/download logic, OS-specific library loading
+- Depends on: `purego`, `unsafe`, native ONNX Runtime shared libraries, and generated `OrtApi` bindings
+- Used by: all example programs and all packages under `embeddings/`
+
+**Model Adapter Layer (`embeddings/*`):**
+- Purpose: hide raw tensor/session wiring behind model-specific APIs
+- Contains: `minilm.Embedder`, `splade.Embedder`, `openclip.Embedder`, session cache management, tokenizer usage, pooling/post-processing
+- Depends on: `ort/`, `pure-tokenizers`, and model-specific local artifacts
+- Used by: example programs and downstream applications that want dense, sparse, or CLIP embeddings
+
+**Utility Layer (`embeddings/internal/ortutil`):**
+- Purpose: small shared helpers for resource cleanup
+- Contains: `DestroyAll`
+- Depends on: only local interfaces and the Go standard library
+- Used by: embedding packages when tearing down grouped ORT resources
+
+**Executable/Tooling Layer (`examples/`, `tools/`, `.github/workflows/`):**
+- Purpose: provide runnable demos, generators, and CI/release automation
+- Contains: basic/inference/openclip examples, OpenCLIP export tooling, `gen_ortapi.go`, GitHub Actions workflows
+- Depends on: lower library layers plus external services such as GitHub and Hugging Face
+- Used by: maintainers, CI, and users validating real-world flows
+
+## Data Flow
+
+**Core ORT Inference Flow:**
+
+1. Caller resolves a runtime library path explicitly or via `EnsureOnnxRuntimeSharedLibrary()` in `ort/bootstrap.go`
+2. `ort.InitializeEnvironment()` loads the native library, registers function pointers, and creates a process-global ORT environment
+3. Caller creates tensors via `ort.NewTensor` / `ort.NewEmptyTensor`
+4. Caller constructs an `ort.AdvancedSession` with model path, input names, and output names
+5. `AdvancedSession.Run()` acquires session-local and global runtime locks, converts names/values into native handles, and invokes ORT
+6. Callers read tensor-backed output slices and then destroy sessions/tensors/environment in reverse order
+
+**Embedding Flow:**
+
+1. Caller initializes `ort` once for the process
+2. Embedder loads tokenizer/model metadata and validates artifact paths
+3. Inputs are tokenized or preprocessed into reusable backing buffers
+4. Package-specific session caches keyed by batch size reuse tensors and `AdvancedSession` instances
+5. Post-processing converts raw outputs into dense vectors, sparse vectors, or CLIP similarity matrices
+
+**Bootstrap/Asset Resolution Flow:**
+
+1. Bootstrap resolves cache directory and target artifact names from platform/runtime config
+2. Downloads are guarded by file/process locks
+3. Archives/files are validated for size, checksum, and path traversal
+4. Resolved local file paths are returned to the caller for normal `ort` or embedding setup
+
+**State Management:**
+- Process-global ORT state lives in package globals in `ort/environment.go`
+- Session-level mutable state lives on `AdvancedSession` and embedder caches
+- Persistent state is file-based only (user cache directories and generated artifacts)
+
+## Key Abstractions
+
+**`AdvancedSession`:**
+- Purpose: wrap an ONNX Runtime session plus fixed input/output bindings
+- Examples: `ort/session.go`, used directly by `examples/inference/main.go` and all embedding packages
+- Pattern: stateful handle wrapper with explicit `Run()` / `Destroy()`
+
+**`Tensor[T]`:**
+- Purpose: represent ORT values backed by Go slices pinned for native access
+- Examples: `ort/tensor.go`
+- Pattern: generic resource wrapper with finalizer safety net and explicit destroy semantics
+
+**`BootstrapOption` / embedding `Option`:**
+- Purpose: configure bootstrap and embedder behavior without large constructors
+- Examples: `ort/bootstrap.go`, `embeddings/minilm/embedder.go`, `embeddings/splade/embedder.go`, `embeddings/openclip/embedder.go`
+- Pattern: functional options
+
+## Entry Points
+
+**Library Entry Points:**
+- `ort/environment.go` - global runtime initialization and teardown
+- `ort/session.go` - model session creation and inference
+- `ort/tensor.go` - tensor allocation and lifecycle
+
+**Example Programs:**
+- `examples/basic/main.go` - minimal runtime initialization example
+- `examples/inference/main.go` - end-to-end single-model inference flow driven by env vars
+- `examples/openclip/main.go` - OpenCLIP text/image embedding demo with manifest-backed fixtures
+
+**Tooling:**
+- `tools/gen_ortapi.go` - regenerates `ort/ortapi_generated.go` from ONNX Runtime headers
+
+## Error Handling
+
+**Strategy:** return Go `error` values aggressively, validate inputs early, and use deferred cleanup to keep native handles from leaking.
+
+**Patterns:**
+- Wrap lower-level failures with `fmt.Errorf(...: %w)`
+- Translate ORT `OrtStatus` handles into Go strings via helper functions in `ort/environment.go`
+- Use `errors.Join` when cleanup steps can fail independently
+- Treat nil receivers as safe no-ops for most `Destroy()` methods
+
+## Cross-Cutting Concerns
+
+**Concurrency:**
+- `ort/environment.go` defines a lock hierarchy spanning global runtime state, session runs, and tensor lifetimes
+- Several tests in `ort/session_test.go` and `ort/environment_test.go` exist specifically to protect these invariants
+
+**Memory Management:**
+- The code pins Go slice backing arrays while native ORT uses them (`ort/tensor.go`)
+- Finalizers are present as leak backstops, but the design still expects explicit `Destroy()` calls
+
+**Security and Integrity:**
+- Bootstrap code validates checksums, path traversal, redirect safety, and download size limits in both `ort/bootstrap.go` and `embeddings/openclip/bootstrap.go`
+
+---
+
+*Architecture analysis: 2026-03-18*
+*Update when major patterns change*

.planning/codebase/CONCERNS.md

@@ -0,0 +1,105 @@
+# Codebase Concerns
+
+**Analysis Date:** 2026-03-18
+
+## Tech Debt
+
+**Generated ORT API binding pipeline:**
+- Issue: `tools/gen_ortapi.go` uses regex parsing of upstream headers instead of a real C parser
+- Why: simple generation was fast to bootstrap
+- Impact: upstream header format changes could silently break `ort/ortapi_generated.go` generation or field ordering
+- Fix approach: replace the parser with a proper C AST approach or add stronger structural validation around generator output
+
+**Stale maintainer guidance in `CLAUDE.md`:**
+- Issue: the file still references `main2.go` as the main implementation entry point, but that file does not exist in the current tree
+- Why: documentation drift as the repo evolved
+- Impact: maintainers or automation may follow outdated guidance
+- Fix approach: update `CLAUDE.md` to match the current `examples/`-based layout and active packages
+
+**Partially implemented wrapper types:**
+- Issue: `ort/types.go` still contains placeholder `Status.GetErrorCode()` and `Status.GetErrorMessage()` implementations
+- Why: the public API surface was sketched before all wrappers were wired to real ORT calls
+- Impact: contributors can mistake these types for production-ready abstractions
+- Fix approach: either finish the implementation or clearly de-emphasize/remove unused wrapper surfaces
+
+## Known Bugs
+
+**Incorrect usage docs are possible around version support:**
+- Symptoms: docs and code can drift between `DefaultOnnxRuntimeVersion`, CI-pinned runtime versions, and README examples
+- Trigger: bumping runtime versions in one place but not the others
+- Workaround: treat `ort/bootstrap.go`, `.github/workflows/ci.yml`, and `README.md` as a synchronized set during upgrades
+- Root cause: multiple version pins live in different files for different purposes
+
+## Security Considerations
+
+**Native binary bootstrap paths must stay hardened:**
+- Risk: relaxing checksum, redirect, or path traversal checks in `ort/bootstrap.go` or `embeddings/openclip/bootstrap.go` would expose consumers to malicious binary/model downloads
+- Current mitigation: checksum validation, redirect policy enforcement, size limits, path sanitization, and lock-guarded extraction
+- Recommendations: preserve these checks during refactors and add tests first when changing bootstrap behavior
+
+**Credential-bearing env vars are handled by runtime/test code:**
+- Risk: `GITHUB_TOKEN`, `GH_TOKEN`, `HF_TOKEN`, and release secrets could be leaked through unsafe logging or insecure redirects
+- Current mitigation: token use is limited and OpenCLIP bootstrap rejects insecure token-bearing base URLs
+- Recommendations: keep logs free of raw env values and treat any new download URL override as security-sensitive
+
+## Performance Bottlenecks
+
+**Session creation remains expensive compared with warm reuse:**
+- Problem: embedder packages build LRU caches to avoid repeatedly creating ORT sessions per batch size
+- Measurement: the existence of dedicated benchmarks in `ort/session_benchmark_test.go` and `embeddings/splade/benchmark_test.go` indicates session setup cost is material
+- Cause: native session creation and tensor wiring are heavier than reuse
+- Improvement path: preserve cache hit paths and benchmark any refactor that touches session lifecycle
+
+**Bootstrap and integration flows download large artifacts:**
+- Problem: real-model tests and bootstrap code move hundreds of MB of runtime/model data
+- Measurement: CI allocates dedicated cache keys and download steps in `.github/workflows/ci.yml`
+- Cause: native runtime archives and model bundles are large by nature
+- Improvement path: keep cache keys stable, avoid redundant downloads, and be careful with checksum/version churn
+
+## Fragile Areas
+
+**Global ORT lifecycle and lock ordering:**
+- Why fragile: `ort/environment.go`, `ort/session.go`, and `ort/tensor.go` coordinate multiple locks and native handles with a strict ordering contract
+- Common failures: deadlocks, use-after-destroy bugs, or release calls after environment teardown
+- Safe modification: change these paths only with concurrency tests running and keep lock-order comments accurate
+- Test coverage: strong relative coverage in `ort/environment_test.go` and `ort/session_test.go`, but still high-risk code
+
+**Embedder tensor/session cache internals:**
+- Why fragile: `embeddings/minilm`, `embeddings/splade`, and `embeddings/openclip` reuse backing slices and tensor handles across runs
+- Common failures: accidental slice reallocation, stale cache entries, or broken LRU eviction
+- Safe modification: preserve buffer ownership assumptions and run integration plus cache-behavior tests after changes
+- Test coverage: good package-local tests, but behavior still depends on real model contracts
+
+## Scaling Limits
+
+**Single-process global runtime model:**
+- Current capacity: one process-global ORT environment shared across sessions
+- Limit: alternative multi-runtime or per-tenant isolation models are not represented in the current architecture
+- Symptoms at limit: awkward lifecycle coordination for complex host applications
+- Scaling path: introduce a more explicit runtime/handle ownership model only with careful API redesign
+
+## Dependencies at Risk
+
+**Pure FFI dependence on ONNX Runtime ABI stability:**
+- Risk: upstream ABI or packaging changes can break symbol registration, archive naming, or runtime expectations
+- Impact: bootstrap, initialization, or session creation failures across supported platforms
+- Migration plan: keep `internal/c_api/` snapshots, generator output, CI runtime versions, and bootstrap rules aligned when upgrading
+
+## Test Coverage Gaps
+
+**Python tooling is effectively outside the Go test matrix:**
+- What's not tested: `tools/openclip_export_onnx.py`, `tools/openclip_generate_golden.py`, and `tools/splade_generate_golden.py`
+- Risk: export or dataset-generation regressions may not be caught until maintainers run them manually
+- Priority: medium
+- Difficulty to test: requires Python environment setup, heavyweight ML dependencies, and artifact generation time
+
+**Release workflow behavior is mostly validated indirectly:**
+- What's not tested: end-to-end `.github/workflows/release.yml` behavior against real R2 credentials and signed artifact publishing
+- Risk: release-only failures can slip through normal PR CI
+- Priority: medium
+- Difficulty to test: depends on secrets, tag pushes, and external infrastructure
+
+---
+
+*Concerns audit: 2026-03-18*
+*Update as issues are fixed or new ones are discovered*

.planning/codebase/CONVENTIONS.md

@@ -0,0 +1,116 @@
+# Coding Conventions
+
+**Analysis Date:** 2026-03-18
+
+## Naming Patterns
+
+**Files:**
+- lower_snake_case for implementation files: `environment.go`, `bootstrap_lock_unix.go`, `golden_dataset_parity_test.go`
+- `*_test.go` for all test files, often with specialized suffixes like `*_integration_test.go` or `*_benchmark_test.go`
+- generated code is named explicitly with `_generated`: `ort/ortapi_generated.go`
+
+**Functions:**
+- Exported APIs use Go-standard PascalCase: `InitializeEnvironment`, `EnsureDefaultAssets`, `WithSequenceLength`
+- Unexported helpers use lowerCamelCase: `resolveRuntimeArtifact`, `validateFilePath`, `setupORTTestEnvironment`
+- Functional option constructors consistently start with `With` / `Without`
+
+**Variables:**
+- lowerCamelCase for locals and fields
+- package-level constants are mostly PascalCase when exported and lowerCamelCase when internal
+- all-caps names are reserved for compatibility constants or env-oriented identifiers such as `ORT_API_VERSION`
+
+**Types:**
+- PascalCase for structs, interfaces, and enums: `AdvancedSession`, `MemoryInfo`, `PoolingStrategy`
+- Package names stay short and lowercase: `ort`, `minilm`, `splade`, `openclip`
+
+## Code Style
+
+**Formatting:**
+- `gofmt` and `goimports` are the formatting source of truth (`.golangci.yml`, `Makefile`)
+- Standard Go formatting conventions apply: tabs, grouped imports, no manual alignment
+- Error strings are generally lowercase and sentence-fragment style
+
+**Linting:**
+- `go vet`, `golangci-lint`, and `gosec` are the main static checks
+- `precommit` in `Makefile` mirrors CI blockers, with opt-out env vars for local workflows
+- `.golangci.yml` keeps the rule set small and targeted instead of enabling everything
+
+## Import Organization
+
+**Order:**
+1. Go standard library imports
+2. Repository-local imports such as `github.com/amikos-tech/pure-onnx/ort`
+3. Third-party imports such as `github.com/ebitengine/purego`
+
+**Grouping:**
+- One blank line between import groups
+- No path aliases beyond short clarity-driven aliases like `tokenizers`
+- Side-effect imports are only used when required, for example image codecs in `examples/openclip/main.go`
+
+**Path Aliases:**
+- None; imports use full module paths
+
+## Error Handling
+
+**Patterns:**
+- Validate aggressively at function entry and return early on bad inputs
+- Wrap underlying failures with `fmt.Errorf(...: %w)`
+- Join cleanup failures with `errors.Join` where multiple destroy steps can fail
+- Native-handle wrappers usually treat nil receivers as safe no-ops on `Destroy()`
+
+**Error Types:**
+- Most code returns plain `error` rather than custom error structs
+- A few package-private sentinel or wrapper types exist where retry/permanence matters, such as `permanentBootstrapError` in `ort/bootstrap.go`
+- Tests assert on message content frequently, so wording changes can be user-visible
+
+## Logging
+
+**Framework:**
+- Standard library `log` package only
+
+**Patterns:**
+- Library code logs sparingly, mainly for warnings or bootstrap path visibility
+- Examples use `log.Fatal` / `log.Printf` for CLI-style behavior
+- There is no structured logging abstraction
+
+## Comments
+
+**When to Comment:**
+- Explain FFI safety assumptions, lock ordering, and lifetime rules
+- Document why `unsafe` usage is acceptable in specific places
+- Keep obvious code uncommented
+
+**Doc Comments:**
+- Exported types, constants, and functions are usually documented
+- Internal helpers receive comments when concurrency, lifecycle, or security behavior is non-obvious
+
+**Security/Lint Markers:**
+- `#nosec` annotations are used narrowly to justify intentional `unsafe` pointer conversions or checksum-like constants
+
+## Function Design
+
+**Size:**
+- Public APIs tend to be moderate-sized with early validation and deferred cleanup
+- The longest functions are concentrated in bootstrap/download code and embedder setup paths
+
+**Parameters:**
+- Constructors prefer explicit required parameters plus functional options
+- Helpers avoid large configuration structs at call sites unless state needs to persist
+
+**Return Values:**
+- APIs nearly always return `(value, error)` or `error`
+- Cleanup methods return `error` rather than panicking
+
+## Module Design
+
+**Exports:**
+- Packages expose focused public APIs and keep helpers unexported
+- `internal/` is used only where cross-package reuse should remain private
+
+**Generated Code:**
+- `ort/ortapi_generated.go` is treated as generated output; changes should come from `tools/gen_ortapi.go` and header inputs, not hand edits
+
+---
+
+*Convention analysis: 2026-03-18*
+*Update when patterns change*