Skip to content

Commit 6fbb76a

Browse files
tazarovtazarov
committed
Initial commit: embedded Chroma Go shim with tests and benchmarks
0 parents  commit 6fbb76a

27 files changed

Lines changed: 14749 additions & 0 deletions

‎.gitignore‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
# OS artifacts
2+
.DS_Store
3+
4+
# Editor/IDE
5+
.idea/
6+
.vscode/
7+
8+
# Go test/coverage artifacts
9+
coverage.out
10+
*.coverprofile
11+
*.test
12+
*.prof
13+
14+
# Rust/Cargo build artifacts
15+
target/
16+
shim/target/
17+
18+
# Generated local Chroma runtime data
19+
chroma_test_data/
20+
chroma_test_data_builder/
21+
chroma_test_data_embedded/
22+
23+
# Proptest regression files (local debug artifacts)
24+
shim/proptest-regressions/
25+
26+
# Binary artifacts
27+
*.dylib
28+
*.so
29+
*.dll
30+
*.a
31+
*.o
32+
*.rlib

‎.golangci.yml‎

Lines changed: 64 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,64 @@
1+
version: "2"
2+
run:
3+
modules-download-mode: readonly
4+
timeout: 30m
5+
concurrency: 8
6+
7+
linters:
8+
enable:
9+
- dupword
10+
- ginkgolinter
11+
- gocritic
12+
- mirror
13+
settings:
14+
gocritic:
15+
disable-all: false
16+
staticcheck:
17+
checks:
18+
- all
19+
- ST1000
20+
- ST1001
21+
- ST1003
22+
dot-import-whitelist:
23+
- fmt
24+
exclusions:
25+
generated: lax
26+
presets:
27+
- comments
28+
- common-false-positives
29+
- legacy
30+
- std-error-handling
31+
rules:
32+
- linters:
33+
- ineffassign
34+
path: conversion\.go
35+
- linters:
36+
- staticcheck
37+
text: 'ST1003: should not use underscores in Go names; func (Convert_.*_To_.*|SetDefaults_)'
38+
- linters:
39+
- ginkgolinter
40+
text: use a function call in (Eventually|Consistently)
41+
paths:
42+
- third_party$
43+
- builtin$
44+
- examples$
45+
issues:
46+
max-same-issues: 0
47+
formatters:
48+
enable:
49+
- gci
50+
settings:
51+
gci:
52+
sections:
53+
- standard
54+
- default
55+
- prefix(github.com/chaoslabs-bg/tclr-v2/)
56+
- blank
57+
- dot
58+
custom-order: true
59+
exclusions:
60+
generated: lax
61+
paths:
62+
- third_party$
63+
- builtin$
64+
- examples$

‎AGENTS.md‎

Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,53 @@
1+
# AGENTS.md
2+
3+
Guidance for coding agents working in this repository.
4+
5+
## Project Summary
6+
7+
- `local-go-chroma` is a Go wrapper for running Chroma as an embedded server.
8+
- It uses a Rust FFI shim plus `purego` (no `cgo`).
9+
- Primary languages: Go and Rust.
10+
11+
## Requirements
12+
13+
- Go 1.21+
14+
- Rust 1.70+
15+
- `golangci-lint` for Go linting
16+
17+
## Common Commands
18+
19+
- Build debug shim: `make build`
20+
- Build release shim: `make build-release`
21+
- Run tests (debug): `make test`
22+
- Run tests (release): `make test-release`
23+
- Run linters: `make lint`
24+
- Format code: `make fmt`
25+
26+
Notes:
27+
- `make test` and `make test-release` set `CHROMA_LIB_PATH` automatically.
28+
- Prefer Make targets over ad-hoc commands for reproducibility.
29+
30+
## Code Map
31+
32+
- `chroma.go`: server lifecycle and public Go API
33+
- `config.go`: server config and builder options (`With...`)
34+
- `library.go`: dynamic library loading and symbol binding via `purego`
35+
- `errors.go`: error handling types and codes
36+
- `shim/src/lib.rs`: Rust FFI exports and runtime-backed server operations
37+
- `chroma_test.go`: integration-style tests against real server instances
38+
39+
## Implementation Rules
40+
41+
- Preserve the no-`cgo` design.
42+
- Keep Go and Rust FFI contracts in sync when changing signatures.
43+
- Maintain resource cleanup behavior (`Stop`, `Close`, and finalizers).
44+
- Keep public API changes backward compatible unless explicitly requested.
45+
- Add or update tests for behavior changes.
46+
47+
## Validation Before Handoff
48+
49+
- Run relevant checks for touched areas:
50+
- `make test`
51+
- `make lint`
52+
53+
If a full run is not possible, document exactly what was not executed and why.

‎CLAUDE.md‎

Lines changed: 75 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,75 @@
1+
# CLAUDE.md
2+
3+
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
4+
5+
## Project Overview
6+
7+
A minimal Go wrapper for running Chroma (vector database) as an embedded server using a Rust FFI shim and purego (no cgo required).
8+
9+
## Requirements
10+
11+
- Go 1.21+
12+
- Rust 1.70+
13+
- golangci-lint (for linting)
14+
15+
## Build Commands
16+
17+
```bash
18+
make build # Build Rust shim (debug)
19+
make build-release # Build Rust shim (release)
20+
make test # Build debug + run Go tests
21+
make test-release # Build release + run Go tests
22+
make lint # Run all linters (Go + Rust)
23+
make fmt # Format all code (Go + Rust)
24+
make clean # Clean build artifacts
25+
```
26+
27+
## Testing
28+
29+
Tests require the Rust shim to be built first. The Makefile handles this automatically:
30+
- `make test` builds debug and runs tests
31+
- `make test-release` builds release and runs tests
32+
- `CHROMA_LIB_PATH` is set automatically by Makefile
33+
34+
Tests are integration tests that start actual servers and make HTTP requests.
35+
36+
## Architecture
37+
38+
```
39+
Go Package (chroma/) Rust Shim (shim/)
40+
├── chroma.go ─────────► src/lib.rs (FFI exports)
41+
│ (Server lifecycle) ├── chroma_server_start
42+
├── config.go ├── chroma_server_stop
43+
│ (Builder pattern) ├── chroma_server_port
44+
├── library.go ├── chroma_server_address
45+
│ (purego FFI loading) └── ...
46+
└── errors.go
47+
(Error codes)
48+
```
49+
50+
- **No cgo**: Uses purego for pure Go FFI
51+
- **Tokio runtime**: Managed per Server instance in Rust
52+
- **Configuration**: YAML-based with environment overrides (CHROMA_ prefix)
53+
- **Resource cleanup**: Go runtime finalizers for server instances
54+
55+
## Key Patterns
56+
57+
Builder pattern for configuration:
58+
```go
59+
server, err := chroma.NewServer(
60+
chroma.WithPort(8000),
61+
chroma.WithPersistPath("./chroma_data"),
62+
)
63+
```
64+
65+
YAML string config alternative:
66+
```go
67+
server, err := chroma.StartServer(chroma.StartServerConfig{
68+
ConfigString: yamlString,
69+
})
70+
```
71+
72+
## Linting
73+
74+
- Go: `golangci-lint run ./...` (config in `.golangci.yml`)
75+
- Rust: `cargo clippy -- -D warnings` (warnings as errors)

‎EMBEDDED_PARITY_MATRIX.md‎

Lines changed: 74 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,74 @@
1+
# Embedded API Parity Matrix
2+
3+
Scope: parity between Chroma capabilities and this repo's embedded (in-process) API.
4+
5+
Legend:
6+
- `done`: implemented in embedded mode
7+
- `in-progress`: currently being implemented
8+
- `planned`: not yet implemented
9+
- `out-of-scope`: intentionally not exposed in current embedded Go surface
10+
11+
## System and Lifecycle
12+
13+
| Capability | Embedded Status | Notes |
14+
|---|---|---|
15+
| Init library | done | `Init` |
16+
| Start embedded from YAML path/string | done | `StartEmbedded` / `NewEmbedded` |
17+
| Close embedded handle | done | `(*Embedded).Close()` |
18+
| Heartbeat | done | `(*Embedded).Heartbeat()` |
19+
| Max batch size | done | `(*Embedded).MaxBatchSize()` |
20+
| Reset | done | `(*Embedded).Reset()` |
21+
22+
## Databases
23+
24+
| Capability | Embedded Status | Notes |
25+
|---|---|---|
26+
| Create database | done | `(*Embedded).CreateDatabase()` |
27+
| List databases | done | `(*Embedded).ListDatabases()` |
28+
| Get database | done | `(*Embedded).GetDatabase()` |
29+
| Delete database | done | `(*Embedded).DeleteDatabase()` |
30+
31+
## Collections
32+
33+
| Capability | Embedded Status | Notes |
34+
|---|---|---|
35+
| Create collection | done | `(*Embedded).CreateCollection()` |
36+
| List collections | done | `(*Embedded).ListCollections()` |
37+
| Get collection | done | `(*Embedded).GetCollection()` |
38+
| Count collections | done | `(*Embedded).CountCollections()` |
39+
| Update collection | done | `(*Embedded).UpdateCollection()` (rename-focused) |
40+
| Delete collection | done | `(*Embedded).DeleteCollection()` |
41+
| Fork collection | done | `(*Embedded).ForkCollection()` (may return unimplemented on local Chroma backend) |
42+
43+
## Records and Query
44+
45+
| Capability | Embedded Status | Notes |
46+
|---|---|---|
47+
| Add records | done | `(*Embedded).Add()` |
48+
| Query records | done | `(*Embedded).Query()` |
49+
| Get records | done | `(*Embedded).GetRecords()` |
50+
| Count records | done | `(*Embedded).CountRecords()` |
51+
| Update records | done | `(*Embedded).UpdateRecords()` |
52+
| Upsert records | done | `(*Embedded).UpsertRecords()` |
53+
| Delete records | done | `(*Embedded).DeleteRecords()` (ids and/or filters) |
54+
| Query/get/delete filters (`where`, `where_document`) | done | Supported in query/get/delete record calls |
55+
56+
## Tenants and Admin
57+
58+
| Capability | Embedded Status | Notes |
59+
|---|---|---|
60+
| Create tenant | done | `(*Embedded).CreateTenant()` |
61+
| Get tenant | done | `(*Embedded).GetTenant()` |
62+
| Update tenant | done | `(*Embedded).UpdateTenant()` (resource visibility may depend on backend) |
63+
| Healthcheck | done | `(*Embedded).Healthcheck()` |
64+
| Indexing status | done | `(*Embedded).IndexingStatus()` (may return unimplemented on local backend) |
65+
66+
## Advanced and Function Management
67+
68+
| Capability | Embedded Status | Notes |
69+
|---|---|---|
70+
| Search API (`search`) | planned | Distinct from nearest-neighbor `query` |
71+
| Get collection by CRN | out-of-scope | Internal/advanced lookup not in current Go surface |
72+
| Attach function | out-of-scope | Upstream `attach_function` not yet bridged |
73+
| Get attached function | out-of-scope | Upstream `get_attached_function` not yet bridged |
74+
| Detach function | out-of-scope | Upstream `detach_function` not yet bridged |

0 commit comments

Comments
 (0)