Skip to content

CLAUDE.md

Latest commit

 

History

History
111 lines (74 loc) · 3.87 KB

CLAUDE.md

File metadata and controls

111 lines (74 loc) · 3.87 KB
Raw

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

Project Overview

lindb/arrow is a Go library providing high-performance columnar storage for OpenTelemetry observability signals (Logs, Metrics, Traces) using Apache Arrow. It bridges OTel data models and columnar storage for LinDB analytics.

Module: github.com/lindb/arrow Go version: 1.25.0 License: Apache 2.0

Commands

# Run all tests
go test ./...

# Run tests for a specific package
go test ./pkg/traces/...

# Run a single test
go test ./pkg/traces/... -run TestSpanBuilder

# Run tests with verbose output
go test -v ./pkg/logs/...

# Update dependencies
make deps   # runs go mod verify && go mod tidy -v

# Check license headers
make header  # uses go-licenser with Apache 2.0

Architecture

The library converts OTel data into multi-record-batch Arrow representations. Data is split into normalized tables with foreign key references, similar to a columnar relational model.

Package Map

pkg/
├── arrow/           # Core utilities: builders, readers, serializers, extension types
│   └── array/       # Custom Arrow extension arrays (TimeSeries, Aggregation, Exemplar)
├── attributes/      # Attribute key-value store with deduplication via uint32 foreign keys
├── constants/       # Field names (Timestamp, TraceID, etc.) and DataType enum
├── logs/            # Log record builder + reader (wraps OTel plog)
├── metrics/         # Metric types (Exemplar model)
├── model/           # Internal pool-managed wrappers: model.Log, model.Span
├── pages/           # Page handling utilities
└── traces/          # Span builder + reader (wraps OTel ptrace)

Data Flow

Write path: OTel data → model.Log/model.Span → Builder → []arrow.RecordBatch → IPC bytes Read path: IPC bytes → BinaryReader → []arrow.RecordBatch → Reader → field accessors

Multi-Batch Schema Pattern

Trace data is stored as 6 related record batches in a fixed order defined by TraceSchema:

[SpanSchema, AttributeSchema, ResourceSchema, ScopeSchema, EventSchema, LinkSchema]

Logs similarly separate attributes into a normalized batch. Each signal type has its schema defined under pkg/<signal>/v1/schema.go.

Attribute Normalization

Attributes are deduplicated into a separate table (AttributeSchema). Main tables reference attributes via uint32 foreign key lists. The AttributeBuilder maintains an internal map for deduplication; AttributeID() generates stable IDs from key-value pairs.

Key Interfaces

EntryBuilder[V EntryType] in pkg/arrow is the generic builder interface used by both LogBuilder and SpanBuilder:

type EntryType interface { *model.Span | *model.Log }

type EntryBuilder[V EntryType] interface {
    Append(V)
    Build() []arrow.RecordBatch
    Schemas() []*arrow.Schema
    NumOfRows() int
    Bytes() int
    Release()
}

FilterableRecord

pkg/arrow/filter.go provides lazy filtering on record batches via a bitmask — no data is copied until necessary.

Custom Extension Types

Registered Arrow extension types in pkg/arrow/array/:

  • TimeSeries ("time_series") — start, end, interval, values
  • Aggregation — per-kind extensions: Sum, Min, Max, Last, First
  • Exemplar

Object Pooling

model.Log and model.Span use sync.Pool via GetLog()/PutLog() to reduce GC pressure. Always return instances after use.

Field Index Caching

Readers cache Arrow schema field indexes at construction time (indexes struct fields) to avoid repeated schema.FieldIndex() calls in hot paths.

Versioning

Schema definitions live under pkg/<signal>/v1/schema.go. New schema versions go in a new v2/ subdirectory. Metadata on each schema carries MetadataNameKey and MetadataVersionKey (from pkg/constants).