[WIP] BBolt PoC with generative AI

.tmp-ai-io/model-selection-analysis.md

@@ -0,0 +1,205 @@
+# AI Model Selection for BBolt Registry Backend Task
+
+## Task Complexity Analysis
+
+### Task Overview
+- **Phase 1**: Implement bbolt backend (on-disk storage with GC)
+- **Phase 2**: Add in-memory hot storage layer with TTL-based GC
+- **Complexity**: High - requires understanding existing architecture, implementing new backend, GC logic, configuration parsing, comprehensive testing
+
+### Key Requirements
+1. Implement `backend.Registry` and `backend.Store` interfaces
+2. Add configuration parsing for `registry.type`, `registry.cache.ttl`, `registry.disk.ttl`
+3. Implement 2-layer caching (in-memory + bbolt on-disk)
+4. Background GC goroutines for both layers
+5. TTL-based expiration (access-time based)
+6. Make bbolt the default backend
+7. Comprehensive test coverage
+
+## Model Selection Criteria
+
+### Planning Phase Needs
+- **Codebase understanding**: Deep analysis of existing backend implementations (memlog, es)
+- **Architecture design**: 2-layer cache design, GC strategy, thread-safety
+- **Integration points**: Configuration parsing, initialization flow
+- **Risk assessment**: Migration path, backward compatibility
+
+### Execution Phase Needs
+- **Go expertise**: Idiomatic Go, error handling, concurrency (goroutines, mutexes)
+- **bbolt knowledge**: Database operations, transactions, bucket management
+- **Testing**: Unit tests, integration tests, compliance tests
+- **Code quality**: Following Beats patterns, proper error handling, logging
+
+## Available Models in cursor-agent CLI
+
+### Claude Models
+- `sonnet-4.5` - Claude Sonnet 4.5
+- `sonnet-4.5-thinking` - Claude Sonnet 4.5 (thinking mode)
+- `opus-4.5` - Claude Opus 4.5
+- `opus-4.5-thinking` - Claude Opus 4.5 (thinking mode)
+- `opus-4.1` - Claude Opus 4.1
+
+### GPT Models
+- `gpt-5.2` - GPT-5.2
+- `gpt-5.1` - GPT-5.1
+- `gpt-5.2-high` - GPT-5.2 (high capability)
+- `gpt-5.1-high` - GPT-5.1 (high capability)
+- `gpt-5.1-codex` - GPT-5.1 Codex
+- `gpt-5.1-codex-high` - GPT-5.1 Codex (high)
+- `gpt-5.1-codex-max` - GPT-5.1 Codex Max
+- `gpt-5.1-codex-max-high` - GPT-5.1 Codex Max (high)
+
+### Other Models
+- `gemini-3-pro` - Google Gemini 3 Pro
+- `gemini-3-flash` - Google Gemini 3 Flash
+- `grok` - Grok
+- `composer-1` - Composer model
+- `auto` - Auto-select model
+
+## Recommended Approach
+
+### Option 1: Single Model (Recommended)
+**Model**: `gpt-5.2` or `gpt-5.2-high`
+- **Strengths**: 
+  - Latest GPT model with excellent capabilities
+  - Strong Go knowledge and implementation skills
+  - Good at codebase analysis and architecture
+  - Can handle both planning and execution
+- **Use case**: Best balance for end-to-end task completion with context continuity
+
+### Option 2: Two-Model Approach (Recommended for Complex Tasks)
+**Planning Model**: `sonnet-4.5` or `sonnet-4.5-thinking`
+- **Why**: Superior at understanding complex codebases, architectural design, identifying edge cases
+- **Deliverable**: Detailed implementation plan, architecture diagram, file-by-file breakdown
+
+**Execution Model**: `gpt-5.2-high` or `gpt-5.1-codex-max-high`
+- **Why**: Strong Go implementation skills, excellent at following detailed plans, code-focused
+- **Deliverable**: Complete implementation with tests
+
+### Option 3: Specialized Models
+**Planning**: `sonnet-4.5-thinking` (best for deep architecture analysis)
+**Execution**: `gpt-5.1-codex-max-high` (strong Go implementation, excellent at following patterns)
+
+## Detailed Recommendation: Two-Model Approach
+
+### Phase 1: Planning (`sonnet-4.5` or `sonnet-4.5-thinking`)
+
+**Tasks**:
+1. Analyze existing backend implementations (`memlog`, `es`)
+2. Map configuration flow (`filebeat/beater/store.go`)
+3. Design bbolt backend structure
+4. Design GC mechanisms (in-memory + disk)
+5. Create implementation checklist
+6. Identify test requirements
+
+**Expected Output**:
+- Architecture document
+- File structure plan
+- Implementation steps with file locations
+- Test strategy
+- Configuration schema
+
+### Phase 2: Execution (`gpt-5.2-high` or `gpt-5.1-codex-max-high`)
+
+**Tasks**:
+1. Implement bbolt backend (`libbeat/statestore/backend/bbolt/`)
+2. Implement configuration parsing
+3. Implement GC goroutines
+4. Write tests (unit + compliance)
+5. Update initialization code
+6. Make bbolt default
+
+**Expected Output**:
+- Complete implementation
+- Test suite
+- Updated configuration handling
+
+## Why This Approach?
+
+### Planning Benefits
+- **Deep analysis**: Understanding memlog's checkpoint system, es backend patterns
+- **Design decisions**: TTL tracking strategy, GC intervals, thread-safety approach
+- **Risk mitigation**: Identifying edge cases before implementation
+
+### Execution Benefits
+- **Focused implementation**: Following detailed plan reduces errors
+- **Pattern consistency**: Matching existing codebase style
+- **Test coverage**: Comprehensive testing strategy
+
+## Alternative: Single Model Workflow
+
+If using one model:
+1. **First session**: Planning + initial implementation (bbolt backend only)
+2. **Review**: Test, validate Phase 1
+3. **Second session**: Add in-memory cache layer (Phase 2)
+
+## Model-Specific Notes
+
+### GPT-5.2 / GPT-5.2-high
+- **Best for**: General-purpose implementation, codebase analysis, comprehensive code generation
+- **Go skills**: Excellent
+- **Code quality**: High, good at matching existing style
+- **Recommendation**: Primary choice for both planning and execution
+
+### GPT-5.1 Codex Max / Codex Max-high
+- **Best for**: Code-focused implementation, following patterns, comprehensive code generation
+- **Go skills**: Excellent
+- **Code quality**: High, excellent at matching existing patterns
+- **Recommendation**: Best for execution phase when following detailed plans
+
+### Claude Sonnet 4.5 / Sonnet 4.5-thinking
+- **Best for**: Architecture, codebase analysis, design patterns, deep thinking
+- **Go skills**: Excellent
+- **Code quality**: High, follows best practices
+- **Recommendation**: Best for planning phase, thinking mode for complex architecture
+
+### Claude Opus 4.5
+- **Best for**: Complex problem-solving, advanced architecture
+- **Go skills**: Excellent
+- **Code quality**: Very high
+- **Recommendation**: Alternative to Sonnet 4.5 for planning
+
+### GPT-5.1 / GPT-5.1-high
+- **Best for**: General implementation, good balance of capability
+- **Go skills**: Excellent
+- **Code quality**: High
+- **Recommendation**: Alternative to GPT-5.2 if needed
+
+## Final Recommendation
+
+### Primary Recommendation: Single Model Approach
+**Use `gpt-5.2` or `gpt-5.2-high` for both phases**, with clear separation:
+
+1. **Planning session**: 
+   - "Analyze the codebase and create a detailed implementation plan for bbolt backend"
+   - Review plan before proceeding
+
+2. **Execution session**:
+   - "Implement the bbolt backend according to the plan"
+   - Iterate on implementation
+
+**Why**: GPT-5.2 is the latest model with excellent Go skills and can handle both codebase analysis and implementation. The two-phase approach ensures thorough planning before implementation, reducing refactoring needs.
+
+### Alternative: Two-Model Approach
+**Planning**: `sonnet-4.5-thinking` (superior architecture analysis)
+**Execution**: `gpt-5.2-high` or `gpt-5.1-codex-max-high` (strong implementation)
+
+**Why**: Leverages Claude's strength in architecture design and GPT's strength in code implementation. Best for complex tasks requiring deep analysis.
+
+### Usage Example
+```bash
+# Planning phase
+cursor-agent agent --model sonnet-4.5-thinking "Analyze the codebase and create a detailed implementation plan for bbolt backend"
+
+# Execution phase
+cursor-agent agent --model gpt-5.2-high "Implement the bbolt backend according to the plan"
+```
+
+## Key Files to Reference
+
+- `libbeat/statestore/backend/backend.go` - Interface definitions
+- `libbeat/statestore/backend/memlog/` - Reference implementation
+- `libbeat/statestore/backend/es/` - Alternative backend pattern
+- `filebeat/beater/store.go` - Backend initialization
+- `libbeat/statestore/registry.go` - Registry wrapper
+- `libbeat/statestore/internal/storecompliance/` - Compliance test framework

.tmp-ai-io/phase3-update-summary.md

@@ -0,0 +1,85 @@
+# Implementation Plan Update Summary
+
+## Changes Made
+
+Updated the BBolt registry backend implementation plan to include **Phase 3: Incremental GC Optimization**.
+
+### Three-Phase Approach
+
+**Phase 1: BBolt Backend (5-9 days)**
+- Basic bbolt backend with full-scan GC
+- Suitable for registries < 100K entries
+- GC time: ~1-2 seconds for 100K entries, ~7-13 seconds for 1M entries
+
+**Phase 2: In-Memory Cache (4-5 days)**
+- Two-layer caching system
+- Full-scan GC for both cache and disk
+- Performance improvement for hot data
+
+**Phase 3: Incremental GC (4-5 days)**
+- Batch-based incremental scanning
+- Cursor tracking across GC cycles
+- Adaptive strategy (auto-selects based on registry size)
+- Performance: ~0.5 seconds per 50K batch (vs 7-13s for 1M full scan)
+- Scalable to 10M+ entries
+
+### New Configuration Parameters
+
+```yaml
+filebeat.registry:
+  type: bbolt
+  bbolt:
+    disk_ttl: 30d
+    cache_ttl: 1h
+    gc_batch_size: 50000  # NEW: Entries per GC cycle
+```
+
+### New Files Added (Phase 3)
+
+- `gc_incremental.go` - Incremental GC implementation
+- `gc_adaptive.go` - Adaptive GC strategy selector
+
+### Key Additions
+
+1. **Incremental GC Algorithm**
+   - Cursor-based scanning
+   - Configurable batch size (default: 50K)
+   - Wrap-around logic when reaching end
+   - Full cycle tracking and logging
+
+2. **Adaptive Strategy**
+   - Auto-selects GC method based on registry size
+   - < 100K entries: full scan (simpler, fast enough)
+   - \> 100K entries: incremental scan (scalable)
+
+3. **Performance Metrics**
+   - GC duration logging
+   - Entries scanned/deleted tracking
+   - Full cycle completion reporting
+
+4. **Cache Incremental GC**
+   - Snapshot-based scanning for in-memory cache
+   - Batch processing of cache entries
+   - Position tracking
+
+### Timeline Impact
+
+- **Previous total: 9-14 days**
+- **New total: 13-19 days**
+- Phase 3 is **optional** - can be deferred if registries remain small
+
+### Decision Rationale
+
+Phase 3 addresses scalability concerns for large registries (1M+ entries) while keeping Phase 1 & 2 simple. The incremental GC:
+- Reduces GC pause time from 7-13s to ~0.5s per batch
+- Enables horizontal scalability
+- Provides smooth, predictable performance
+- Can be enabled/disabled via configuration
+
+### Documentation Updates
+
+- GC performance analysis section added
+- Configuration examples updated
+- Sizing guidelines documented
+- Success criteria expanded
+- Timeline adjusted

.tmp-ai-io/pr-bbolt-registry.md

@@ -0,0 +1,39 @@
+## Summary
+- Add new `bbolt` statestore backend under `libbeat/statestore/backend/bbolt/` (registry + store + metadata + full-scan TTL GC).
+- Make **bbolt the default Filebeat registry backend** via `filebeat.registry.type` (still supports `memlog`).
+- Add a Filebeat integration test to ensure **registry state is persisted to bbolt on shutdown** (even with `flush: 24h`).
+- Update OSS + x-pack Filebeat tests/templates that assume `registry/filebeat/log.json` to explicitly use `memlog`.
+
+## Details
+- **New backend package**: `libbeat/statestore/backend/bbolt/`
+  - `registry.go`: registry implementation, per-store DB files, background disk GC ticker (interval = `disk_ttl`).
+  - `store.go`: `backend.Store` implementation (CRUD + `Each`), JSON encoding, metadata updates on Get/Set, `slices.Clone` for bbolt value copies, injectable clock (`now func() time.Time`).
+  - `gc.go`: Phase 1/2 full-scan GC based on `metadata.last_access` vs `disk_ttl`, logs duration/scanned/deleted.
+  - `doc.go`, `error.go`.
+  - Debug helpers (read-only accessors) added in later edits: `Registry.GetDB(...)`, `store.DB()`.
+- **Filebeat config**: `filebeat/config/config.go`
+  - `filebeat.registry.type` (default **`bbolt`**).
+  - `filebeat.registry.bbolt.*` settings (disk TTL, timeout, bbolt options, etc.).
+  - `Registry.ValidateConfig()` used by the beater to validate `type` and config values without triggering validation during `Unpack`.
+  - Debug config added in later edits: `filebeat.registry.debug_port` (default 8000).
+- **Backend selection**: `filebeat/beater/store.go`
+  - Select between `bbolt` (default) and `memlog`.
+  - Debug accessors added in later edits: keep `*bbolt.Registry` on `filebeatStore` + `BBoltRegistry()` getter.
+- **Debug webserver (note)**: `filebeat.registry.debug_port` was introduced (default `8000`) to enable a read-only web interface for inspecting the registry during development/troubleshooting. Intended for local/debug use only.
+- **Tests**
+  - New test: `filebeat/tests/integration/bbolt_registry_shutdown_test.go` verifies bbolt DB contains filestream state after shutdown.
+  - Existing tests/templates that rely on memlog’s `registry/filebeat/log.json` are pinned to `memlog`:
+    - OSS: `filebeat/tests/integration/*`, `filebeat/testing/integration/*`, `filebeat/tests/system/config/*.yml.j2`, etc.
+    - x-pack: `x-pack/filebeat/tests/system/config/filebeat_modules.yml.j2`, `x-pack/filebeat/tests/integration/registrydiagnostics_test.go`, `x-pack/filebeat/fbreceiver/receiver_test.go`.
+
+## Test plan
+- Unit tests:
+  - `go test ./libbeat/statestore/backend/...`
+  - `go test ./filebeat/config`
+  - `go test ./filebeat/beater`
+- Build:
+  - `go build .` (from `filebeat/`)
+- Integration (requires `filebeat/filebeat.test`):
+  - `go test -c ./filebeat -o filebeat/filebeat.test`
+  - `go test -tags=integration ./filebeat/tests/integration -run TestBBoltRegistrySyncedOnShutdown -count=1`
+