[WIP] BBolt PoC with generative AI

[belimawr]

Summary

This draft PR is an experiment of using generative AI to write code in Beats. Cursor as used with a variety of models and using AGENTS.md as the configuration/guide for AI agents.

The description of the tasks are in ai-tasks. Using them as a staring point, gen AI was used to generate a detailed implementation plan, those plans are in .tmp-ai-io, the key files to read:

• model-selection-analysis.md is the summary of choosing the best models for planning and executing the task.
• bbolt-implementation-plan.md is the implementation plan for ai-tasks/bbolt-registry.md
• gc-performance-analysis.md is a performance analysis
• phase3-update-summary.md contains the plan to implement an incremental GC as suggested by the performance analysis

Known issues

• The updated field from the registry entry is not being serialised in the correct format. That's why TestJournaldInputDoesNotDuplicateData is failing.

---

AI Generated PR description/summary

Summary

• Add new bbolt statestore backend under libbeat/statestore/backend/bbolt/ (registry + store + metadata + full-scan TTL GC).
• Make bbolt the default Filebeat registry backend via filebeat.registry.type (still supports memlog).
• Add a Filebeat integration test to ensure registry state is persisted to bbolt on shutdown (even with flush: 24h).
• Update OSS + x-pack Filebeat tests/templates that assume registry/filebeat/log.json to explicitly use memlog.

Details

• New backend package: libbeat/statestore/backend/bbolt/
  • registry.go: registry implementation, per-store DB files, background disk GC ticker (interval = disk_ttl).
  • store.go: backend.Store implementation (CRUD + Each), JSON encoding, metadata updates on Get/Set, slices.Clone for bbolt value copies, injectable clock (now func() time.Time).
  • gc.go: Phase 1/2 full-scan GC based on metadata.last_access vs disk_ttl, logs duration/scanned/deleted.
  • doc.go, error.go.
  • Debug helpers (read-only accessors) added in later edits: Registry.GetDB(...), store.DB().
• Filebeat config: filebeat/config/config.go
  • filebeat.registry.type (default bbolt).
  • filebeat.registry.bbolt.* settings (disk TTL, timeout, bbolt options, etc.).
  • Registry.ValidateConfig() used by the beater to validate type and config values without triggering validation during Unpack.
  • Debug config added in later edits: filebeat.registry.debug_port (default 8000).
• Backend selection: filebeat/beater/store.go
  • Select between bbolt (default) and memlog.
  • Debug accessors added in later edits: keep *bbolt.Registry on filebeatStore + BBoltRegistry() getter.
• Debug webserver (note): filebeat.registry.debug_port was introduced (default 8000) to enable a read-only web interface for inspecting the registry during development/troubleshooting. Intended for local/debug use only.
• Tests
  • New test: filebeat/tests/integration/bbolt_registry_shutdown_test.go verifies bbolt DB contains filestream state after shutdown.
  • Existing tests/templates that rely on memlog’s registry/filebeat/log.json are pinned to memlog:
    • OSS: filebeat/tests/integration/*, filebeat/testing/integration/*, filebeat/tests/system/config/*.yml.j2, etc.
    • x-pack: x-pack/filebeat/tests/system/config/filebeat_modules.yml.j2, x-pack/filebeat/tests/integration/registrydiagnostics_test.go, x-pack/filebeat/fbreceiver/receiver_test.go.

Test plan

• Unit tests:
  • go test ./libbeat/statestore/backend/...
  • go test ./filebeat/config
  • go test ./filebeat/beater
• Build:
  • go build . (from filebeat/)
• Integration (requires filebeat/filebeat.test):
  • go test -c ./filebeat -o filebeat/filebeat.test
  • go test -tags=integration ./filebeat/tests/integration -run TestBBoltRegistrySyncedOnShutdown -count=1

---

Proposed commit message

TBD

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works. Where relevant, I have used the stresstest.sh script to run them under stress conditions and race detector to verify their stability.
• I have added an entry in ./changelog/fragments using the changelog tool.

Disruptive User Impact

Author's Checklist

• [ ]

How to test this PR locally

Manual test

1. Create a log file

docker run -it --rm mingrammer/flog -n 50 > /tmp/flog.log

2. Create a configuration file and start Filebeat

Adjust the following configuration file to match the log file you created

filebeat.yml

filebeat.inputs:
- type: filestream
  id: unique-id
  paths:
    - /tmp/flog.log

output.discard:
  enabled: true

registry:
  flush: 0

logging:
  to_stderr: true
  level: debug

3. Check the debug endpoint

In your browser open http://localhost:8000/. You will see a debug UI showing the registry entries. Wait a few seconds and you will see the cursor correctly updated.

Debug UI screenshot

## Related issues
## Use cases
## Screenshots
## Logs

[github-actions]

🤖 GitHub comments

Just comment with:

• run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

[mergify]

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @belimawr? 🙏.
For such, you'll need to label your PR with:

• The upcoming major version of the Elastic Stack
• The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit
• backport-active-all is the label that automatically backports to all active branches.
• backport-active-8 is the label that automatically backports to all active minor branches for the 8 major.
• backport-active-9 is the label that automatically backports to all active minor branches for the 9 major.

[belimawr]

This was a good PoC, now we have BBolt in Beats, closing.