|
| 1 | +## Repository Overview |
| 2 | + |
| 3 | +Elastic Beats is a Go monorepo containing lightweight data shippers for the Elastic Stack. The module path is `github.com/elastic/beats/v7`. Go version: see `.go-version`. |
| 4 | + |
| 5 | +## Build System |
| 6 | + |
| 7 | +The primary build tool is **Mage** (Go-based). Makefile wraps mage for some targets. Each beat has its own `magefile.go`; shared build logic lives in `dev-tools/mage/`. |
| 8 | + |
| 9 | +### Per-Beat Commands (run from the beat's directory, e.g. `cd filebeat`) |
| 10 | + |
| 11 | +```bash |
| 12 | +mage build # Build the beat binary |
| 13 | +mage unitTest # Run Go unit tests |
| 14 | +mage integTest # Run integration tests (requires Docker) |
| 15 | +mage goIntegTest # Go integration tests only |
| 16 | +mage pythonIntegTest # Python integration tests only |
| 17 | +mage update # Regenerate fields, configs, dashboards, includes |
| 18 | +mage fields # Regenerate fields.yml and fields.go |
| 19 | +mage config # Regenerate config files |
| 20 | +mage check # Run checks (lint, headers, go mod) |
| 21 | +mage docker:composeUp # Start integration test containers, wait until healthy |
| 22 | +mage docker:composeDown # Stop integration test containers |
| 23 | +mage crossBuild # Cross-compile for all platforms |
| 24 | +SNAPSHOT=true DEV=true PLATFORMS=$GOOS/$GOARCH PACKAGES=tar.gz mage package # Build and package the beat |
| 25 | +``` |
| 26 | + |
| 27 | +For `mage package`, set `PLATFORMS` to match the current OS/architecture. Check with `go env GOOS GOARCH` before running. Common values: `linux/amd64`, `linux/arm64`, `darwin/amd64`, `darwin/arm64`. `PACKAGES` sets the format: `tar.gz`, `zip`, `rpm`, `deb`, `docker` (comma-separated). |
| 28 | + |
| 29 | +### Running Tests |
| 30 | + |
| 31 | +**Do not run all unit or integration tests** (e.g. `mage unitTest` from root) unless explicitly requested. They take too long and require too many dependencies. Always run tests scoped to the package you're working on: |
| 32 | + |
| 33 | +```bash |
| 34 | +# Run a single test or package tests |
| 35 | +cd filebeat # or any beat directory |
| 36 | +go test -v -race -run TestName ./path/to/package/... |
| 37 | + |
| 38 | +# Stress test to find flaky tests (runs a test repeatedly with x/tools/cmd/stress) |
| 39 | +script/stresstest.sh [--tags integration] [--race] ./path/to/package ^TestName$ -p 32 |
| 40 | +``` |
| 41 | + |
| 42 | +### Integration Tests |
| 43 | + |
| 44 | +Integration tests require building the test binary and some need external dependencies |
| 45 | +that can be started with mage or manually. |
| 46 | + |
| 47 | +```bash |
| 48 | +# Build the system test binary if the beat defines it |
| 49 | +mage buildSystemTestBinary |
| 50 | + |
| 51 | +# Run integration tests for a specific package |
| 52 | +go test -v -race -run TestName -tags integration ./path/to/package/... |
| 53 | +``` |
| 54 | + |
| 55 | +### Running a Beat |
| 56 | + |
| 57 | +```bash |
| 58 | +DEV=true mage build |
| 59 | +./filebeat -e --strict.perms=false -c filebeat.yml -path.home=<temp_dir> |
| 60 | +``` |
| 61 | + |
| 62 | +Always pass `--strict.perms=false` during development — it disables strict config |
| 63 | +file permission checks. Set `-path.home` to a temporary directory to avoid |
| 64 | +writing data/logs into the source tree. |
| 65 | + |
| 66 | +### Pre-Push Check |
| 67 | + |
| 68 | +Before pushing, run `mage check` from the beat directory — it's a required CI check and catches lint, header, and module issues early. |
| 69 | + |
| 70 | +### Root-Level Commands |
| 71 | + |
| 72 | +```bash |
| 73 | +mage fmt # Format code + add license headers |
| 74 | +make check # Full check suite (lint, headers, go mod, python) |
| 75 | +``` |
| 76 | + |
| 77 | +### Linting |
| 78 | + |
| 79 | +```bash |
| 80 | +# Running golangci-lint for the whole codebase is slow, prefer running only on changed files by default |
| 81 | +golangci-lint run --max-issues-per-linter 0 --max-same-issues 0 --whole-files --new-from-merge-base upstream/main |
| 82 | +``` |
| 83 | + |
| 84 | +## Architecture |
| 85 | + |
| 86 | +### Beats |
| 87 | + |
| 88 | +| Beat | Description | |
| 89 | +|------|-------------| |
| 90 | +| `filebeat` | Lightweight agent that ships log files and journals | |
| 91 | +| `metricbeat` | Collects metrics from systems and services | |
| 92 | +| `heartbeat` | Monitors availability of services and endpoints, detecting uptime/downtime | |
| 93 | +| `auditbeat` | Gathers audit data from systems to track security events, user activities, and compliance requirements | |
| 94 | +| `packetbeat` | Analyzes network traffic by capturing and inspecting packets for application and network visibility | |
| 95 | +| `winlogbeat` | Collects Windows Event logs for centralized monitoring and troubleshooting of Windows systems | |
| 96 | +| `x-pack/osquerybeat` | Manages and queries Osquery endpoints (Elastic license) | |
| 97 | +| `x-pack/dockerlogbeat` | Ships Docker container logs as a Docker log driver plugin (Elastic license) | |
| 98 | +| `x-pack/otel` | OTel Collector components that wrap beats as receivers, processors, and exporters for the Elastic Distribution of OpenTelemetry (Elastic license) | |
| 99 | + |
| 100 | +Each beat follows the pattern: `cmd/` (CLI entry), `beater/` (implements `beat.Beater` interface from `libbeat/beat/beat.go`), `module/` or `input/` (data collection). |
| 101 | + |
| 102 | +### libbeat (shared framework) |
| 103 | + |
| 104 | +All beats build on `libbeat/`, which provides: |
| 105 | +- `beat/` — `Beater` interface (Run/Stop), `Beat` struct, `Pipeline`/`Client` for event publishing |
| 106 | +- `outputs/` — Output plugins (Elasticsearch, Logstash, Kafka, Redis, etc.) |
| 107 | +- `processors/` — Event processing pipeline |
| 108 | +- `publisher/` — Internal publishing pipeline |
| 109 | +- `cmd/` — Shared CLI infrastructure and beat instance bootstrapping |
| 110 | +- `autodiscover/` — Dynamic service discovery (Docker, Kubernetes) |
| 111 | +- `management/` — Elastic Agent management integration |
| 112 | +- `statestore/` — Persistent state |
| 113 | + |
| 114 | +### x-pack |
| 115 | + |
| 116 | +`x-pack/` contains Elastic-licensed extensions for each beat and x-pack-only beats. Each `x-pack/{beat}` extends the corresponding OSS beat with additional modules, inputs, or features. |
| 117 | + |
| 118 | +### Licensing Boundary |
| 119 | + |
| 120 | +OSS code (Apache 2.0) **cannot** import from `x-pack/` or `elastic-agent-client`. This is enforced by `depguard` in `.golangci.yml`. X-pack code can import OSS code freely. |
| 121 | + |
| 122 | +## Code Rules |
| 123 | + |
| 124 | +- Logging: accept `*logp.Logger` as a parameter. For tests prefer `logptest` |
| 125 | +- Use `github.com/stretchr/testify` in tests — always add a message explaining the failure. Use `assert` by default, `require` only when you actually need to interrupt the test |
| 126 | +- Use `github.com/gofrs/uuid/v5` |
| 127 | +- Write unit tests for new functionality. Use table-driven tests when appropriate |
| 128 | +- Document exported functions and types |
| 129 | +- Keep comments concise, focused on "why" not "what" |
| 130 | +- Update documentation when changing behavior |
| 131 | + |
| 132 | +## Code References |
| 133 | + |
| 134 | +When referencing code, always use exact locations and names: |
| 135 | +- Format: `filepath:line_number` or `filepath:start-end` |
| 136 | +- Use specific element names (functions, methods, variables) |
| 137 | +- Quote exact text from code and logs — zero paraphrasing |
| 138 | + |
| 139 | +## Changelog |
| 140 | + |
| 141 | +PRs require a changelog fragment in `changelog/fragments/` (CI enforced unless `skip-changelog` label is applied). Create using `elastic-agent-changelog-tool`: |
| 142 | + |
| 143 | +```bash |
| 144 | +go run github.com/elastic/elastic-agent-changelog-tool@latest new --component <beat> --kind <kind> |
| 145 | +``` |
| 146 | + |
| 147 | +Fragment format (`changelog/fragments/<timestamp>-<slug>.yaml`): |
| 148 | +```yaml |
| 149 | +kind: bug-fix # bug-fix, enhancement, breaking-change, deprecation, known-issue |
| 150 | +summary: Short description of the change |
| 151 | +component: filebeat # the affected beat/component |
| 152 | +``` |
| 153 | +
|
| 154 | +## Commits and PRs |
| 155 | +
|
| 156 | +When writing commit messages, explain WHAT changed and WHY — the rationale and motivation, not just a description of the diff. The commit title is used as the squash-merge message. Add a trailer line `Assisted-By: <tool name>` for each AI tool used. |
| 157 | + |
| 158 | +When creating PRs, follow the template in `.github/PULL_REQUEST_TEMPLATE.md` |
0 commit comments