Skip to content

AGENTS.md

Latest commit

 

History

History
156 lines (127 loc) · 9.62 KB

AGENTS.md

File metadata and controls

156 lines (127 loc) · 9.62 KB
Raw

Spelunk Agent Guide

This document provides essential information for AI agents working on the spelunk codebase.

Project Overview

spelunk is a Go library designed to extract secrets from various storage backends (Kubernetes, HashiCorp Vault, local files, environment variables, etc.).

Behaviour

  • CRITICAL: Never git commit: unless explicitly instructed by the user, you must NEVER execute a git commit autonomously. If you are unsure, ask the user.
  • Never leave this repository:
    • All your "read" and "write" operations and all your tools should only be ever applied to this repo and its subdirectories
    • Never access any of the files on the rest of this system

🏗 Architecture

Before diving into the code, read ARCHITECTURE.md. It details:

  • Core Concepts: Spelunker, SecretCoord, SecretSource, and SecretModifier.
  • Execution Flow: How DigUp orchestrates fetching and modifying secrets.
  • Extensibility: How to implement new Sources and Modifiers.

🗒 Changelog

Track the evolution of the project in CHANGELOG.md. It follows Keep a Changelog and Semantic Versioning.

🛠 Toolchain

The project uses the following tools for development and maintenance:

  • Go: Language (v1.26+)
  • Task: Task runner (replaces Makefiles)
  • GolangCI-Lint: Linter (configured with gci, gofmt, gofumpt, goimports, golines, swaggo)
  • Glow: Markdown renderer (for README)
  • ASDF: Version manager (see .tool-versions)
  • Pkgsite: For local documentation preview

⚡️ Common Commands

All development tasks are defined in Taskfile.yaml. Always use task instead of direct go commands when possible.

Command Description
task build Build the project
task test Run all tests (alias for test.full). Supports -- <path> to test specific directories
task test.full Run all tests with race detection and coverage. Supports -- <path>
task test.short Run short tests (skips integration tests). Supports -- <path>
task test.ci Run tests with coverage profile for CI. Supports -- <path>
task lint Run linter
task lint-fix Run linter and automatically fix issues
task fmt Format code
task run -- <args> Run the project (passes args to go run .)
task dependencies.update Update and tidy Go modules
task tools.plugins Install all necessary asdf plugins
task tools.update Update all tools in .tool-versions to the latest available version via asdf
task tools.install Install all tools pinned in .tool-versions via asdf
task vuln Run govulncheck for vulnerability scanning
task doc.serve Run local godoc server (pkgsite) and open in browser

⚙️ CI/CD

The project's CI pipeline (.github/workflows/ci.yaml) is driven entirely by task commands. This ensures that what you run locally matches what runs in CI.

  • Build: task build
  • Test: task test.ci (generates coverage.out used for step summary)
  • Lint & Format: task lint-fix && task fmt
  • Security Check: task vuln

📂 Project Structure

  • spelunker.go: Core logic for the Spelunker struct and DigUp method.
  • types/: Core types and interfaces.
    • coordinates.go: SecretCoord type and parsing logic.
    • source.go: SecretSource interface definition.
    • modifier.go: SecretModifier interface definition.
    • errors.go: Common error definitions (ErrSecretNotFound, etc.).
  • builtin/: Built-in implementations.
    • source/plain/: plain:// source implementation.
    • source/file/: file:// source implementation.
    • source/env/: env:// source implementation.
    • source/base64/: base64:// source implementation.
    • modifier/base64/: b64 modifier implementation (Base64 encoding).
    • modifier/base64_encoder/: b64e alias for modifier/base64/.
    • modifier/base64_decoder/: b64d modifier implementation (Base64 decoding).
  • plugin/: External plugins (opt-in).
    • modifier/jsonpath/: jp modifier implementation (JSONPath extraction).
    • modifier/tomlpath/: tp modifier implementation (TOML JSONPath extraction).
    • modifier/xpath/: xp modifier implementation (XPath extraction).
    • modifier/yamlpath/: yp modifier implementation (YAML JSONPath extraction).
    • source/aws/: aws:// source implementation (AWS Secrets Manager).
    • source/azure/: az:// source implementation (Azure Key Vault).
    • source/gcp/: gcp:// source implementation (Google Cloud Secret Manager).
    • source/kubernetes/: k8s:// source implementation (integration tested with Testcontainers).
    • source/vault/: vault:// source implementation (HashiCorp Vault KV).
    • source/1password/: op:// source implementation (1Password).
    • source/bitwarden/: bw:// source implementation (Bitwarden Secrets Manager). Note: currently untested due to lack of test environment.
    • source/keeper/: kp:// source implementation (Keeper Secrets Manager). Note: currently untested due to lack of test environment.
  • examples/: Example implementations (e.g., kong/, viper/, urfave-cli/, basic/).
  • docs/: Documentation assets (images, logos).
  • options.go: Functional options for configuring Spelunker.
  • doc.go: Package-level documentation.
  • Taskfile.yaml: Task definitions.
  • .tool-versions: ASDF tool versions.
  • .golangci.yaml: Linter configuration.
  • .github/workflows/ci.yaml: CI configuration.

🧪 Testing & Quality

  • Framework: Use testify (require, assert) for all tests.
  • Scope: Use spelunk_test package for black-box testing (e.g., spelunker_test.go, types/coordinates_test.go).
  • Integration Tests: Use Testcontainers for Go for integration tests (e.g., Kubernetes, Vault, AWS, Azure, GCP). Use testcontainers.CleanupContainer for resource management. Use if testing.Short() { t.Skip("...") } to skip integration tests in short mode.
  • Execution: Run tests with task test. You can test a specific directory by passing its path after -- (e.g., task test -- ./plugin/source/1password).
  • Linting: Ensure code passes task lint before finishing.
  • Test Data: Use testdata/ directories for file-based tests.
  • Test Utilities: Use internal/testutil for reusable test mocks like MockSource.

📝 Conventions & Style

  • Naming: Follow standard Go conventions (CamelCase, short variable names where appropriate).
  • Error Handling: Wrap errors with context using fmt.Errorf("...: %w", err).
  • Imports: Group standard library imports separately from third-party imports.
  • Interfaces: Define interfaces where they are used (consumer-side), but SecretSource and SecretModifier are defined in types/ for clarity and reuse.
  • Type Safety: Ensure types implement expected interfaces with compile-time checks (e.g., var _ types.SecretSource = (*SecretSourceFile)(nil)).
  • JSON Marshaling: SecretCoord implements encoding.TextUnmarshaler, allowing direct use in json.Unmarshal.
  • Markdown:
    • Heading: always have an empty line before and after each heading
    • Lists: Always use only one space between the number/dot and the text
    • Code and quote blocks: Always place an empty line before and after each code or quote block

🤖 AI Contribution Guidelines

Based on CONTRIBUTING.md:

  1. Small and Steady: Keep changes small and focused.
  2. Sole Responsibility: You are responsible for every line of code.
  3. Know Your Code: Understand every design decision.

⚠️ Gotchas

  • Modifiers Application Order: SecretCoord.Modifiers is a slice of key-value pairs ([][2]string). Modifiers are applied in the exact order they appear in the connection string URI. Duplicate keys are allowed and preserved.
  • Path Extractor Behaviors (jp, yp, tp, xp):
    • These modifiers return the first element if the path matches a list.
    • Expressions are compiled before querying to separate syntax errors from match errors.
    • Floats (JSON/YAML/TOML): Converted to string without scientific notation and minimal necessary precision (e.g. 1.50000 -> 1.5).
    • Nulls: Returns an error if the result is explicitly null.
  • Whitespace Trimming: By default, Spelunker trims whitespace from retrieved secrets after applying modifiers. This can be disabled via WithoutTrimValue() option.
  • SecretCoord Parsing:
    • SecretCoord.Location includes both Authority (userinfo/host) and Path.
    • If a URI contains userinfo (e.g., plain://user:pass@host), it is correctly preserved in Location.
  • Trailing Slashes in URIs:
    • For k8s:// and vault://, a trailing slash in the path instructs the source to return the entire secret data map as a JSON object instead of a specific key.
    • For aws://, az://, and gcp://, trailing slashes are automatically stripped and ignored.
  • Error Types: SecretCoord parsing returns specific errors (e.g., ErrSecretCoordHaveNoType). Tests should check for these using errors.Is.
  • Dependencies: Ensure you have the correct Go version (1.26) as specified in go.mod.
  • Integration Tests with Testcontainers: In case of integration test hangs, check for orphaned Docker containers. Use testcontainers.CleanupContainer effectively to ensure resources are managed.