Skip to content

AGENTS.md

Latest commit

 

History

History
164 lines (136 loc) · 7.8 KB

AGENTS.md

File metadata and controls

164 lines (136 loc) · 7.8 KB
Raw

Apache DevLake - AI Coding Agent Instructions

Project Overview

Apache DevLake is a dev data platform that ingests data from DevOps tools (GitHub, GitLab, Jira, Jenkins, etc.), transforms it into standardized domain models, and enables metrics/dashboards via Grafana.

Architecture

Three-Layer Data Model

  1. Raw Layer (_raw_* tables): JSON data collected from APIs, stored for replay/debugging
  2. Tool Layer (_tool_* tables): Plugin-specific models extracted from raw data
  3. Domain Layer (standardized tables): Normalized models in backend/core/models/domainlayer/ - CODE, TICKET, CICD, CODEREVIEW, CODEQUALITY, CROSS

Key Components

  • backend/: Go server + plugins (main codebase)
  • backend/python/: Python plugin framework via RPC
  • config-ui/: React frontend (TypeScript, Vite, Ant Design)
  • grafana/: Dashboard definitions

Plugin Development (Go)

Plugin Structure

Each plugin in backend/plugins/<name>/ follows this layout:

api/         # REST endpoints (connections, scopes, scope-configs)
impl/        # Plugin implementation (implements core interfaces)
models/      # Tool layer models + migrationscripts/
tasks/       # Collectors, Extractors, Converters
e2e/         # Integration tests with CSV fixtures

Required Interfaces

See backend/plugins/gitlab/impl/impl.go for reference:

  • PluginMeta: Name, Description, RootPkgPath
  • PluginTask: SubTaskMetas(), PrepareTaskData()
  • PluginModel: GetTablesInfo() - must list all models or CI fails
  • PluginMigration: MigrationScripts() for DB schema evolution
  • PluginSource: Connection(), Scope(), ScopeConfig()

Advanced Plugin Interfaces

  • PluginInit: Optional initialization hook with Init(basicRes) method for resource setup
  • PluginOpenApiSpec: Remote plugins can expose OpenAPI specs via OpenApiSpec() method
  • PluginMetric: For metrics plugins requiring RequiredDataEntities(), IsProjectMetric(), RunAfter(), Settings()
  • DataSourcePluginBlueprintV200: Project-aware pipeline generation with MakeDataSourcePipelinePlanV200() for cross-plugin scope mapping
  • MetricPluginBlueprintV200: Similar to DataSourcePluginBlueprintV200 for metric calculation plugins

Authentication Patterns

Plugins can support multiple authentication methods via these interfaces:

  • CacheableConnection: Extends ApiConnection with GetHash() for connection caching
  • MultiAuthenticator: Base interface with GetAuthMethod() returning BasicAuth, AccessToken, or AppKey
  • BasicAuthenticator: Implement GetBasicAuthenticator() for HTTP Basic auth
  • AccessTokenAuthenticator: Implement GetAccessTokenAuthenticator() for Bearer token auth
  • AppKeyAuthenticator: Implement GetAppKeyAuthenticator() for API key/secret pairs
  • PrepareApiClient: Hook in connection for initialization (e.g., token refresh) via PrepareApiClient(apiClient)

Dynamic Models

  • DynamicTabler interface: For runtime-generated models with methods Unwrap(), NewValue(), From(), To()

Subtask Pattern (Collector → Extractor → Converter)

// 1. Register subtask in tasks/register.go via init()
func init() {
    RegisterSubtaskMeta(&CollectIssuesMeta)
}

// 2. Define dependencies for execution order
var CollectIssuesMeta = plugin.SubTaskMeta{
    Name:         "Collect Issues",
    Dependencies: []*plugin.SubTaskMeta{}, // or reference other metas
}

API Collectors

Migration Scripts

  • Located in models/migrationscripts/
  • Register all scripts in register.go's All() function
  • Version format: YYYYMMDD_description.go

Build & Development Commands

# From repo root
make dep              # Install Go + Python dependencies
make build            # Build plugins + server
make dev              # Build + run server
make godev            # Go-only dev (no Python remote plugins)
make unit-test        # Run all unit tests
make e2e-test         # Run E2E tests

# From backend/
make swag             # Regenerate Swagger docs (required after API changes)
make lint             # Run golangci-lint
make mock             # Regenerate mocks from interfaces
make migration-script-lint  # Validate migration script format
make build-plugin-debug     # Build plugins with debug symbols (DEVLAKE_DEBUG=1)
make build-pydevlake        # Install/sync Python plugin framework dependencies
make e2e-test-go-plugins    # Run E2E tests for Go plugins only

Running Locally

docker-compose -f docker-compose-dev.yml up mysql grafana  # Start deps
make dev                                                     # Run server on :8080
cd config-ui && yarn && yarn start                          # UI on :4000

Testing

Unit Tests

Place *_test.go files alongside source. Use mocks from backend/mocks/. Mocks are auto-generated via make mock from all interfaces in core/ and helpers/.

E2E Tests for Plugins

Use CSV fixtures in e2e/ directory. See backend/test/helper/ for the Go test client that can spin up an in-memory DevLake instance.

Integration Testing

helper.ConnectLocalServer(t, &helper.LocalClientConfig{
    ServerPort:   8080,
    DbURL:        "mysql://merico:merico@127.0.0.1:3306/lake",
    CreateServer: true,
    Plugins:      []plugin.PluginMeta{gitlab.Gitlab{}},
})

Model Validation

Run make migration-script-lint from backend/ to validate all migration scripts follow the correct format (YYYYMMDD_description.go).

Python Plugins

Located in backend/python/plugins/. Use Poetry for dependencies. See backend/python/README.md.

Code Conventions

  • Tool model table names: _tool_<plugin>_<entity> (e.g., _tool_gitlab_issues)
  • Domain model IDs: Use didgen.NewDomainIdGenerator for consistent cross-plugin IDs
  • All plugins must be independent - no cross-plugin imports
  • Apache 2.0 license header required on all source files
  • Mocks are auto-generated: don't edit files in backend/mocks/, regenerate with make mock
  • Optional plugin interfaces (PluginInit, PluginOpenApiSpec, PluginMetric) should only be implemented if functionality is needed
  • Authentication: Use CacheableConnection interface if connection needs caching; implement appropriate *Authenticator for each auth method supported

Common Pitfalls

  • Forgetting to add models to GetTablesInfo() fails plugins/table_info_test.go
  • Migration scripts must be added to All() in register.go AND follow YYYYMMDD_description.go naming (validate with make migration-script-lint)
  • API changes require running make swag to update Swagger docs (this runs make mock first)
  • Python plugins require libgit2 for gitextractor functionality
  • New Plugin interfaces like PluginInit or PluginOpenApiSpec are optional - only implement if needed
  • Blueprint V2.0 implementation required for plugins that support project-aware scope mapping