Skip to content

linuxfoundation/lfx-v2-query-service

BranchesTags

Repository files navigation

lfx-v2-query-service

HTTP service for LFX API consumers to perform access-controlled queries for LFX resources, including typeahead and full-text search.

Architecture Overview

The implementation follows the clean architecture principles where:

  • Domain Layer: Contains business logic and interfaces
  • Service Layer: Orchestrates business operations
  • Infrastructure Layer: Implements external dependencies
  • Presentation Layer: Handles HTTP/API concerns (generated by Goa)

Directory Structure

├── .github/                        # Github files
│   └── workflows/                  # Github Action workflow files
├── charts/                         # Helm charts
├── design/                         # GOA design specification files
├── gen/                            # GOA generated code (HTTP server, client, OpenAPI)
├── cmd/                            # Services (main packages)
│   └── service/                    # Service implementation
├── internal/                       # Internal service packages
│   ├── domain/                     # Domain logic layer
│   │   ├── model/                  # Domain models and entities
│   │   └── port/                   # Domain interfaces/ports
│   ├── service/                    # Business logic/use cases layer
│   ├── infrastructure/             # Infrastructure layer
│   └── middleware/                 # HTTP middleware components
└── pkg/                            # Shared packages for internal and external services

Key Components

Domain Layer (internal/domain/)

  • ResourceService: Contains business logic and validation

Model (internal/domain/model/)

  • Domain Models: Core business entities and data structures
  • Value Objects: Immutable objects that represent domain concepts

Ports (internal/domain/port/)

  • ResourceSearcher Interface: Defines the contract for resource search operations
  • OrganizationSearcher Interface: Defines the contract for organization search operations
  • AccessControlChecker Interface: Defines the contract for access control operations
  • Authenticator Interface: Defines the contract for authentication operations

Service Layer (internal/service/)

  • Business Logic: Application-specific business rules and operations
  • Service Orchestration: Coordinates between domain models and infrastructure

Infrastructure Layer (internal/infrastructure/)

Authentication Implementation

The authentication system provides JWT-based authentication with support for Heimdall tokens:

  • JWT Authentication (internal/infrastructure/auth/jwt.go):

    • Validates JWT tokens using JWKS (JSON Web Key Set)
    • Extracts custom claims including principal and email
    • Supports PS256 signature algorithm (default for Heimdall)
    • Configurable JWKS URL and audience

  • Mock Authentication (internal/infrastructure/mock/auth.go):

    • Uses environment variable for mock principal
    • Bypasses JWT validation for local development

Authentication Configuration:

  • AUTH_SOURCE: Choose between "mock" or "jwt" (default: "jwt")
  • JWKS_URL: JSON Web Key Set endpoint URL
  • JWT_AUDIENCE: Intended audience for JWT tokens
  • JWT_AUTH_DISABLED_MOCK_LOCAL_PRINCIPAL: Mock principal for development

OpenSearch Implementation

The OpenSearch implementation includes query templates, a searcher, and a client for interacting with the OpenSearch cluster.

NATS Implementation

The NATS implementation consists of a client, access control logic, and request/response models for messaging and access control.

Clearbit Implementation

The Clearbit implementation provides organization search capabilities using the Clearbit Company API. It includes a client for API communication, searcher for organization queries, and configuration management for API credentials and settings.

Dependency Injection

Dependency injection is performed in cmd/main.go, where the concrete implementations for resource search and access control are selected based on configuration and then injected into the service constructor.

Benefits of This Architecture

  1. Testability: Easy to swap implementations for testing
  2. Flexibility: Can easily switch between different search backends and access control systems
  3. Maintainability: Clear separation of concerns
  4. Scalability: Easy to add new search and access control implementations
  5. Independence: Layers don't depend on external frameworks

Docker

Building the Docker Image

Example: Build the Docker image with Make

make docker-build

Running with Docker

Basic Docker Run

make docker-run

Usage

Running Locally

With Mock Implementation (Default for Development)

# Using mock implementations
SEARCH_SOURCE=mock ACCESS_CONTROL_SOURCE=mock go run cmd/main.go

# With custom port
SEARCH_SOURCE=mock ACCESS_CONTROL_SOURCE=mock go run cmd/main.go -p 3000

With Production Services

# production-like setup
SEARCH_SOURCE=opensearch \
ORG_SEARCH_SOURCE=clearbit \
ACCESS_CONTROL_SOURCE=nats \
OPENSEARCH_URL={{placeholder}} \
OPENSEARCH_INDEX=resources \
NATS_URL{{placeholder}} \
CLEARBIT_CREDENTIAL=your_clearbit_api_key \
CLEARBIT_BASE_URL=https://company.clearbit.com \
CLEARBIT_TIMEOUT=30s \
CLEARBIT_MAX_RETRIES=5 \
CLEARBIT_RETRY_DELAY=2s \
go run cmd/main.go

Available Environment Variables

Search Implementation:

  • SEARCH_SOURCE: Choose between "mock" or "opensearch" (default: "opensearch")

Organization Search Implementation:

  • ORG_SEARCH_SOURCE: Choose between "mock" or "clearbit" (default: "clearbit")

OpenSearch Configuration:

  • OPENSEARCH_URL: OpenSearch URL (default: http://localhost:9200)
  • OPENSEARCH_INDEX: OpenSearch index name (default: "resources")

Access Control Implementation:

  • ACCESS_CONTROL_SOURCE: Choose between "mock" or "nats" (default: "nats")

NATS Configuration:

  • NATS_URL: NATS server URL (default: nats://localhost:4222)
  • NATS_TIMEOUT: Request timeout duration (default: "10s")
  • NATS_MAX_RECONNECT: Maximum reconnection attempts (default: "3")
  • NATS_RECONNECT_WAIT: Time between reconnection attempts (default: "2s")

Clearbit Configuration:

  • CLEARBIT_CREDENTIAL: Clearbit API key (required for organization search)
  • CLEARBIT_BASE_URL: Clearbit API base URL (default: https://company.clearbit.com)
  • CLEARBIT_TIMEOUT: HTTP client timeout for API requests (default: "10s")
  • CLEARBIT_MAX_RETRIES: Maximum number of retry attempts for failed requests (default: "3")
  • CLEARBIT_RETRY_DELAY: Delay between retry attempts (default: "1s")

Authentication Configuration:

  • AUTH_SOURCE: Choose between "mock" or "jwt"
  • JWKS_URL: JSON Web Key Set endpoint URL
  • JWT_AUDIENCE: Intended audience for JWT tokens
  • JWT_AUTH_DISABLED_MOCK_LOCAL_PRINCIPAL: Mock principal for development (required when AUTH_SOURCE=mock)

Server Configuration:

  • -p: HTTP port (default: "8080")
  • -bind: Interface to bind on (default: "*")
  • -d: Enable debug logging

API Usage

The service exposes a RESTful API through the Goa framework with JWT authentication:

Resource Search API

GET /query/resources?name=committee&type=committee&v=1
Authorization: Bearer <jwt_token>

Parameters:

  • name: Resource name or alias (supports typeahead search)
  • type: Resource type to filter by
  • parent: Parent resource for hierarchical queries <<<<<<< HEAD
  • tags: Array of tags to filter by (OR logic - matches resources with any of these tags)
  • tags_all: Array of tags to filter by (AND logic - matches resources that have all of these tags)
  • date_field: Date field to filter on (within data object) - used with date_from and/or date_to
  • date_from: Start date (inclusive). Format: ISO 8601 datetime or date-only (YYYY-MM-DD). Date-only uses start of day UTC
  • date_to: End date (inclusive). Format: ISO 8601 datetime or date-only (YYYY-MM-DD). Date-only uses end of day UTC =======
  • tags: Array of tags to filter by (OR logic)
  • tags_all: Array of tags where all must match (AND logic)
  • cel_filter: CEL expression for advanced post-query filtering (see CEL Filter section)

3e45fc4d33aba656a5abe1c3df0d3f2bd0fd6be7

  • sort: Sort order (name_asc, name_desc, updated_asc, updated_desc)
  • page_token: Pagination token
  • v: API version (required)

Response:

{
  "resources": [
    {
      "type": "committee",
      "id": "123",
      "data": {
        "name": "Technical Advisory Committee",
        "description": "Main technical governance body",
        "status": "active"
      }
    }
  ],
  "page_token": "offset_50",
  "cache_control": "public, max-age=300"
}

<<<<<<< HEAD Date Range Filtering Examples:

Filter resources updated between two dates (date-only format):

GET /query/resources?v=1&date_field=updated_at&date_from=2025-01-10&date_to=2025-01-28
Authorization: Bearer <jwt_token>

Filter resources with precise datetime filtering (ISO 8601 format):

GET /query/resources?v=1&date_field=created_at&date_from=2025-01-10T15:30:00Z&date_to=2025-01-28T18:45:00Z
Authorization: Bearer <jwt_token>

Filter resources created after a specific date (open-ended range):

GET /query/resources?v=1&date_field=created_at&date_from=2025-01-01
Authorization: Bearer <jwt_token>

Combine date filtering with other parameters:

GET /query/resources?v=1&type=project&tags=active&date_field=updated_at&date_from=2025-01-01&date_to=2025-03-31
Authorization: Bearer <jwt_token>

Date Format Notes:

  • ISO 8601 datetime format: 2025-01-10T15:30:00Z (time is used as provided)
  • Date-only format: 2025-01-10 (automatically converted to start/end of day UTC)
    • For date_from: Converts to 2025-01-10T00:00:00Z (start of day)
    • For date_to: Converts to 2025-01-10T23:59:59Z (end of day)
  • All dates are inclusive (uses gte and lte operators)
  • The date_field parameter is automatically prefixed with "data." to scope to the resource's data object =======

CEL Filter

The cel_filter query parameter enables advanced filtering of search results using Common Expression Language (CEL). CEL is a non-Turing complete expression language designed for safe, fast evaluation of expressions in performance-critical applications.

Why CEL Filter?

CEL filtering was added to provide flexible, dynamic filtering capabilities on arbitrary resource data fields without modifying the OpenSearch query structure. This allows API consumers to:

  • Filter on any field within the resource data
  • Combine multiple conditions with boolean logic
  • Perform complex comparisons beyond simple equality checks
  • Apply filters without requiring backend code changes

What is CEL?

CEL (Common Expression Language) is an open-source expression language developed by Google. It provides:

  • Safety: Non-Turing complete, no side effects, no infinite loops
  • Performance: Linear time evaluation with compilation and caching
  • Portability: Language-agnostic with implementations in multiple languages
  • Security: Execution timeouts and resource constraints

Learn more: CEL Specification | CEL-Go Documentation

How It Works

CEL filters are applied after the OpenSearch query executes but before access control checks. This means:

  1. OpenSearch returns initial results based on primary search criteria (type, name, parent, tags)
  2. CEL filter evaluates each resource and removes non-matching items
  3. Access control checks are performed only on filtered results (improved performance)
  4. Final results are returned to the client

Available Variables

CEL expressions have access to the following variables for each resource:

  • data (map): The resource's data object containing all custom fields
  • resource_type (string): The type of the resource (e.g., "project", "committee")
  • id (string): The unique identifier of the resource

Security Constraints

  • Maximum expression length: 1000 characters
  • Evaluation timeout: 100ms per resource
  • Expression caching: Compiled programs cached with LRU and 5-minute TTL
  • No external access: Cannot make network calls or access filesystem

Usage Examples

Filter projects by slug:

GET /query/resources?type=project&cel_filter=data.slug == "tlf"&v=1

Filter by status and priority:

GET /query/resources?type=project&cel_filter=data.status == "active" && data.priority > 5&v=1

Filter by resource type:

GET /query/resources?parent=org:123&cel_filter=resource_type == "committee"&v=1

Complex boolean logic:

GET /query/resources?type=project&cel_filter=data.status == "active" || (data.priority > 8 && data.category == "security")&v=1

String operations:

GET /query/resources?type=project&cel_filter=data.name.contains("LF") && data.description.startsWith("Open")&v=1

Check field existence:

GET /query/resources?type=project&cel_filter=has(data.archived) && data.archived == false&v=1

List membership:

GET /query/resources?type=project&cel_filter=data.category in ["security", "networking", "storage"]&v=1

Nested field access:

GET /query/resources?type=project&cel_filter=data.metadata.owner == "admin" && data.metadata.region == "us-west"&v=1

Supported Operators

  • Comparison: ==, !=, <, <=, >, >=
  • Logical: && (AND), || (OR), ! (NOT)
  • Arithmetic: +, -, *, /, %
  • String: contains(), startsWith(), endsWith(), matches() (regex)
  • Membership: in
  • Field check: has()

Important Limitations

⚠️ Pagination Consideration: CEL filters are applied to the results from each OpenSearch page. If you're looking for a specific resource that matches your CEL filter but it's not in the first page of OpenSearch results, it may not be found. For best results when using CEL filters, use more specific primary search parameters (type, name, parent, tags) to narrow down the OpenSearch results first.

Error Handling

Invalid CEL expressions return a 400 Bad Request with details:

{
  "error": "filter expression failed: ERROR: <input>:1:6: Syntax error: mismatched input 'invalid' expecting {'[', '{', '(', '.', '-', '!', 'true', 'false', 'null', NUM_FLOAT, NUM_INT, NUM_UINT, STRING, BYTES, IDENTIFIER}"
}

3e45fc4d33aba656a5abe1c3df0d3f2bd0fd6be7

Organization Search API

Query Organizations:

GET /query/orgs?name=Linux Foundation&domain=linuxfoundation.org&v=1
Authorization: Bearer <jwt_token>

Parameters:

  • name: Organization name (optional)
  • domain: Organization domain or website URL (optional)
  • v: API version (required)

Response:

{
  "name": "Linux Foundation",
  "domain": "linuxfoundation.org",
  "industry": "Non-Profit",
  "sector": "Technology",
  "employees": "100-499"
}

Organization Suggestions API:

GET /query/orgs/suggest?query=linux&v=1
Authorization: Bearer <jwt_token>

Parameters:

  • query: Search query for organization suggestions (required, minimum 1 character)
  • v: API version (required)

Response:

{
  "suggestions": [
    {
      "name": "Linux Foundation",
      "domain": "linuxfoundation.org",
      "logo": "https://example.com/logo.png"
    }
  ]
}

Clearbit API Integration

The service integrates with Clearbit's Company API to provide enriched organization data for search operations. This integration allows the service to fetch detailed company information including industry classification, employee count, and domain information.

Clearbit API Setup

1. Obtain API Credentials

To use Clearbit integration, you need to obtain an API key from Clearbit:

  1. Sign up for a Clearbit account at https://clearbit.com
  2. Navigate to your API settings to generate an API key
  3. Copy the API key for use in your environment configuration

2. Configure Environment Variables

Set the required environment variables for Clearbit integration:

# Required: Clearbit API key
export CLEARBIT_CREDENTIAL=your_clearbit_api_key_here

# Optional: Custom configuration (defaults shown)
export CLEARBIT_BASE_URL=https://company.clearbit.com
export CLEARBIT_TIMEOUT=30s
export CLEARBIT_MAX_RETRIES=3
export CLEARBIT_RETRY_DELAY=1s

# Set organization search source to use Clearbit
export ORG_SEARCH_SOURCE=clearbit

3. API Usage and Features

The Clearbit integration supports the following search operations:

Search by Company Name:

  • Searches for companies using their registered business name
  • Falls back to domain-based search for additional data enrichment

Search by Domain:

  • More accurate search method using company domain names
  • Provides comprehensive company information

4. Error Handling

The Clearbit integration includes robust error handling:

  • 404 Not Found: Returns appropriate "organization not found" errors
  • Rate Limiting: Automatic retry with exponential backoff
  • Network Issues: Configurable retry attempts with delays
  • API Errors: Proper error propagation with context

Testing

The clean architecture makes testing straightforward:

// Use mock implementations for unit tests
searcher := mock.NewMockResourceSearcher()
searcher.AddResource(testResource)

accessChecker := mock.NewMockAccessControlChecker()
accessChecker.SetAccessResult(testResult)

service := service.NewResourceService(searcher, accessChecker)
result, err := service.QueryResources(ctx, criteria)

Extending the Architecture

To add a new search implementation:

  1. Create a new package in internal/infrastructure/
  2. Implement the domain.ResourceSearcher interface
  3. Add configuration options to main.go
  4. Update the dependency injection switch statement

To add a new access control implementation:

  1. Create a new package in internal/infrastructure/
  2. Implement the domain.AccessControlChecker interface
  3. Add configuration options to main.go
  4. Update the dependency injection switch statement

Development

Prerequisites

This project uses the GOA Framework for API generation. You'll need to install GOA before building the project.

Installing GOA Framework

Follow the GOA installation guide to install GOA:

go install goa.design/goa/v3/cmd/goa@latest

Verify the installation:

goa version

Building and Development

1. Generate Code

The project uses GOA to generate API code from the design specification. Run the following command to generate all necessary code:

goa gen github.com/linuxfoundation/lfx-v2-query-service/design

This command generates:

  • HTTP server and client code
  • OpenAPI specification
  • Service interfaces and types
  • Transport layer implementations

2. Initial Project Structure

Note: The initial cmd structure was generated using GOA's example generator:

goa example github.com/linuxfoundation/lfx-v2-query-service/design

This command generated the basic server structure, which was then customized and adjusted to fit our project's clean architecture principles.

3. Development Workflow

  1. Make design changes: Edit files in the design/ directory

  2. Regenerate code: Run goa gen github.com/linuxfoundation/lfx-v2-query-service/design after design changes

  3. Build the project:

    go build cmd

  4. Run with mock data (for development):

    SEARCH_SOURCE=mock ACCESS_CONTROL_SOURCE=mock go run ./cmd

  5. Run tests:

    make test

# or run go test to set custom flags
go test ./... -v

  6. Lint the code

    # From the root of the directory, run megalinter (https://megalinter.io/latest/mega-linter-runner/) to ensure the code passes the linter checks. The CI/CD has a check that uses megalinter.
npx mega-linter-runner .

  7. Docker build + K8

    # Build the dockerfile (from the root of the repo)
docker build -t lfx-v2-query-service:<release_number> .

# Install the helm chart for the service into the lfx namespace (from the root of the repo)
helm install lfx-v2-query-service ./charts/lfx-v2-query-service/ -n lfx

# Once you have already installed the helm chart and need to just update it, use the following command (from the root of the repo):
helm upgrade lfx-v2-query-service ./charts/lfx-v2-query-service/ -n lfx

Contributing

To contribute to this repository:

  1. Fork the repository
  2. Make your changes
  3. Submit a pull request

License

Copyright The Linux Foundation and each contributor to LFX.

This project’s source code is licensed under the MIT License. A copy of the license is available in LICENSE.

This project’s documentation is licensed under the Creative Commons Attribution 4.0 International License (CC-BY-4.0). A copy of the license is available in LICENSE-docs.

About

LFX v2 Platform Query Service

Resources

Readme

License

MIT, CC-BY-4.0 licenses found

Licenses found

MIT
LICENSE
CC-BY-4.0
LICENSE-docs

Security policy

Security policy
Activity
Custom properties

Stars

0 stars

Watchers

0 watching

Forks

2 forks
Report repository

Releases 18

v0.4.12 Latest
Feb 22, 2026
+ 17 releases

Packages

 
 
 

Contributors

Languages