Skip to content

README.md

Latest commit

 

History

History
193 lines (149 loc) · 10.3 KB

README.md

File metadata and controls

193 lines (149 loc) · 10.3 KB
Raw

Cloud Query

Cloud Query is a service part of the Plural Console ecosystem that provides cloud resource querying capabilities. It uses an embedded PostgreSQL database to cache and process data about cloud resources.

Features

  • Query cloud resources across multiple providers
  • Embedded PostgreSQL database for data storage and retrieval through PostgreSQL FDW steampipe extension
  • gRPC API for integration with other services
  • Sandboxed execution for Lua and Monty's limited Python subset
  • Containerized deployment for easy scaling

CloudQuery provider support includes AWS, Azure, GCP, and VMware vSphere.

Prerequisites

  • Go 1.26.6 or higher
  • Docker (for containerized deployment)
  • Make

Running the complete service locally requires a writable temporary directory. The Python worker extracts gomonty's embedded native library there on first use.

Getting Started

Local Development

  1. Clone the repository:
git clone https://github.com/pluralsh/console.git
cd console/go/cloud-query
  1. Build the binary:
make build

This will create the binary in the bin directory.

  1. Run the server locally:
./bin/cloud-query

Docker Deployment

  1. Build the Docker image:
make image
  1. Run the container:
make run

Or manually:

docker run --rm -p 9192:9192 cloud-query:latest

Configuration

The service can be configured using command-line flags or environment variables.

For a comprehensive list of all available command-line arguments, their default values, and usage examples, see the Command-Line Arguments Reference.

API Reference

Cloud-Query exposes a gRPC API for querying cloud resources. Client implementations can use this API to retrieve information about cloud resources across different providers.

For detailed information about the API endpoints, request/response schemas, and usage examples, see the API Reference Documentation.

CloudQuery vSphere connections require server, user, and password; allow_unverified_ssl can be set to "true" when connecting to vCenter endpoints with untrusted certificates.

Tool Integrations (ToolQuery)

Cloud-Query also exposes ToolQuery gRPC endpoints for observability tools (metrics, logs, traces). Compatibility is per operation:

Tool Metrics Metric Label Search Logs Traces Notes
Prometheus Yes Yes No No Prometheus HTTP API via prometheus/client_golang with optional bearer token or basic auth
Datadog Yes Yes Yes Yes Datadog API v1/v2 via datadog-api-client-go (requires API key + app key; site optional)
Elasticsearch No No Yes No Elasticsearch typed client v9 Search API (API key required)
Loki No No Yes No REST client to /loki/api/v1/query_range (bearer token; optional X-Scope-OrgID)
Splunk No No Yes No Splunk export search API (token or basic auth)
Tempo No No No Yes REST client to /api/search and /api/traces/{traceID} (bearer token; optional X-Scope-OrgID)
Jaeger No No No Yes Jaeger Query v3 REST API (GET /api/v3/traces) with structured trace filters
Dynatrace Yes No Yes Yes Dynatrace Grail Query API (DQL via /platform/storage/query/v1/query:*, bearer token required)
CloudWatch Yes Yes Yes No AWS SDK v2 (GetMetricData, ListMetrics, Logs Insights) with optional assume-role
Azure Yes Yes Yes No Azure Monitor Go SDK with Azure AD client credentials

ToolQuery also supports cloud function invocation via InvokeLambda for AWS Lambda, GCP Cloud Run services (Gen2), and Azure Functions using canonical identifiers and cloud connection credentials.

Sandboxed Python execution

RunPython executes Monty's limited Python subset in a crash-isolated cloud-query python-worker subprocess. The request's optional JSON object is available as input; output starts as an empty dictionary and must remain a JSON-serializable dictionary. The response returns that dictionary as result_json and standard-output text from print() separately as stdout.

This is not CPython. The sandbox has no host filesystem, environment, network, subprocess, shell, pip, third-party packages, or callback access. Two workers start with only TMPDIR=/tmp; each request gets a fresh gomonty REPL. Failed or canceled workers are killed and replaced, and healthy workers are periodically recycled. Source is limited to 64 KiB, input and result JSON to 1 MiB, stdout to 64 KiB, interpreter execution to 10 seconds, interpreter-managed memory to 64 MiB, and recursion to 200 frames. Two runs execute concurrently and up to 16 more wait in a FIFO queue. A full queue is rejected. A parent watchdog ends a run after 15 seconds or the caller's earlier deadline.

The image embeds gomonty v0.0.14 and its platform-specific glibc library, built against official Monty commit c9802b5f30d11fecf9f153feb1dfdab3abda070e. It contains no separate monty executable. Both pins are recorded in OCI labels. Monty's memory limit covers interpreter-managed allocations rather than total pod RSS; operators should measure the workload before reducing the default cloud-query memory allocation.

Tool Provider Credentials and Permissions

  • Dynatrace:
    • Use a Dynatrace Platform token (platformToken).
    • Required scopes:
      • storage:logs:read
      • storage:metrics:read
      • storage:spans:read
      • storage:entities:read
      • storage:buckets:read
  • Datadog:
    • Requires apiKey and appKey for ToolQuery operations.
    • Metric label search returns indexed metric tags from the previous hour.
  • Elasticsearch:
    • Requires URL + username/password + index.
  • Prometheus / Loki / Tempo:
    • Use bearer token and/or basic auth credentials when required by your backend.
    • If multi-tenant, also configure tenant_id (X-Scope-OrgID).
  • Jaeger:
    • Uses Jaeger stable v3 Query API (GET /api/v3/traces).
    • Traces.query is interpreted as Jaeger service_name.
    • Additional structured filters are provided via Traces.options.jaeger (operation_name, duration_min, duration_max, attributes).
    • Handles both application/json and text/plain Content-Type responses with automatic fallback parsing.
    • Supports bearer token and basic authentication.
  • Splunk:
    • Use bearer token or username/password.
  • CloudWatch:
    • region is required.
    • Supports optional static AWS credentials, optional assume role (role_arn), and default AWS credential chain fallback (including pod identity).
    • For logs queries, configure log_group_names or use SOURCE in the query string.
    • log_group_names applies to Logs only (not Metrics / MetricsSearch).
    • Examples:
      • Logs with provider log groups: set "log_group_names": ["/aws/eks/prod/app"] and use query like "fields @timestamp, @message | sort @timestamp desc".
      • Logs without provider log groups: omit log_group_names and use query like "SOURCE logGroups(namePrefix: [\"/aws/eks/prod/app\"]) | fields @timestamp, @message | sort @timestamp desc".
      • Metrics: use CloudWatch metric math expression, for example SEARCH("{AWS/EC2,InstanceId} MetricName=\"CPUUtilization\"", "Average", 300).
      • Metric label search: use the MetricsSearch result format <namespace>/<metric>, for example "AWS/EC2/CPUUtilization"; labels are CloudWatch dimensions.
  • Azure:
    • Connection requires subscription_id, tenant_id, client_id, and client_secret.
    • Metrics use azmetrics.QueryResources:
      • query is a comma-separated metric name list (for example "Percentage CPU,Network In Total").
      • options.azure.resource_id and options.azure.metrics_namespace are required.
      • Optional Azure metrics options: aggregation, filter, order_by, roll_up_by, metrics_endpoint.
      • options.azure.metrics_endpoint overrides the metrics endpoint per request. If omitted, Cloud Query falls back to https://global.metrics.monitor.azure.com.
    • Metric label search:
      • Azure Managed Prometheus (options.azure.prometheus_url) delegates to Prometheus label APIs.
      • Native Azure Monitor label names come from metric-definition dimensions.
      • Native Azure Monitor label values require options.azure.metrics_namespace and are inferred from recent metric time-series metadata.
    • Logs use azlogs.QueryResource:
      • query is Azure Log Analytics query syntax (KQL).

InvokeLambda Provider Permissions

  • AWS:
    • Caller needs permission to invoke the target Lambda function (lambda:InvokeFunction).
  • GCP:
    • Caller needs:
      • run.routes.invoke
      • run.services.get
    • These map to Cloud Run IAM roles such as roles/run.invoker (invoke) and roles/run.viewer (service read).
  • Azure:
    • Caller needs permissions to read function metadata/secrets and invoke the function endpoint.

For provider-specific request payloads, query formats, and examples, see the API Reference Documentation.

Documentation

The project includes detailed documentation:

Project Structure

  • /api - API proto definitions for gRPC services
  • /cmd - Command-line entry points
  • /docs - Project documentation
  • /internal - Internal packages
    • /connection - Database connection management
    • /extension - Database extensions management
    • /pool - Connection pooling
    • /server - gRPC server implementation
  • /hack - Scripts and tools for development

Contributing

  1. Fork the project
  2. Create your feature branch (git checkout -b feature/amazing-feature)
  3. Commit your changes (git commit -m 'Add some amazing feature')
  4. Push to the branch (git push origin feature/amazing-feature)
  5. Open a Pull Request