IceGate

icegate.tech · Documentation · Slack

An Observability Data Lake engine designed to be fast, easy-to-use, cost-effective, scalable, and fault-tolerant.

IceGate supports open protocols and APIs compatible with standard ingesting and querying tools. All data is persisted in Object Storage, including WAL for ingested data, catalog metadata, and the data layer.

Features

• Highly Scalable: Scale compute resources independently based on workload demands of specific components
• ACID Transactions: Full transaction support without requiring a dedicated OLTP database
• Exactly-Once Delivery: Reliable data ingestion with no data loss or duplication
• Real-Time Queries: Access live data through WAL while maintaining historical query capabilities
• Open Standards: Built on Apache Iceberg, Arrow, and Parquet with OpenTelemetry protocol support
• Cost-Effective: Object storage-based architecture minimizes infrastructure costs
• Fault-Tolerant: Designed for resilience and high availability

Architecture

IceGate employs a compute-storage separation architecture, allowing independent scaling of processing and storage resources. This design enables cost-effective scaling where compute resources (Ingest, Query, Maintain, Alert) can be scaled independently based on workload demands, while all data resides in object storage.

The system consists of five core components for handling observability data (metrics, traces, logs, and events):

Catalog

• Technology: Apache Iceberg
• Purpose: Organizes the data lake with ACID transaction support
• Key Feature: Custom S3-backed catalog (the default) that doesn't require a dedicated OLTP database while still supporting transactions

Ingest

• Protocol: OpenTelemetry
• Purpose: Accept observability data and persist it in Object Storage
• Implementation: WAL using Parquet files organized in a special way to be compatible with the Storage data layer
• Delivery Guarantee: Exactly-once delivery
• Note: WAL files can be used by the Query layer to provide real-time data access

Query

• Technology: Apache DataFusion and Apache Arrow
• Purpose: Query engine for processing logs, metrics, traces, and events
• Implementation: Rust-native query engine built on Apache Arrow, providing a foundation to build query engines using various protocols

Maintain

• Data Format: Apache Parquet
• Features:
  • Statistics and bloom filters for efficient querying
  • Additional custom statistics and filters for optimization
  • Data optimization operations: merge, TTL support, manifest optimization, and orphan resource cleanup
• Purpose: Maximize query efficiency and maintain data lake health

Alert

• Purpose: Provides management of alerting rules, analyzing observability data, and generating alert events
• Features:
  • Rule management for defining alert conditions
  • Real-time analysis of observability data (logs, metrics, traces)
  • Event generation and delivery based on rule evaluation
• Data Type: Events are treated as a dedicated data type alongside logs, metrics, and traces
• Convention: Follows OpenTelemetry Events Semantic Conventions
• Implementation: Leverages the Query layer for querying observability data to evaluate alert rules

Getting Started

Docker compose

Prerequisites

• Docker and Docker Compose - for running the application
• Git - for cloning the repository

Quick Start

1. Clone the repository:

   git clone https://github.com/icegatetech/icegate.git
   cd icegate

2. Start the core services in docker compose:

   make run-docker-core-release

   This command builds and runs all core services in Docker containers. The first run will take a few minutes to build the images and start all services.

Kubernetes

Prerequisites

• Orbstack, kubectl and Kustomize - for running the application
• Git - for cloning the repository

Quick Start

1. Clone the repository:

   git clone https://github.com/icegatetech/icegate.git
   cd icegate

2. Start the core services in Kubernetes on Orbstack (Mac):

   make run-kubernetes-core-release

This command builds and runs all core services in Kubernetes. The first run will take a few minutes to build the images and start all services.

Available Services

Once running, the following services are available:

Data Ingestion

• OTLP gRPC: http://localhost:4317 - Ingest observability data using OpenTelemetry gRPC protocol
• OTLP HTTP: http://localhost:4318 - Ingest observability data using OpenTelemetry HTTP protocol
• Ingest Metrics: http://localhost:9091/metrics - Prometheus metrics for ingest service

Query APIs

• Loki API: http://localhost:3100 - Query logs using Loki-compatible API ✅ Currently Supported (query, query_range, labels, label/{name}/values, series)
• Tempo API: http://localhost:3200 - Query traces using Tempo-compatible API ✅ Currently Supported (traces/{id}, search with TraceQL, search/tags, search/tag/{name}/values, plus the v2 variants). Unsupported TraceQL features return 501
• Prometheus API: http://localhost:9090 - Query metrics ⚠️ Planned — the routes are mounted but every handler returns 501 Not Implemented; only /-/ready responds
• Arrow Flight SQL: grpc://localhost:8815 - General-purpose SQL over the merged WAL + Iceberg view; read-only, tenant identified by the x-scope-orgid gRPC metadata header (defaults to default). See Querying via Flight SQL below for supported clients.

Visualization

• Grafana: http://localhost:3000 - Dashboard and visualization (no login required)

Infrastructure

• RustFS Console: http://localhost:9001 - Object storage web interface (login: rustfsadmin/rustfsadmin)
• RustFS API: http://localhost:9000 - S3-compatible object storage API

The default stack uses IceGate's own S3-backed catalog (see Catalog); no external catalog service is required. The optional analytics profile additionally exposes that same catalog over a read-only Iceberg REST endpoint (see Additional Profiles).

Additional Profiles

Run with additional services for testing and monitoring:

# Include load generator for testing
make run-docker-load-release

# Pure runtime services
make run-docker-core-release

# Include analytics tools (Trino SQL query engine at http://localhost:8082)
make run-docker-analytics-release

The analytics profile starts Trino together with an Iceberg REST endpoint (http://localhost:8181) over the default stack's own catalog, so no separate catalog service and no separate copy of the data is involved. The endpoint serves reads only — writes go through ingest. For SQL without a separate engine, use Arrow Flight SQL at grpc://localhost:8815.

Next Steps

• Send telemetry data to localhost:4317 (gRPC) or localhost:4318 (HTTP) using any OpenTelemetry SDK
• View data in Grafana at http://localhost:3000
• Query data using the Query APIs above — note the current status marked against each

Querying via Flight SQL

IceGate exposes an Apache Arrow Flight SQL gRPC endpoint on :8815. Queries run server-side through Apache DataFusion against the merged WAL + Iceberg view, so only result sets cross the wire — aggregations and joins do not pull raw rows to the client. The endpoint is strictly read-only (DDL and DML are rejected) and identifies tenants from the x-scope-orgid gRPC metadata header.

Supported clients

Client	How
Python	pip install adbc-driver-flightsql adbc-driver-manager
JDBC (dbt, DataGrip, DBeaver, Apache Superset, Metabase)	org.apache.arrow:flight-sql-jdbc-driver
ODBC (Tableau, Power BI, Excel)	Apache Arrow Flight SQL ODBC Driver
Native	arrow-flight C++ / Rust / Go libraries
Loki / Tempo HTTP	Endpoints on :3100 / :3200 — preferred for those workloads. Prometheus (:9090) is not implemented yet; see Query APIs

Python example

import adbc_driver_flightsql.dbapi

conn = adbc_driver_flightsql.dbapi.connect(
    "grpc://localhost:8815",
    db_kwargs={"adbc.flight.sql.rpc.call_header.x-scope-orgid": "tenant-alpha"},
)
with conn.cursor() as cur:
    cur.execute("SELECT count(*) FROM iceberg.icegate.logs")
    print(cur.fetchall())

Warning

DuckDB users: there is no first-party DuckDB Flight SQL extension. Connect either through the Apache Arrow Flight SQL ODBC driver via DuckDB's odbc core extension, or through Python with adbc_driver_flightsql and import results into DuckDB as Arrow.

Do not use the airport community extension or duckhog against IceGate.

• airport looks like a Flight SQL client but speaks a different protocol over the same Arrow Flight transport. DuckDB owns the planner and pulls raw rows over the wire, then aggregates locally. On a billion-row observability table a single count(*) GROUP BY hour can ship gigabytes to the client. With Flight SQL via ODBC or ADBC, DataFusion aggregates server-side and only the result set crosses the wire.
• duckhog is an unmaintained third-party Flight SQL client whose quirks (mandatory BeginTransaction handshakes, user=<tenant> connection-string parameter as identity) IceGate explicitly does not accommodate.

Development

For developers who want to build from source and contribute:

cargo build            # Build the project
cargo test             # Run tests
make dev               # Start development stack with hot-reload

For detailed development setup, build commands, and code quality guidelines, see CONTRIBUTING.md.

Contributing

Contributions are welcome! See CONTRIBUTING.md for guidelines.

License

This project is licensed under the Apache License 2.0 - see the LICENSE file for details.

Who builds IceGate

IceGate is built by TripleCloud, and is open source under Apache 2.0 — self-host it, fork it, and run it against object storage you own, with no TripleCloud account and no hosted component in the path.

TripleCloud is the LLM observability platform built on this engine. IceGate is not a trial edition of it and is not feature-gated against it; the relationship runs the other way, and the engineering write-ups below describe the engine itself rather than the platform.

• Observability ingestion on S3 alone — how ingestion runs with no Kafka, no local disks and no coordination service
• Instrumenting an LLM agent with OpenTelemetry — end-to-end OTLP traces landing in IceGate, with runnable recipes in icegatetech/integrations

Acknowledgments

Built with:

• Apache Iceberg - Table format for data lakes
• Apache Arrow - Columnar memory format
• Apache Parquet - Columnar storage format
• OpenTelemetry - Observability framework

Trademarks

IceGate names third-party projects to describe, factually, which formats it writes and which wire protocols its APIs implement. Such nominative use implies no affiliation with, endorsement by, or sponsorship from the owners of those marks.

Apache®, Apache Iceberg, Apache Arrow, Apache Parquet, Apache DataFusion, Apache Arrow Flight SQL and associated project logos are either registered trademarks or trademarks of The Apache Software Foundation in the United States and/or other countries. OpenTelemetry® and Prometheus® are registered trademarks of The Linux Foundation. Grafana®, Loki® and Tempo® are registered trademarks of Raintank, Inc. dba Grafana Labs. IceGate is not affiliated with, endorsed by, or sponsored by any of these organizations. All other trademarks are the property of their respective owners.

Where an API is described as Loki- or Tempo-compatible, IceGate implements a subset of that project's HTTP read API — not a complete reimplementation. See Query APIs for the endpoints served today and the current status of each.

Status

IceGate is currently in prototype development. APIs and features are subject to change.