Discussion: pluggable Connection design for chDB integration in @clickhouse/client

[ShawnChen-Sirius]

TL;DR

To let @clickhouse/client users target either a remote ClickHouse server or in-process chDB with a one-line change — without forking the client, without losing chDB's zero-copy advantage, and without making @clickhouse/client carry chDB-specific code — this proposal introduces a pluggable Connection behind a single public client API:

• @clickhouse/client exposes a public Connection interface (it already has private HttpConnection/HttpsConnection — promote them to a public contract).
• createClient() accepts an injected connection option.
• chdb-node ships a ChdbConnection implementation that consumes its existing Layer 1 native session.
• @clickhouse/client ships zero chDB code, zero chDB tests, zero chDB CI dependency.
• A supportsZeroCopyStreaming capability flag lets the client route streaming through chDB's in-process record-batch path when available, preserving end-to-end zero-copy.
• chDB-unique surface (Python() table function, UDF registration, in-memory vs persistent path management) lives in a typed extension namespace (client.chdb.*), available only when the chdb connection is in use.
• Tests run on every installed connection via a parameterized fixture; chDB owns the skip manifest in its own repo via a published JSON profile.
• CI is split: @clickhouse/client CI never touches chDB; chdb-node CI runs @clickhouse/client's test suite against ChdbConnection.

Core principle (one sentence)

@clickhouse/client provides "one ClickHouseClient + a public Connection interface + a connection injection point on createClient()"; chdb-node provides a ChdbConnection implementation consumed by user code via that injection point. The two packages are coupled only through the Connection interface — @clickhouse/client ships zero chDB code, zero chDB tests, and zero chDB CI dependency.

---

The detailed design is folded below — expand the sections you want to dig into.

📖 1. Background — what users want, and two tempting approaches that this proposal rejects

1.1 What users want

Two recurring asks from the chDB and @clickhouse/client user communities:

1. One-line switch between remote ClickHouse and in-process chDB — a developer writing code against @clickhouse/client wants to point that same code at chDB for local development, CI tests, ephemeral environments, or embedded analytics, without rewriting any business logic.
2. No compromise on the embedded engine's advantage — chDB's value over a remote server is in-process zero-copy data exchange. If the integration loses that, users have no reason to choose chDB over a local ClickHouse container.

A third constraint comes from the @clickhouse/client maintainers: the public client surface is intentionally slim, and bolting a second client family onto it (one full surface for HTTP, another for chDB) is a maintenance trap — every new public method, every parameter change, every streaming or settings refactor would have to be applied twice.

1.2 Two tempting approaches that this proposal rejects

Approach A — A parallel "compat client" for chDB. Ship a separate ChdbClickHouseClient class that mirrors @clickhouse/client's public surface and translates calls onto chDB's native API. This is byte-compatible and works, but:

• Every public method, type, format, settings handler, and error class lives twice. Upstream changes drift silently.
• Preserving byte-compatibility tends to force eager-buffered semantics for query() and stream() (so errors surface where users expect them), which erases the zero-copy advantage that justifies using chDB in the first place.
• The translation glue grows linearly with the public surface — easily 1,500+ LOC of types, params, settings, formats, URL parsing, error mapping, and result reshaping that exists only to mirror upstream shapes.

Approach B — A loopback HTTP server inside chdb-node. Expose chDB through a localhost-only HTTP endpoint and let @clickhouse/client's existing HttpConnection consume it. Minimal JS-side code, but:

• Two serialization edges (chDB columns → wire format; wire format → JS objects), plus HTTP chunk framing, plus an extra buffer copy.
• chDB's native record-batch streaming becomes inaccessible behind a socket.
• chdb-node's embedded path is deliberately networkless (footprint, port management, lifecycle, single-process security model).

Both approaches trade chDB's reason for existing for ergonomics. This proposal proposes a third path that keeps the ergonomics and preserves the embedded advantage.

📦 2. Repository boundary — what lives where, and what is the only coupling surface

┌──────────────────────────────────────────┐    ┌──────────────────────────────────────────┐
│        @clickhouse/client repo           │    │       chdb-io/chdb-node repo             │
│   github.com/ClickHouse/clickhouse-js    │    │       github.com/chdb-io/chdb-node       │
│        (owned by CH team)                │    │           (owned by chDB team)           │
├──────────────────────────────────────────┤    ├──────────────────────────────────────────┤
│  src/                                    │    │  src/                                    │
│    client.ts        public semantics     │    │    connection.ts        ChdbConnection   │
│    connection.ts    Connection iface[new]│    │    extension.ts         .chdb extension  │
│    http_connection.ts  HTTP impl         │    │    test_profile.ts      skip manifest    │
│    https_connection.ts HTTPS impl        │    │  package.json                            │
│                                          │◄───┤    "exports": {                          │
│  createClient(opts: { connection? })     │    │      "./connection":                     │
│                                          │    │        "./dist/connection.js"            │
│  test/                                   │    │    }                                     │
│    fixtures/  parameterized over conns   │    │  test/                                   │
│    contract/  Connection contract suite  │    │    runs full @clickhouse/client suite    │
│                                          │    │      against ChdbConnection              │
│  ✗ zero chdb code                        │    │  ✓ CI installs chdb-node + clickhouse-js │
│  ✗ zero chdb tests                       │    │  ✓ test_profile.json is authoritative    │
│  ✗ CI does not depend on chdb-node       │    │                                          │
└──────────────────────────────────────────┘    └──────────────────────────────────────────┘
                    ▲                                              │
                    │     interface signature + named exports       │
                    └──────────────────────────────────────────────┘
                            (the only coupling surface)

The coupling surface is intentionally tiny:

• The exported Connection TypeScript interface signature.
• One package subpath export: chdb-node/connection.

Everything else — chDB version, Layer 1 native API quirks, output format details, FFI shape, filesystem paths — is invisible to @clickhouse/client. The ChdbConnection adapter consumes chdb-node's already-public Layer 1 API (Session, queryAsync, streamQuery, record-batch reader) and does not require any changes to chdb-core.

🔧 3. Refactor and abstraction — before/after, plus the interface-evolution discipline

Before

ClickHouseClient
  ├─ Public API:   query / query_arrow / insert / exec / command / ping / close
  ├─ Semantics:    ResultSet, type handling, settings, error model, format negotiation
  └─ Transport:    HttpConnection / HttpsConnection (private internal classes)

ClickHouseClient is the only client; transport is selected internally based on URL scheme but cannot be replaced by a user.

After

ClickHouseClient                          ← single semantics layer, shared by all connections
  ├─ query / query_arrow / insert / exec / command / ping / close
  ├─ ResultSet / type handling / format negotiation
  ├─ settings normalization, unified errors
  └─ this.connection: Connection ◄─┐
                                   │  injected by createClient({ connection })
        ┌──────────────────────────┴──────────────────┐
        │                                             │
   HttpConnection                              ChdbConnection
   (@clickhouse/client repo)                   (chdb-node repo)
   - query / insert / command via HTTP         - query via chdb Session.queryAsync()
   - undici / agentkeepalive pool              - query stream: native record-batch reader
   - supportsZeroCopyStreaming = false           via Layer 1 streamQuery
     (HTTP wire bytes always cross a            - insert via Python() table function for
      socket boundary; streaming gains are        zero-copy DataFrame ingest where the
      bounded by network framing)                 caller passes a typed array
                                                - supportsZeroCopyStreaming = true

Transport is extracted from the client's internal selection logic into an explicit connection injection. Public semantics, type handling, settings, the error model, and the async/Promise surface all stay in one shared layer.

Interface-evolution discipline

To prevent the Connection interface from churning and breaking registered connections:

1. Methods are additive only — never remove, never change a method signature.
2. Every new method ships with a default implementation that falls back to existing primitives, so old connections keep working without code changes.
3. Capabilities are declared via supports* flags and are additive — flag meanings are never repurposed; new capabilities get new flag names.

With this discipline, interface evolution is a one-way ratchet: @clickhouse/client can add anything, but cannot reach back and break an already-shipped connection.

🧬 4. API design — Connection interface, capability negotiation, .chdb extension namespace

4.1 Connection interface (defined in @clickhouse/client)

// @clickhouse/client/src/connection.ts (public export)

export interface Connection {
  // ── Identity ──────────────────────────────────────────────
  readonly connectionName: string

  // ── Capability flags ──────────────────────────────────────
  // Routing flag with real semantic impact today:
  //   true  → ClickHouseClient.query streaming should prefer the
  //           native record-batch path (genuinely zero-copy in-process)
  //   false → ClickHouseClient.query streaming should keep using
  //           the chunked-bytes path
  readonly supportsZeroCopyStreaming: boolean

  // ── Core transport (mandatory) ────────────────────────────
  query(params: QueryParams): Promise<QueryResult>
  queryStream(params: QueryParams): AsyncIterable<Uint8Array | RecordBatch>
  insert(params: InsertParams): Promise<InsertResult>
  exec(params: ExecParams): Promise<ExecResult>
  command(params: CommandParams): Promise<CommandResult>
  ping(): Promise<boolean>
  close(): Promise<void>

  // ── Error mapping (connection-specific errors never leak) ─
  mapError(err: unknown): ClickHouseError

  // ── Server metadata ───────────────────────────────────────
  readonly serverVersion: Promise<string>
  readonly serverTimezone: Promise<string>
}

export interface QueryResult {
  stream: AsyncIterable<Uint8Array>
  summary: ClickHouseSummary
  queryId: string
}

Connection	supportsZeroCopyStreaming	Why
HttpConnection	false	HTTP wire bytes always cross a socket; streaming gains are bounded by chunked-transfer framing. The client's existing chunked-bytes path is the right model.
HttpsConnection	false	Same as HTTP.
ChdbConnection	true	chDB exposes an in-process native record-batch reader via Layer 1's streamQuery. The client routes streaming through it directly, zero-copy.

4.2 Capability negotiation in ClickHouseClient (this is where zero-copy is preserved)

// @clickhouse/client/src/client.ts
async queryStream(params: QueryParams): Promise<AsyncIterable<RowOrBatch>> {
  if (this.connection.supportsZeroCopyStreaming) {
    // in-process; native record-batch reader; no socket; no wire format
    return this.connection.queryStream(params)
  }
  // existing chunked-bytes streaming path; behavior unchanged for HTTP users
  return this.#chunkedStream(params)
}

The capability flag controls only the routing decision inside the streaming entry points. On chDB the native path is strictly better (zero-copy); on HTTP it isn't (no socket-side benefit). For the materialized-row paths (query().then(rs => rs.json())) both connections converge through queryStream and the same row-shaping code.

4.3 .chdb extension namespace (for chDB-unique APIs)

chDB has capabilities that have no equivalent on a remote ClickHouse server. Rather than adding optional methods to ClickHouseClient, they surface via a typed extension that is only present when the chdb connection is in use:

import { createClient } from '@clickhouse/client'
import { createChdbConnection } from 'chdb-node/connection'

const client = createClient({ connection: createChdbConnection({ path: ':memory:' }) })

// chdb-only surface, typed:
await client.chdb.queryPython({ sql, df: myDataFrame })  // Python() table function
client.chdb.registerFunction(myUdf)                       // UDF registration
client.chdb.sessionPath                                   // in-memory vs persistent path
client.chdb.cursor()                                      // native DB-API cursor (escape hatch)

The typing trick (TypeScript discriminated brand on the connection) makes client.chdb exist in the type system only when the connection is ChdbConnection:

// In @clickhouse/client:
type ConnectionBrand<C extends Connection> = C extends { __brand: infer B } ? B : 'http'
type ChdbSurface<C> = ConnectionBrand<C> extends 'chdb' ? { chdb: ChdbExtension } : {}
type ClickHouseClient<C extends Connection = Connection> = BaseClient & ChdbSurface<C>

// In chdb-node:
export interface ChdbConnection extends Connection { readonly __brand: 'chdb' }

Accessing client.chdb on an HTTP connection is a TypeScript error at compile time and a runtime undefined. The ChdbExtension interface and its implementation live in the chdb-node repo (src/extension.ts); @clickhouse/client provides only the type-level brand check.

👤 5. User experience — installation and one-line switch

import { createClient } from '@clickhouse/client'

// Default HTTP — existing users: zero changes, full forward compatibility.
const client = createClient({
  url: 'https://prod.clickhouse.cloud',
  username: '...',
  password: '...',
})

// Switch to chDB in-process — one line.
import { createChdbConnection } from 'chdb-node/connection'
const client = createClient({
  connection: createChdbConnection({ path: ':memory:' }),
})

// All downstream code is identical across connections.
const rs = await client.query({ query: 'SELECT * FROM events LIMIT 100' })
for await (const row of rs.stream()) { /* ... */ }
await client.insert({ table: 'events', values: rows })

Installation:

npm install @clickhouse/client                # existing users, unchanged
npm install @clickhouse/client chdb-node      # users who want chDB add chdb-node

After chdb-node is installed, importing chdb-node/connection is the explicit opt-in. The @clickhouse/client bundle size, dependency tree, and TypeScript declarations are unchanged for users who don't install chdb-node.

If a user passes a connection instance built against an incompatible Connection interface version, createClient() throws a clear IncompatibleConnectionError pointing at the install command and the supported interface version.

Note on registration mechanism. This proposal uses explicit dependency injection via the connection option — no global registries, no side-effect imports, easier to reason about under tree-shaking and bundlers. The trade-off is that the user writes one extra import line; in exchange they get a fully typed, explicit configuration with no runtime magic.

📋 6. Code-change responsibility matrix — who changes what, when

Scenario	@clickhouse/client changes?	chdb-node changes?
chDB engine upgrade (new version, perf, bug fix)	❌ no	✅ connection.ts updated
chDB output-format behavior change	❌ no	✅ connection.ts adapted
CH server adds a new public feature → ClickHouseClient gains a new API	✅ add client.foo() (default impl on Connection falls back to query)	usually no; optionally implement a fast path in ChdbConnection
HttpConnection perf / pool tweaks	✅ HttpConnection	❌ no
chDB adds a unique capability (new UDF kind, new Python integration)	❌ no	✅ extend .chdb namespace in src/extension.ts
Test discovers chDB doesn't support a CH feature	❌ no	✅ add a marker to test_profile.SKIP_MARKERS
Connection interface itself needs a new method	✅ add method + default impl + capability flag	optionally implement optimized path
User reports buggy SQL behavior on chDB	❌ no	✅ chdb-node team investigates
User reports buggy behavior on HTTP	✅ CH team investigates	❌ no

Invariant: a chDB upstream change never forces a @clickhouse/client change.

🧪 7. Testing architecture — parameterized fixtures, marker hygiene, skip manifest, CI split, contract suite

7.1 Parameterized fixtures — write once in @clickhouse/client

// @clickhouse/client/test/fixtures.ts
import { describe, beforeEach } from 'vitest'
import { createClient } from '@clickhouse/client'

const connections = ['http']
try {
  const { createChdbConnection } = await import('chdb-node/connection')
  connections.push('chdb')
} catch { /* chdb-node not installed; HTTP-only run */ }

export function describeForEachConnection(name: string, body: (factory: () => ClickHouseClient) => void) {
  for (const conn of connections) {
    describe(`${name} [${conn}]`, () => body(() => makeClient(conn)))
  }
}

This fixture does not let chdb-node affect @clickhouse/client's CI — the dynamic import() either succeeds (chdb-node is installed → adds the [chdb] parameter) or throws and is silently caught (chdb-node is not installed → fixture only enumerates ['http']):

• In the @clickhouse/client repo's CI, chdb-node is not installed (package.json dependencies / devDependencies / workflow files don't reference it) → the dynamic import rejects → the fixture produces ['http'] only → tests fan out on http only, behavior identical to today.
• The same fixture in the chdb-node repo's CI (which does install chdb-node and @clickhouse/client) automatically picks up the [chdb] parameter dimension and runs the full suite against ChdbConnection — that's chdb-node's CI burden, and @clickhouse/client doesn't see it.

So CH developers writing new tests do not need to know anything about chDB — their tests fan out across connections for free, and CI behavior is unchanged.

7.2 Marker hygiene (the only new habit for CH developers)

Tag tests with the server feature they depend on. This is good test hygiene that pays for itself independently of chDB:

test.requiresFeature('distributed', async () => { /* ... */ })
test.requiresFeature('keeper',      async () => { /* ... */ })
test.requiresFeature('replicated',  async () => { /* ... */ })
test.requiresFeature('grants',      async () => { /* ... */ })
test.requiresFeature('http_only',   async () => { /* ... */ })  // compression negotiation, proxy

The feature set lives in @clickhouse/client's test helpers and grows organically.

7.3 Skip manifest — owned by chdb-node

// chdb-node/dist/test_profile.json  ── inside the chdb-node repo
{
  "skipMarkers": [
    "requires_distributed",
    "requires_replicated",
    "requires_keeper",
    "requires_grants",
    "requires_async_insert",
    "http_only"
  ],
  "skipNodeIds": {
    "test/query.test.ts > complex codecs": "chdb does not yet support ALTER MODIFY CODEC"
  }
}

@clickhouse/client's test runner loads the profile via the package's subpath export and applies skips dynamically. This code is written once and never touched again.

chdb-node owns data, not code. The skip manifest is a declarative file maintained in the chdb-node repo.

7.4 CI matrix split

@clickhouse/client repo CI:                    chdb-node repo CI:
  matrix: connection=[http]                      npm install chdb-node @clickhouse/client
  runs against a real CH server                  vitest --pool=threads test/upstream
  ─ never fails because of chdb-node             ─ runs the entire CH suite on chdb conn
  ─ never pins to a chdb-node version            ─ chdb-node upstream changes surface here first
                                                 ─ chdb-node team fixes connection or
                                                   extends skip manifest

The chdb-node repo also subscribes to @clickhouse/client's release tags and re-runs the regression on each new CH release. CH developers are transparent to this pipeline — they do nothing for it.

7.5 Connection-contract suite (@clickhouse/client repo)

A small generic suite asserting invariants any connection must satisfy. Runs against the HTTP connection and a mock connection in CH's CI. The chdb-node repo runs the same suite against ChdbConnection. This is the safety net that guarantees connection behavior is predictable.

---

Decoupling mechanisms — summary

┌──────────────────────────────────────────────────────────────────┐
│  1. Connection interface (code contract)                         │
│     additive only + default impls + capability flags             │
│     → interface evolution never breaks shipped connections       │
├──────────────────────────────────────────────────────────────────┤
│  2. Explicit DI via createClient({ connection })                 │
│     no global registry; no side-effect imports                   │
│     → @clickhouse/client does not import chdb-node;              │
│       chdb-node does not fork @clickhouse/client                 │
├──────────────────────────────────────────────────────────────────┤
│  3. .chdb extension namespace (unique-API outlet)                │
│     TS-typed brand check; runtime presence iff chdb connection   │
│     → chDB-only capabilities do not pollute the base interface   │
├──────────────────────────────────────────────────────────────────┤
│  4. Skip profile (testing boundary as data)                      │
│     feature markers in CH, JSON manifest in chdb-node            │
│     → CH writes tests without knowing chDB; chdb-node does not   │
│       fork CH's test suite                                       │
├──────────────────────────────────────────────────────────────────┤
│  5. CI matrix split                                              │
│     CH CI depends on zero chdb-node state                        │
│     → chDB upstream changes never turn CH CI red                 │
└──────────────────────────────────────────────────────────────────┘

---

Next step

If this direction looks right to the maintainers, the staged landing plan would be:

1. Promote Connection to a public interface in @clickhouse/client (extract today's private shape, add the connection?: option to createClient(), default behavior byte-identical to today).
2. Add the capability flag + extension brand (supportsZeroCopyStreaming, the TS brand machinery for .chdb) — purely additive.
3. Ship ChdbConnection in chdb-node (src/connection.ts, src/extension.ts, src/test_profile.ts, the ./connection subpath export, plus a chdb-node-side CI job that runs @clickhouse/client's test suite against ChdbConnection).

Step 1 is the only step that touches production HTTP behavior; steps 2 and 3 are strictly additive on the @clickhouse/client side.

The same architectural shape is being discussed in parallel for the Python ecosystem at ClickHouse/clickhouse-connect#809; adopting it here keeps the two ecosystems aligned.

Happy to discuss any part of the proposal here or in a sync.