[Design] Generic ADBC connector for scanning relational databases (MySQL, PostgreSQL, ...)

[jja725]

Hi all — I'd like to gather community feedback on adding a generic ADBC connector to Velox, so applications embedding Velox can scan relational databases (MySQL, PostgreSQL, SQLite, Snowflake, Flight SQL, ...) natively. An initial read-only implementation is up in #18258, with design notes in docs/designs/adbc-connector.md on that branch. There is also a companion Presto RFC proposing to use it from Prestissimo C++ workers (prestodb/rfcs#64), as a complement to the Java connector federation approach (RFC-0018).

Why ADBC

Arrow ADBC is a vendor-neutral C API for database access whose drivers return results as an Arrow ArrowArrayStream. Since Velox already has an Arrow C-ABI bridge, the whole data path — database wire protocol → Arrow (inside the driver) → Velox vectors (importFromArrowAsOwner, buffers adopted without copying) — runs in-process with a single format conversion. One connector serves every database; per-database behavior lives in the driver and in configuration, similar to how JDBC-based connectors work on the Java side.

Design summary (as implemented in #18258)

• Driver loading: the ADBC driver manager (Apache 2.0, one .cc + two headers, no third-party deps at the vendored apache-arrow-adbc-14 tag) is vendored into velox/external/adbc, following the velox/external pattern. Drivers are shared libraries loaded at runtime; embedders that link a driver statically can pass an AdbcDriverInitFunc instead.
• Configuration: adbc.driver selects the driver; any adbc.option.<name> key passes through to AdbcDatabaseSetOption (adbc.option.uri, adbc.option.username, ...), so new drivers need no Velox changes. adbc.identifier-quote controls quoting in generated SQL (backtick for MySQL).
• Scan model: one split per scan (a remote database is not splittable). AdbcTableHandle names either a table — the connector generates SELECT <projected columns> FROM <table> (projection pushdown) — or a raw SQL query executed verbatim, which lets host engines do their own SQL generation/pushdown and hand Velox the final text.
• Schema handling: result columns are matched to the output type by case-insensitive name and validated for type equivalence; the stream schema is deep-copied per chunk so each imported batch owns its Arrow structures.
• Build: everything is behind VELOX_ENABLE_ADBC_CONNECTOR (default OFF). Tests use a hermetic in-process fake driver (serves Velox vectors exported through the Arrow bridge, records the SQL it receives) — no database or driver library needed in CI.

Open questions where input would help

1. Dependency approach: is vendoring the driver manager into velox/external acceptable, or would resolve_dependency on arrow-adbc be preferred (at the cost of a new third-party dependency and the toml++ transitive dep in newer releases)?
2. Pushdown surface: today filter pushdown is delegated to the host engine via the query pass-through mode. Should Velox itself grow a SubfieldFilters → SQL translation (dialect-aware), or is engine-side SQL generation the right long-term boundary?
3. Parameter binding: engines like Presto push predicates as SQL with ? placeholders plus bound constants. Supporting this natively maps to AdbcStatementBind (parameters as an Arrow batch). Worth adding to the connector API, or should engines inline literals?
4. Memory accounting: imported vectors reference driver-allocated Arrow buffers outside the Velox MemoryPool (like other importFromArrowAsOwner users). Is per-scan accounting of driver buffers something we should design for now?
5. Parallelism: single split per scan today. Is key-range sharding (N splits with generated range predicates) interesting enough to shape the split API now, or fine as a follow-up?
6. Writes: AdbcStatementBind also enables INSERT via a DataSink. Any appetite / use cases?
7. CI: default-OFF means no coverage unless a CI job enables the flag. Preference between adding it to an existing Linux job vs a dedicated adapter-style job?

Would love feedback on the overall direction as well as the specific questions. If the direction looks good I'll iterate on #18258 accordingly.

[jja725]

@mbasmanova @Yuhta @PingLiuPing would look to hear feedback from u guys

[mbasmanova]

@jja725 Could you say more about the real motivation? As written the design reads as theoretical rather than driven by a workload in production — which databases actually need to be queried, in what deployment, and what volumes of data are being fetched? The volume question matters most: a non-splittable, synchronous connection is a single stream pulled by one driver thread, so I'd like to understand whether the data sizes you have in mind are feasible on it at all, and what throughput looks like against a real database.

Second, if the motivation is Prestissimo, why does this need to live in Velox? Prestissimo already hosts its own connectors — presto_cpp/main/connectors has arrow_flight (including arrow_federation) and SystemConnector, registered through its own Registration.cpp — so there is precedent for exactly this shape one layer up, and arrow_federation looks like it already overlaps what you're proposing. What belongs in Velox is what multiple embedders need — would Gluten use this too? Both Spark and Presto already reach relational databases from the JVM (you cite RFC-0018 for Presto), so the prior question is what moving the scan into the C++ worker buys over that existing path, for either engine. If the answer only holds for Prestissimo, that argues for keeping it there.

[mbasmanova]

CC: @majetideepak @aditi-pandit

[jja725]

Thanks @mbasmanova , these are fair questions. There is a concrete production motivation: we’re migrating production workloads using the Presto MySQL connector to Velox/Prestissimo.
One option is Presto’s FlightShim connector federation (https://prestodb.github.io/docs/current/presto_cpp/federation.html), but that requires operating a separate JVM service and forwarding reads over Arrow Flight. For relational databases supported by ADBC, loading the driver directly in the native worker seems operationally much simpler. The primary motivation is simplicity rather than a throughput claim at this point.
I agree that we need real workload numbers and benchmarks, especially given the single-stream limitation. I’ll gather those as we proceed with the MySQL migration.
I’m also fine keeping this in the Presto repo if that’s the right boundary. I posted here mainly to see whether Velox maintainers consider it a useful direction for multiple embedders. Spark/Gluten could potentially benefit from the same native relational-database connector, although I haven’t validated that with their maintainers yet.

[mbasmanova]

CC: @amitkdutta

[mbasmanova]

@jja725 Sounds good.

I’ll gather those as we proceed with the MySQL migration.

Please, share these when you have them. Let us know if you need any further help.

[aditi-pandit]

@mbasmanova @jja725 : The Prestissimo Federation connector lets us expose any Presto Java connector behind an Arrow Flight server facade. This helped us claim that Prestissimo could run any query that Presto Java with federation can. That project didn't seem like a good fit for Velox repo so we kept it in Presto repo.

ADBC is the more modern and generic approach for database connectivity that we should follow for Velox and Prestissimo. So I'm in favor of adding a connector for it.

I'm fine with any choice of repository.