Add support for Cloudflare D1 via external DB-API + SQLAlchemy dialect

[ChadRosseau]

Add support for Cloudflare D1 via external DB-API + SQLAlchemy dialect

Problem

Superset requires a Python DB-API driver and a SQLAlchemy dialect in order to connect to different database engines. Cloudflare D1 (a serverless SQLite-compatible database built on Workers) isn’t currently supported. This makes it impossible for teams using D1 as their backend to connect it as a datasource in Superset.

Ourselves and many of our peers are avid users of the Cloudflare stack, including D1, and would welcome a Superset integration with D1.

Proposal

Our team* would like to build an external Python package that adds:

• A DB-API 2.0 compliant driver for D1, wrapping its HTTP API
• A SQLAlchemy dialect built on top of that driver
• The database-specific connector code required to enable extra features, as per this page

As per the Superset 'Connecting to Databases' documentation, these are the two crucial components in a database integration.

Users would then be able to pip install the driver and set up connections in Superset using a connection string such as:

d1://<api_token>@<account_id>/<database_name>

This approach keeps changes outside of Superset core. Superset will discover the dialect if installed. The only possible core addition later would be adding d1-specific instructions to superset documentation, such as in docs/docs/configuration/databases.mdx.

Limitations We’re Aware Of

D1 is not a traditional socket-based database, so some DB-API features can’t be fully supported. For example:

• commit() will effectively be a no-op, since D1 autocommits.
• rollback() cannot be supported and would raise NotImplementedError.

We believe these tradeoffs are acceptable since they align with how D1 itself works.

Questions for Maintainers

1. While we've looked through current issues and discussions and not found anything, has there been any previous work on Cloudflare D1 integration?
2. Is the plan to implement this as an external package (and not in core) the right path?
3. Would Superset be open to listing D1 as a supported engine in the docs once the package is stable?
4. Are there any dialect features beyond reflection and query execution that Superset expects at minimum?

Thank you in advance for your ideas and feedback.

*cc: @danielalyoshin, @jang-35, @murphylee10, @ThunderRoar

[ThunderRoar]

Hi @rusackas! I am Shreyas Rao, a student at University of Toronto Scarborough — it was great meeting you and the community during the Townhall today. Our team would like to take the initiative to work on the above feature to add Cloudflare D1 as a valid Superset data source. Before we begin, we would appreciate your feedback on whether this approach aligns with Superset’s expectations. Thank you!

[rusackas]

@betodealmeida can provide advice on the pros and cons of externalizing the DB Engine Spec, but all in all we agree this is a fantastic addition! And yes, we/you can definitely add it to the docs!

[ChadRosseau]

Thank you for your feedback, it's great to hear that the Superset team is on board with this idea. We'd appreciate any insights on externalizing the spec. We'll get started on an implementation in the meantime, and keep you in the loop. Looking forward to working with you all!