feat: implement templating system based on tstrings#409
Open
NickCrews wants to merge 29 commits intoduckdb:mainfrom
Open
feat: implement templating system based on tstrings#409NickCrews wants to merge 29 commits intoduckdb:mainfrom
NickCrews wants to merge 29 commits intoduckdb:mainfrom
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is an implementation of #370. It is still a WIP, but it is now good enough that I think the shape of the API is 90% there. I wanted to open up this PR so @evertlammerts can take a look at it and give some high level comments. Once we iron the large scale things out then I can fix those up and do more polishing before I ask you for a more detailed review.
Summary
This PR implements a SQL templating system for duckdb-python based on Python t-strings (Discussion #370), while remaining usable from non-3.14 code paths.
The core idea is:
__duckdb_template__and are expanded as SQL fragments/subqueries.!s,!r,!a) are treated as explicit raw interpolation (matching Python semantics).For python 3.14+, a kitchen sink example:
What’s included
1) New
duckdb.templatemoduleIntroduces
CompiledSql(sql: str, params: dict[str, object])which is a simple dataclass to hold the compiled sql (with param names resolved).To create this for python <3.14 , you will most likely use the
template(*parts: str | Interpolationish | Param)to build aSqlTemplateobject. This SqlTemplate is very analogous to the Template builtin to python 3.14. It is made of a sequence of alternating strs and Interpolations. This also supports evaluating any object that implements a__duckdb_template__()method, which allows for some nice usage patterns.I keep this SqlTemplate a first class citizen so users can have programmatic access to the raw data structure, in case they want to get in the middle of the process. For example, the Interpolations in the SqlTemplate could themselves hold another SqlTemplate for arbitrary nesting! But most users skip straight to compiling to a
CompiledSql. They do this in two steps:SqlTemplate.resolve()takes this generic format and performs transformations to get it closer to the output format, returning aResolvedSqlTemplate, which recursively resolves the Templates and Interpolations, so that you get a FLAT sequence of strs and Interpolations, and the Interpolations are guaranteed to hold Params.ResolvedSqlTemplate.compile()actually resolves the final names for all the Params and generates the finalCompiledSqlBehavior highlights:
2) Query API integration (Python typing + runtime)
The main SQL entry points now accept template inputs (
SqlTemplate/CompiledSql) in addition to existing string/statement inputs, including:sql,query,from_query,execute,executemanytemplate()accepts. Eg should we acceptconn.sql(["SELECT * FROM users WHERE id = ", 123])?3) C++ execution path support
Adds normalization in
DuckDBPyConnectionto handle template-like objects at runtime:.compile(), it is compiled before statement parsing..sql/.params, those are consumed directly.This allows passing compiled/template objects directly into execution/query paths without requiring manual
.sql/.paramsunpacking by users.4) Built-in object support for interpolation
Adds
__duckdb_template__implementations for key DuckDB Python objects so they compose naturally in templates:DuckDBPyRelation-> SQL form (ToSQL)DuckDBPyExpression-> expression SQL stringDuckDBPyType-> type stringThis enables patterns like relation/type/expression interpolation directly inside templates/t-strings, eg
template("SELECT 42.5::", duckdb.DOUBLE)5) Tests
Adds broad coverage across: