Skip to content

Commit f18a9fb

Browse files
z3z1maz3z1ma
committed
docs: update readme
1 parent 501229e commit f18a9fb

File tree

2 files changed

+131
-71
lines changed

2 files changed

+131
-71
lines changed

‎README.md‎

Lines changed: 127 additions & 57 deletions
Original file line numberDiff line numberDiff line change
@@ -1,97 +1,167 @@
1-
# Dbt Core Interface
1+
<div id="top">
22

3-
[![PyPI](https://img.shields.io/pypi/v/dbt-core-interface.svg)][pypi_]
4-
[![Status](https://img.shields.io/pypi/status/dbt-core-interface.svg)][status]
5-
[![Python Version](https://img.shields.io/pypi/pyversions/dbt-core-interface)][python version]
6-
[![License](https://img.shields.io/pypi/l/dbt-core-interface)][license]
3+
<!-- HEADER STYLE: CLASSIC -->
74

8-
[![Read the documentation at https://dbt-core-interface.readthedocs.io/](https://img.shields.io/readthedocs/dbt-core-interface/latest.svg?label=Read%20the%20Docs)][read the docs]
9-
[![Tests](https://github.com/z3z1ma/dbt-core-interface/workflows/Tests/badge.svg)][tests]
5+
<div align="left">
106

11-
[![pre-commit](https://img.shields.io/badge/pre--commit-enabled-brightgreen?logo=pre-commit&logoColor=white)][pre-commit]
12-
[![Black](https://img.shields.io/badge/code%20style-black-000000.svg)][black]
7+
<img src="https://chatgpt.com/backend-api/public_content/enc/eyJpZCI6Im1fNjg2NDkwZjkzNjU4ODE5MTgyYTg0ZDA4YTJmZDU4ZGI6ZmlsZV8wMDAwMDAwMDk4Nzg2MWY1YTAyNGVkOGZjZDQ4MDIyOCIsInRzIjoiNDg2NTA1IiwicCI6InB5aSIsInNpZyI6IjgwM2NlMDg1NzM1YTNjMjEzMzVhYWFhMzg5NDcxMThmYTgzYTE4MzhiNGVkMzRjNTNkNjMzZTVlOTg0NGU3NTAiLCJ2IjoiMCIsImdpem1vX2lkIjpudWxsfQ==" width="30%" style="position: relative; top: 0; right: 0;" alt="Project Logo"/>
138

14-
[pypi_]: https://pypi.org/project/dbt-core-interface/
15-
[status]: https://pypi.org/project/dbt-core-interface/
16-
[python version]: https://pypi.org/project/dbt-core-interface
17-
[read the docs]: https://dbt-core-interface.readthedocs.io/
18-
[tests]: https://github.com/z3z1ma/dbt-core-interface/actions?workflow=Tests
19-
[pre-commit]: https://github.com/pre-commit/pre-commit
20-
[black]: https://github.com/psf/black
9+
# DBT-CORE-INTERFACE
2110

22-
## Features
11+
<em>Lightweight, thread-safe, multi-project Python interface to dbt-core</em>
2312

24-
An extremely simplified interface is provided to accomplish all of the following with no dependencies outside dbt-core:
13+
<!-- BADGES -->
2514

26-
- Parse dbt project on disk loading dbt core classes into memory from a single class/interface
15+
<img src="https://img.shields.io/github/license/z3z1ma/dbt-core-interface?style=flat-square&logo=opensourceinitiative&logoColor=white&color=0080ff" alt="license">
16+
<img src="https://img.shields.io/github/last-commit/z3z1ma/dbt-core-interface?style=flat-square&logo=git&logoColor=white&color=0080ff" alt="last-commit">
17+
<img src="https://img.shields.io/github/languages/top/z3z1ma/dbt-core-interface?style=flat-square&color=0080ff" alt="repo-top-language">
18+
<img src="https://img.shields.io/github/languages/count/z3z1ma/dbt-core-interface?style=flat-square&color=0080ff" alt="repo-language-count">
2719

28-
- Automatic management of the adapter and thread-safe efficient connection pool reuse
20+
<em>Built with the tools and technologies:</em>
2921

30-
- Run SQL and get results in python fully independent of the dbt adapter which automatically enables support for many databases
22+
<img src="https://img.shields.io/badge/TOML-9C4121.svg?style=flat-square&logo=TOML&logoColor=white" alt="TOML">
23+
<img src="https://img.shields.io/badge/Rich-FAE742.svg?style=flat-square&logo=Rich&logoColor=black" alt="Rich">
24+
<img src="https://img.shields.io/badge/Ruff-D7FF64.svg?style=flat-square&logo=Ruff&logoColor=black" alt="Ruff">
25+
<img src="https://img.shields.io/badge/GNU%20Bash-4EAA25.svg?style=flat-square&logo=GNU-Bash&logoColor=white" alt="GNU%20Bash">
26+
<img src="https://img.shields.io/badge/FastAPI-009688.svg?style=flat-square&logo=FastAPI&logoColor=white" alt="FastAPI">
27+
<br>
28+
<img src="https://img.shields.io/badge/Pytest-0A9EDC.svg?style=flat-square&logo=Pytest&logoColor=white" alt="Pytest">
29+
<img src="https://img.shields.io/badge/Docker-2496ED.svg?style=flat-square&logo=Docker&logoColor=white" alt="Docker">
30+
<img src="https://img.shields.io/badge/Python-3776AB.svg?style=flat-square&logo=Python&logoColor=white" alt="Python">
31+
<img src="https://img.shields.io/badge/GitHub%20Actions-2088FF.svg?style=flat-square&logo=GitHub-Actions&logoColor=white" alt="GitHub%20Actions">
32+
<img src="https://img.shields.io/badge/uv-DE5FE9.svg?style=flat-square&logo=uv&logoColor=white" alt="uv">
33+
</div>
3134

32-
- Run SQL with dbt SQL from a single method call
35+
---
3336

34-
- Load macros at runtime enabling custom functionality in third party extensions without requiring the dbt packaging system to be managed in userland
37+
## Overview
3538

36-
- Compile dbt jinja extremely fast and efficiently, thread-safe and stress tested at load via a Bottle server which live compiles SQL
39+
`dbt-core-interface` is a lightweight, high-performance Python interface for working directly with `dbt-core` (v1.8+). It allows developers to manage and run dbt projects entirely in memory using an intuitive Python API—enabling runtime SQL compilation, macro evaluation, SQLFluff linting/formatting, and more, all through FastAPI or local usage.
3740

38-
- Manage multiple dbt projects in a single process using the DbtProjectContainer class
41+
It supports dynamic multi-project environments, automatic re-parsing, file watchers, and asynchronous usage. It is the foundation for more complex interfaces such as `dbt-fastapi` and is designed to rapidly prototype ideas outside the constraints of the dbt-core repo itself.
3942

40-
`dbt-core-interface` is a wrapper that allows developers to rapidly develop features and integrations for dbt. This project aims to serve as a place for the community to aggregate the best ways to interface with dbt. It is afforded a much faster iteration cycle and much more freedom due to it's independence from the dbt codebase. It is intended to act as an common library to dbt's existing APIs for developers. Implementations can land here and prove themselves out before landing in the dbt-core codebase and benefit all developers involved. Sqlfluff dbt templater, dbt-osmosis, dbt-fastapi which I am ripping out of dbt-osmosis, an impending metadata manager, a testing framework will all leverage this library. As dbt core evolves and stabilizes its python API, this project will evolve with it. This may manifest in simplification of certain methods but our goal is to maintain the API and focus on driving efficient, innovative/creative, and agile community driven integration patterns.
43+
---
4144

42-
## Requirements
45+
## Features
4346

44-
- The **only** requirement is dbt-core, tested with versions `1.0.*`, `1.1.*`, `1.2.*`, `1.3.*`, `1.4.*`, `1.5.*`
47+
* 🧐 In-memory dbt-core 1.8+ interface with full `RuntimeConfig` hydration
48+
* ⚡ Fast, thread-safe SQL compilation and execution via FastAPI
49+
* 🔬 Interactive linting and formatting with SQLFluff
50+
* 🌐 Live REST API server via FastAPI
51+
* 🌍 Supports multiple projects simultaneously using `DbtProjectContainer`
52+
* 🚀 Dynamic macro parsing, Jinja rendering, manifest manipulation
53+
* 🔄 Background file watching for auto-reparsing
54+
* ⚖ Direct dbt command passthrough (e.g. `run`, `test`, `docs serve`, etc.)
4555

46-
## Installation
56+
---
4757

48-
You can install _Dbt Core Interface_ via [pip] from [PyPI]:
58+
## Requirements
4959

50-
```console
60+
* Python 3.9+
61+
* `dbt-core >= 1.8.0`
62+
63+
Install via PyPI:
64+
65+
```bash
5166
pip install dbt-core-interface
5267
```
5368

69+
---
70+
5471
## Usage
5572

56-
Please see the [Api Reference] for details.
73+
### Programmatic
5774

58-
To launch the Bottle server for live compiling dbt jinja:
75+
```python
76+
from dbt_core_interface import DbtProject
5977

60-
python -m dbt_core_interface.project
78+
# Load your project
79+
project = DbtProject(project_dir="/path/to/dbt_project")
6180

62-
This will launch the server on port 8581. You can then make requests to the server, e.g.:
81+
# Run a simple SQL query
82+
res = project.execute_sql("SELECT current_date AS today")
83+
print(res.table)
6384

64-
curl -X POST -H "Content-Type: application/json" -H "X-dbt-Project: dbt_project" -d '{"project_dir":"/app/tests/sqlfluff_templater/fixtures/dbt/dbt_project/","profiles_dir":"/app/tests/sqlfluff_templater/fixtures/dbt/profiles_yml/","target":"dev"}' http://localhost:8581/register
85+
# Compile SQL (but don't run it)
86+
compiled = project.compile_sql("SELECT * FROM {{ ref('my_model') }}")
87+
print(compiled.compiled_code)
6588

66-
You can change the server hostname and port using the `--host` and `--port` arguments.
89+
# Execute a ref() lookup
90+
node = project.ref("my_model")
91+
print(node.resource_type, node.name)
6792

68-
## Contributing
93+
# Load a source node
94+
source = project.source("my_source", "my_table")
95+
print(source.description)
6996

70-
Contributions are very welcome.
71-
To learn more, see the [Contributor Guide].
97+
# Incrementally parse the project
98+
project.parse_project(write_manifest=True)
7299

73-
## License
100+
# Re-parse a specific path
101+
project.parse_paths("models/my_model.sql")
102+
103+
# Compile a node from path
104+
node = project.get_node_by_path("models/my_model.sql")
105+
compiled = project.compile_node(node)
106+
print(compiled.compiled_code)
107+
108+
# Run a dbt command programmatically
109+
project.run()
110+
project.test()
111+
112+
# SQLFluff linting
113+
lint_result = project.lint("models/my_model.sql")
114+
print(lint_result)
74115

75-
Distributed under the terms of the [MIT license][license],
76-
_Dbt Core Interface_ is free and open source software.
116+
# SQLFluff formatting
117+
success, formatted_sql = project.format("models/my_model.sql")
118+
print(formatted_sql)
119+
120+
# Use the DbtProjectContainer to manage multiple projects
121+
from dbt_core_interface import DbtProjectContainer
122+
123+
container = DbtProjectContainer()
124+
container.create_project(project_dir="/path/to/dbt_project_1")
125+
container.create_project(project_dir="/path/to/dbt_project_2")
126+
print(container.registered_projects())
127+
```
77128

78-
## Issues
129+
### Server Mode (FastAPI)
79130

80-
If you encounter any problems,
81-
please [file an issue] along with a detailed description.
131+
Run:
82132

83-
## Credits
133+
```bash
134+
python -m dbt_core_interface.server --host 0.0.0.0 --port 8581
135+
```
136+
137+
Register a project:
138+
139+
```bash
140+
curl -X POST 'http://localhost:8581/register?project_dir=/your/dbt_project'
141+
```
142+
143+
Compile SQL:
144+
145+
```bash
146+
curl -X POST 'http://localhost:8581/compile' -H 'X-dbt-Project: /your/dbt_project' -d 'select * from my_model'
147+
```
148+
149+
---
150+
151+
## Documentation
152+
153+
* 📚 [Read the Docs](https://dbt-core-interface.readthedocs.io/)
154+
* 📖 [API Reference](https://dbt-core-interface.readthedocs.io/en/latest/reference.html)
155+
* 🚑 [Health Check](http://localhost:8581/health)
156+
157+
---
158+
159+
## License
84160

85-
This project was generated from [@cjolowicz]'s [Hypermodern Python Cookiecutter] template.
161+
This project is licensed under the MIT License. See the [LICENSE](https://github.com/z3z1ma/dbt-core-interface/blob/main/LICENSE) file for more info.
86162

87-
[@cjolowicz]: https://github.com/cjolowicz
88-
[pypi]: https://pypi.org/
89-
[hypermodern python cookiecutter]: https://github.com/cjolowicz/cookiecutter-hypermodern-python
90-
[file an issue]: https://github.com/z3z1ma/dbt-core-interface/issues
91-
[pip]: https://pip.pypa.io/
163+
---
92164

93-
<!-- github-only -->
165+
## Acknowledgments
94166

95-
[license]: https://github.com/z3z1ma/dbt-core-interface/blob/main/LICENSE
96-
[contributor guide]: https://github.com/z3z1ma/dbt-core-interface/blob/main/CONTRIBUTING.md
97-
[api reference]: https://dbt-core-interface.readthedocs.io/en/latest/reference.html
167+
Thanks to the dbt-core maintainers and contributors whose work makes this project possible.

‎src/dbt_core_interface/server.py‎

Lines changed: 4 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -340,26 +340,16 @@ def compile_sql(
340340
@app.post("/register")
341341
def register_project(
342342
response: Response,
343-
project: str = Header(None, alias="X-dbt-Project"),
343+
project_dir: str = Query(...),
344344
profiles_dir: str | None = Query(None),
345-
project_dir: str | None = Query(None),
346345
target: str | None = Query(None),
347346
runners: DbtProjectContainer = Depends(get_container),
348347
) -> ServerRegisterResult | ServerErrorContainer:
349348
"""Register a new dbt project with the server."""
350-
if not project:
351-
response.status_code = 400
352-
return ServerErrorContainer(
353-
error=ServerError(
354-
code=ServerErrorCode.ProjectHeaderNotSupplied,
355-
message="Header X-dbt-Project is required for registration.",
356-
data={},
357-
)
358-
)
359-
project_path = Path(project).expanduser().resolve()
349+
project_path = Path(project_dir).expanduser().resolve()
360350
if project_path in runners.registered_projects():
361351
return ServerRegisterResult(
362-
added=project, projects=list(map(str, runners.registered_projects()))
352+
added=project_path.name, projects=list(map(str, runners.registered_projects()))
363353
)
364354
try:
365355
dbt_project = runners.create_project(
@@ -380,7 +370,7 @@ def register_project(
380370
)
381371
)
382372
return ServerRegisterResult(
383-
added=project, projects=list(map(str, runners.registered_projects()))
373+
added=project_path.name, projects=list(map(str, runners.registered_projects()))
384374
)
385375

386376

0 commit comments

Comments
 (0)