0.1.2 warehousing using dbt, duckdb and iceberg #55

redpheonixx · 2024-10-25T18:29:03Z

Title: Boiler Plate Code for DBT Transformations

Assignees: Amit Singh Labels: Transformation layer

Description:

This pull request introduces a boilerplate code for DBT transformations using the medallion architecture. The transformations are structured across three layers:

Bronze Layer: Initial raw data ingestion and storage.

Silver Layer: Intermediate transformations to refine and standardize data.

Gold Layer: Final transformations for analytical purposes, providing clean and consumable data.

The implementation utilizes the Iceberg REST catalog to manage the datasets and transformations efficiently. This setup aims to streamline the development and maintenance of DBT models, ensuring a robust and scalable data pipelin

#45

wrote dbt transformation layer as per medallion architecture

dbt's low learning curve and SQL-based workflow empower data teams without advanced programming knowledge to build and maintain transformations. This makes it a better fit for companies looking for agility and simplicity in data pipelines.

…ansformation

tusharchou · 2024-10-28T16:33:19Z

local-data-platform/local_data_platform/makefile

@redpheonixx what issue is this associated with?

#57
references :

https://motherduck.com/blog/duckdb-python-e2e-data-engineering-project-part-1/

* Create publish.yml * Update publish.yml

* Create publish.yml * Update publish.yml * Update publish.yml

* Update publish.yml * Update publish.yml * Update publish.yml

* Update publish.yml * Update publish.yml * Update publish.yml * Update publish.yml updated yaml file to copy distribution o/p from build to root directory * Update publish.yml added detailed copy from /ldf/dist to gihub/wo../dist * Update publish.yml added a code for creating dist directory --------- Co-authored-by: Tushar Choudhary <[email protected]>

* Update publish.yml * Update publish.yml * Update publish.yml * Update publish.yml updated yaml file to copy distribution o/p from build to root directory * Update publish.yml added detailed copy from /ldf/dist to gihub/wo../dist * Update publish.yml added a code for creating dist directory * Update publish.yml relocating dots from ./github to /.github --------- Co-authored-by: Tushar Choudhary <[email protected]>

* Update publish.yml * Update pyproject.toml

* Update publish.yml * Update pyproject.toml * Update publish.yml

* Release v1.1 dist changes * Release v1.1 publish.yml changes * Release v1.1 publish.yml changes

Update publish.yml

Create manual.yml

Update publish.yml

* Release v1.1 dist changes * Release v1.1 publish.yml changes * Release v1.1 publish.yml changes * Release v1.1

tusharchou · 2024-11-08T07:47:24Z

Raising a pull request (PR) from a fork's main branch to a repository's main branch is generally discouraged for several reasons:

Complexity and Confusion: Using the main branch of your fork for PRs can create confusion if you make other changes or updates to your fork's main branch. It becomes harder to track what specific changes the PR is addressing.
Continuous Integration Issues: Many projects have automated checks and continuous integration (CI) workflows that run on branches. If you use the main branch for PRs, these workflows might get triggered unnecessarily, potentially causing conflicts or errors.
Maintenance Difficulty: It's easier to maintain a clean and organized history if you create a separate feature branch for each set of changes. This approach avoids cluttering the main branch and makes it simpler to manage and merge changes.
Conflict Management: Keeping changes in isolated branches helps in managing conflicts better. If your main branch is used for a PR and it diverges significantly from the upstream main branch, resolving conflicts can become more complicated.
Best Practices: Following best practices, like creating feature branches for specific changes or issues, improves collaboration and code quality. It also aligns with how most open-source projects and development teams operate.

Best Practice Workflow:

Create a Feature Branch:

git checkout -b feature/your-feature-name

Make Your Changes and Commit:
```
git commit -m "Describe your changes"
```

Push Your Feature Branch to Your Fork:

git push origin feature/your-feature-name

Open a PR from Your Feature Branch to the Upstream Main Branch:
- Go to the repository on GitHub.
- Open a PR from your-fork:feature/your-feature-name to upstream:main.

This approach keeps your development process clean and organized. It ensures that the main branches remain stable and only contain code that is ready for production.

Feel free to ask if you need more details or help with Git workflows! 🚀✨

…form

dbt transformations commit in new branch

* fixed bug warehouse uri Thu Oct 24 9:25 PM IST * added logging to CSV.get() * supported big query ts format * refactored parameter to config from catalog to reduce confusion * Fixed bug of logger in GCP * replaced local path with dynamic path * replaced local path with dynamic path * replaced local path with dynamic path * demo * demo * Release v1.1 dist changes * Release v1.1 publish.yml changes * Release v1.1 publish.yml changes * added a class implementation for github issue hoping it is useful for guiding users to resolution ETAs eventually * added a exception for PlanNotFound to ask users to raise issues on the repository for resolution * Updated overview and milestones. Added directory structure under technical specifications. * Updated components in technical speciifcations * added testing for BigQueryToCSV.extract() * added testing for BigQueryToCSV.extract() * added testing for Iceberg.get() * Release v1.1 * Release v1.1 bug fix * Release v1.1 bug fix * Release v1.1 bug fix * Pytest Added for BigQuery Source --------- Co-authored-by: Tushar Choudhary <[email protected]>

rakhioza07

Please fix review comments and move into draft PR until changes are made.

Great work on exception handling and clean coding efforts!

rakhioza07 · 2024-11-10T17:06:16Z

.github/workflows/publish.yml

+      - name: Set up Python
+        uses: actions/setup-python@v4
+        with:
+          python-version: "3.x"


Discuss and settle on a version of python 3.6.5 or higher

Check for library dependencies

rakhioza07 · 2024-11-11T16:38:56Z

local-data-platform/dist/local_data_platform-0.1.0.tar.gz

Please don't add tar.gz, whl or heavy source files as a part of code commits

Only consider code, sample data and config file, scripts for PRs

rakhioza07 · 2024-11-11T16:41:34Z

local-data-platform/local_data_platform/transformations/dbt_models/bronze/bronze.sql

Consider moving all SQL files to a central place

Consider keeping code, data, queries, configs in seperate directory structures

rakhioza07 · 2024-11-11T16:41:58Z

local-data-platform/local_data_platform/transformations/silver_layer.py

This script needs to be OOP

rakhioza07 · 2024-11-11T16:42:58Z

local-data-platform/poetry.lock

Does this file need to be a part of the commit? If no, please mention in gitignore

rakhioza07 · 2024-11-11T16:43:40Z

local-data-platform/real_world_use_cases/near_data_lake/tmp.py

+
+p=Path('C:/Users//singsina//Desktop//local-data-platform//local-data-platform//tmp//warehouse//')
+print(p)
+p_path="file:///"+str(p)


Use f-strings for string manipulation

redpheonixx added 2 commits October 25, 2024 18:38

folder structure for dbt transformation

0b92abe

adding python scripts for execution and orchestrator.py to trigger tr…

0deb407

…ansformation

tusharchou requested review from mrutunjay-kinagi and abhishekkar0603 October 28, 2024 14:00

tusharchou assigned tusharchou and redpheonixx and unassigned tusharchou Oct 28, 2024

tusharchou added this to the 0.1.2 Warehousing: DuckDB milestone Oct 28, 2024

adding makefile for 0.1.2

daab7d4

tusharchou linked an issue Oct 28, 2024 that may be closed by this pull request

0.1.2 Warehousing: add duckdb, dbt and iceberg packages #57

Open

redpheonixx changed the title ~~boiler plate code for dbt transformation~~ boiler plate code for warehouse 0.1.2 Oct 28, 2024

tusharchou reviewed Oct 28, 2024

View reviewed changes

redpheonixx added 2 commits October 28, 2024 22:12

Merge branch 'tusharchou:main' into main

0e72bdd

adding sqlachemy package

c1414f1

redpheonixx changed the title ~~boiler plate code for warehouse 0.1.2~~ 0.1.2 warehousing using dbt, duckdb and iceberg Oct 28, 2024

tusharchou marked this pull request as draft October 29, 2024 12:19

tusharchou added 8 commits October 30, 2024 19:09

V0.1.1 (tusharchou#58)

8a57bc8

0.1.1 BigQuery Release (tusharchou#46)

d1c0894

* Create publish.yml * Update publish.yml

pypi release through github actions (tusharchou#60)

72ab669

* Create publish.yml * Update publish.yml * Update publish.yml

Create manual.yml

0757da8

Update publish.yml (tusharchou#62)

cd2cbbe

Update publish.yml (tusharchou#63)

80c8286

Update pyproject.toml (tusharchou#64)

48efe67

Update publish.yml (tusharchou#65)

d1dbf62

* Update publish.yml * Update publish.yml * Update publish.yml

brmhastra and others added 17 commits October 30, 2024 21:35

GitHub workflow for pypi release (tusharchou#70)

840223e

* Update publish.yml * Update pyproject.toml

GitHub workflow for pypi release (tusharchou#71)

e39db41

* Update publish.yml * Update pyproject.toml * Update publish.yml

Update publish.yml (tusharchou#72)

8ab9414

Update publish.yml (tusharchou#73)

11143b4

Release v1.1 changes (tusharchou#74)

c9d1101

Github action setup for release v1.1 (tusharchou#76)

c3f2aa1

* Release v1.1 dist changes * Release v1.1 publish.yml changes * Release v1.1 publish.yml changes

Update publish.yml

bf783b4

Merge pull request tusharchou#77 from tusharchou/redpheonixx-patch-1

9cb0189

Update publish.yml

Merge pull request tusharchou#61 from tusharchou/tusharchou-patch-4

81eab88

Create manual.yml

Update publish.yml (tusharchou#78)

00f5009

Update publish.yml

28d5787

Merge pull request tusharchou#79 from tusharchou/redpheonixx-patch-3

5735908

Update publish.yml

Update publish.yml (tusharchou#80)

66b31b0

Update pyproject.toml (tusharchou#81)

6716cfd

Release v1.1 fix (tusharchou#84)

f8c994a

* Release v1.1 dist changes * Release v1.1 publish.yml changes * Release v1.1 publish.yml changes * Release v1.1

redpheonixx requested review from tusharchou, rakhioza07 and brmhastra November 8, 2024 07:39

redpheonixx changed the base branch from main to v0.1.1 November 8, 2024 07:45

dbt transformations commit in new branch

203aab2

redpheonixx force-pushed the main branch from c1414f1 to f8c994a Compare November 8, 2024 08:07

redpheonixx and others added 5 commits November 8, 2024 13:41

Merge branch 'main' of https://github.com/redpheonixx/local-data-plat…

d11c3dc

…form

Merge branch 'main' into 45-014-dbt-transformation-layer

722a303

Merge pull request #1 from redpheonixx/45-014-dbt-transformation-layer

62574fb

dbt transformations commit in new branch

Merge branch 'tusharchou:main' into main

911a62c

rakhioza07 requested changes Nov 11, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

0.1.2 warehousing using dbt, duckdb and iceberg #55

0.1.2 warehousing using dbt, duckdb and iceberg #55

Uh oh!

redpheonixx commented Oct 25, 2024 •

edited

Loading

Uh oh!

tusharchou Oct 28, 2024

Uh oh!

redpheonixx Oct 28, 2024 •

edited

Loading

Uh oh!

tusharchou commented Nov 8, 2024

Uh oh!

rakhioza07 left a comment

Uh oh!

rakhioza07 Nov 10, 2024

Uh oh!

rakhioza07 Nov 11, 2024

Uh oh!

rakhioza07 Nov 11, 2024

Uh oh!

rakhioza07 Nov 11, 2024

Uh oh!

rakhioza07 Nov 11, 2024

Uh oh!

rakhioza07 Nov 11, 2024

Uh oh!

Uh oh!

0.1.2 warehousing using dbt, duckdb and iceberg #55

Are you sure you want to change the base?

0.1.2 warehousing using dbt, duckdb and iceberg #55

Uh oh!

Conversation

redpheonixx commented Oct 25, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tusharchou Oct 28, 2024

Choose a reason for hiding this comment

Uh oh!

redpheonixx Oct 28, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tusharchou commented Nov 8, 2024

Best Practice Workflow:

Uh oh!

rakhioza07 left a comment

Choose a reason for hiding this comment

Uh oh!

rakhioza07 Nov 10, 2024

Choose a reason for hiding this comment

Uh oh!

rakhioza07 Nov 11, 2024

Choose a reason for hiding this comment

Uh oh!

rakhioza07 Nov 11, 2024

Choose a reason for hiding this comment

Uh oh!

rakhioza07 Nov 11, 2024

Choose a reason for hiding this comment

Uh oh!

rakhioza07 Nov 11, 2024

Choose a reason for hiding this comment

Uh oh!

rakhioza07 Nov 11, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

redpheonixx commented Oct 25, 2024 •

edited

Loading

redpheonixx Oct 28, 2024 •

edited

Loading