Skip to content

Commit 07ae2ef

Browse files
AidanHarveyNelsonAidanHarveyNelson
authored
Merge pull request #1 from AidanHarveyNelson/feature/initial-commit
Feature/initial commit
2 parents 32fbf0d + e851f86 commit 07ae2ef

40 files changed

Lines changed: 3355 additions & 0 deletions

‎.github/CODEOWNERS‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
* @AidanHarveyNelson

‎.github/ISSUE_TEMPLATE/bug_report.md‎

Lines changed: 56 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,56 @@
1+
---
2+
name: Bug report
3+
about: Report a bug or an issue you've found with this package
4+
title: ''
5+
labels: bug, triage
6+
assignees: ''
7+
8+
---
9+
10+
### Describe the bug
11+
<!---
12+
A clear and concise description of what the bug is. You can also use the issue title to do this
13+
--->
14+
15+
### Steps to reproduce
16+
<!---
17+
In as much detail as possible, please provide steps to reproduce the issue. Sample data that triggers the issue, example model code, etc is all very helpful here.
18+
--->
19+
20+
### Expected results
21+
<!---
22+
A clear and concise description of what you expected to happen.
23+
--->
24+
25+
### Actual results
26+
<!---
27+
A clear and concise description of what you expected to happen.
28+
--->
29+
30+
### Screenshots and log output
31+
<!---
32+
If applicable, add screenshots or log output to help explain your problem.
33+
--->
34+
35+
### System information
36+
**The contents of your `packages.yml` file:**
37+
38+
**Which database are you using dbt with?**
39+
- [ ] redshift
40+
- [ ] snowflake
41+
- [ ] other (specify: ____________)
42+
43+
44+
**The output of `dbt --version`:**
45+
```
46+
<output goes here>
47+
```
48+
49+
**The operating system you're using:**
50+
51+
**The output of `python --version`:**
52+
53+
### Additional context
54+
<!---
55+
Add any other context about the problem here.
56+
--->

‎.github/ISSUE_TEMPLATE/feature_request.md‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
---
2+
name: Feature request
3+
about: Suggest an idea for this package
4+
title: ''
5+
labels: enhancement, triage
6+
assignees: ''
7+
8+
---
9+
10+
### Describe the feature
11+
A clear and concise description of what you want to happen.
12+
13+
### Describe alternatives you've considered
14+
A clear and concise description of any alternative solutions or features you've considered.
15+
16+
### Additional context
17+
Is this feature database-specific? Which database(s) is/are relevant? Please include any other relevant context here.
18+
19+
### Who will this benefit?
20+
What kind of use case will this feature be useful for? Please be specific and provide examples, this will help us prioritize properly.

‎.github/pull_request_template.md‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
## Description & motivation
2+
<!---
3+
Describe your changes, and why you're making them.
4+
-->
5+
6+
## Checklist
7+
- [ ] I have verified that these changes work locally
8+
- [ ] I have updated the README.md (if applicable)
9+
- [ ] I have added an integration test for my fix/feature (if applicable)

‎.github/workflows/ci.yml‎

Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,53 @@
1+
# **what?**
2+
# Run tests for dbt-external-tables against supported adapters
3+
4+
# **why?**
5+
# To ensure that dbt-external-tables works as expected with all supported adapters
6+
7+
# **when?**
8+
# On every PR, and every push to main and when manually triggered
9+
10+
name: Package Integration Tests
11+
12+
on:
13+
push:
14+
branches:
15+
- main
16+
pull_request_target:
17+
workflow_dispatch:
18+
19+
jobs:
20+
run-tests:
21+
uses: dbt-labs/dbt-package-testing/.github/workflows/run_tox.yml@v1
22+
with:
23+
# redshift
24+
REDSHIFT_HOST: ${{ vars.REDSHIFT_HOST }}
25+
REDSHIFT_USER: ${{ vars.REDSHIFT_USER }}
26+
REDSHIFT_PORT: ${{ vars.REDSHIFT_PORT }}
27+
REDSHIFT_DATABASE: ${{ vars.REDSHIFT_DATABASE }}
28+
REDSHIFT_SCHEMA: "integration_tests_redshift_${{ github.run_number }}"
29+
# snowflake
30+
SNOWFLAKE_USER: ${{ vars.SNOWFLAKE_USER }}
31+
SNOWFLAKE_WAREHOUSE: ${{ vars.SNOWFLAKE_WAREHOUSE }}
32+
SNOWFLAKE_ROLE: ${{ vars.SNOWFLAKE_ROLE }}
33+
SNOWFLAKE_DATABASE: ${{ vars.SNOWFLAKE_DATABASE }}
34+
SNOWFLAKE_SCHEMA: "integration_tests_snowflake_${{ github.run_number }}"
35+
# bigquery
36+
BIGQUERY_PROJECT: ${{ vars.BIGQUERY_PROJECT }}
37+
BIGQUERY_SCHEMA: "integration_tests_bigquery_${{ github.run_number }}"
38+
# synapse
39+
# temporarily removed until we can get the cluster hooked up to the blob correctly
40+
# SYNAPSE_DRIVER: ${{ vars.SYNAPSE_DRIVER }}
41+
# SYNAPSE_HOST: ${{ vars.SYNAPSE_HOST }}
42+
# SYNAPSE_PORT: ${{ vars.SYNAPSE_PORT }}
43+
# SYNAPSE_DATABASE: ${{ vars.SYNAPSE_DATABASE }}
44+
# SYNAPSE_AUTHENTICATION: ${{ vars.SYNAPSE_AUTHENTICATION }}
45+
# SYNAPSE_TENANT_ID: ${{ vars.SYNAPSE_TENANT_ID }}
46+
# SYNAPSE_CLIENT_ID: ${{ vars.SYNAPSE_CLIENT_ID }}
47+
48+
secrets:
49+
DBT_ENV_SECRET_REDSHIFT_PASS: ${{ secrets.DBT_ENV_SECRET_REDSHIFT_PASS }}
50+
SNOWFLAKE_ACCOUNT: ${{ secrets.SNOWFLAKE_ACCOUNT }}
51+
DBT_ENV_SECRET_SNOWFLAKE_PASS: ${{ secrets.DBT_ENV_SECRET_SNOWFLAKE_PASS }}
52+
BIGQUERY_KEYFILE_JSON: ${{ secrets.BIGQUERY_KEYFILE_JSON }}
53+
DBT_ENV_SECRET_SYNAPSE_CLIENT_SECRET: ${{ secrets.DBT_ENV_SECRET_SYNAPSE_CLIENT_SECRET }}

‎.github/workflows/integration_tests.yml‎

Lines changed: 64 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,64 @@
1+
name: Integration Testing
2+
3+
on:
4+
push:
5+
branches: [ "main" ]
6+
pull_request:
7+
branches: [ "main" ]
8+
9+
jobs:
10+
build:
11+
12+
runs-on: ubuntu-latest
13+
strategy:
14+
fail-fast: true
15+
max-parallel: 3
16+
matrix:
17+
python-version: [ "3.11"] # "3.10", "3.12"]
18+
dbt-version: ["1.8.0"] # "1.6.0", , "1.8.0b1"]
19+
data-platform: ["redshift", "snowflake", "bigquery"]
20+
21+
steps:
22+
- uses: actions/checkout@v3
23+
- name: Set up Python ${{ matrix.python-version }}
24+
uses: actions/setup-python@v3
25+
with:
26+
python-version: ${{ matrix.python-version }}
27+
- name: Install dbt-${{ matrix.data-platform }}~=${{ matrix.dbt-version }}
28+
run: |
29+
python -m pip install --upgrade pip
30+
python -m pip install "dbt-${{ matrix.data-platform }}~=${{ matrix.dbt-version }}" "dbt-core~=${{ matrix.dbt-version }}"
31+
- name: run integration_tests project on ${{ matrix.data-platform }}
32+
run: |
33+
cd integration_tests
34+
export DBT_PROFILES_DIR=.
35+
dbt deps --target ${{ matrix.data-platform }}
36+
dbt seed --full-refresh --target ${{ matrix.data-platform }}
37+
dbt run --target ${{ matrix.data-platform }}
38+
dbt run-operation prep_external --target ${{ matrix.data-platform }}
39+
dbt -d run-operation dbt_external_tables.stage_external_sources --vars 'ext_full_refresh: true' --target ${{ matrix.data-platform }}
40+
dbt run-operation dbt_external_tables.stage_external_sources --target ${{ matrix.data-platform }}
41+
dbt test --target ${{ matrix.data-platform }}
42+
env:
43+
44+
env:
45+
# redshift
46+
REDSHIFT_HOST: ${{ secrets.REDSHIFT_HOST }}
47+
REDSHIFT_USER: ${{ secrets.REDSHIFT_USER }}
48+
DBT_ENV_SECRET_REDSHIFT_PASS: ${{ secrets.DBT_ENV_SECRET_REDSHIFT_PASS }}
49+
REDSHIFT_PORT: ${{ secrets.REDSHIFT_PORT }}
50+
REDSHIFT_DATABASE: ${{ secrets.REDSHIFT_DATABASE }}
51+
REDSHIFT_SPECTRUM_IAM_ROLE: ${{ secrets.REDSHIFT_SPECTRUM_IAM_ROLE }}
52+
REDSHIFT_SCHEMA: "dbt_external_tables_integration_tests_redshift"
53+
#snowflake
54+
SNOWFLAKE_ACCOUNT: ${{ secrets.SNOWFLAKE_ACCOUNT }}
55+
SNOWFLAKE_USER: ${{ secrets.SNOWFLAKE_USER }}
56+
DBT_ENV_SECRET_SNOWFLAKE_PASS: ${{ secrets.DBT_ENV_SECRET_SNOWFLAKE_PASS }}
57+
SNOWFLAKE_WAREHOUSE: ${{ secrets.SNOWFLAKE_WAREHOUSE }}
58+
SNOWFLAKE_ROLE: ${{ secrets.SNOWFLAKE_ROLE }}
59+
SNOWFLAKE_DATABASE: ${{ secrets.SNOWFLAKE_DATABASE }}
60+
SNOWFLAKE_SCHEMA: "dbt_external_tables_integration_tests_snowflake"
61+
# bigquery
62+
BIGQUERY_PROJECT: ${{ vars.BIGQUERY_PROJECT }}
63+
BIGQUERY_SCHEMA: "dbt_external_tables_integration_tests_bigquery"
64+
BIGQUERY_KEYFILE_JSON: ${{ secrets.BIGQUERY_KEYFILE_JSON }}

‎.github/workflows/stale.yml‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
# **what?**
2+
# For issues that have been open for awhile without activity, label
3+
# them as stale with a warning that they will be closed out. If
4+
# anyone comments to keep the issue open, it will automatically
5+
# remove the stale label and keep it open.
6+
7+
# Stale label rules:
8+
# awaiting_response, more_information_needed -> 90 days
9+
# good_first_issue, help_wanted -> 360 days (a year)
10+
# tech_debt -> 720 (2 years)
11+
# all else defaults -> 180 days (6 months)
12+
13+
# **why?**
14+
# To keep the repo in a clean state from issues that aren't relevant anymore
15+
16+
# **when?**
17+
# Once a day
18+
19+
name: "Close stale issues and PRs"
20+
on:
21+
schedule:
22+
- cron: "30 1 * * *"
23+
24+
permissions:
25+
issues: write
26+
pull-requests: write
27+
28+
jobs:
29+
stale:
30+
uses: dbt-labs/actions/.github/workflows/stale-bot-matrix.yml@main

‎.github/workflows/triage-labels.yml‎

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
# **what?**
2+
# When the maintenance team triages, we sometimes need more information from the issue creator. In
3+
# those cases we remove the `triage` label and add the `awaiting_response` label. Once we
4+
# recieve a response in the form of a comment, we want the `awaiting_response` label removed
5+
# in favor of the `triage` label so we are aware that the issue needs action.
6+
7+
# **why?**
8+
# To help with out team triage issue tracking
9+
10+
# **when?**
11+
# This will run when a comment is added to an issue and that issue has the `awaiting_response` label.
12+
13+
name: Update Triage Label
14+
15+
on: issue_comment
16+
17+
defaults:
18+
run:
19+
shell: bash
20+
21+
permissions:
22+
issues: write
23+
24+
jobs:
25+
triage_label:
26+
if: contains(github.event.issue.labels.*.name, 'awaiting_response')
27+
uses: dbt-labs/actions/.github/workflows/swap-labels.yml@main
28+
with:
29+
add_label: "triage"
30+
remove_label: "awaiting_response"
31+
secrets: inherit

‎.gitignore‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -205,3 +205,13 @@ cython_debug/
205205
marimo/_static/
206206
marimo/_lsp/
207207
__marimo__/
208+
209+
210+
**/target/
211+
**/dbt_modules/
212+
**/dbt_packages/
213+
**/logs/
214+
**/env/
215+
**/venv/
216+
**/test.env
217+
integration_tests/vars.env

‎.python-version‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
3.11

0 commit comments

Comments
 (0)