Skip to content

Commit ddb0182

Browse files
portovepportovep
authored
Add contract testing examples
* Add source contract tests * Fix source contract test Improved setup instructions * Upgrade to dbt 1.5 Upgrade dbt dependencies * Add model contract for mart model * Add support for Github Codespaces * Fix Github Codespaces name * Add initial CD pipeline definition * Add debug connection step to the pipeline * Inject postgres configuration via configuration variables and secrets * Change CI config to run unit and component tests * Run tests against test postgres database * Add pipeline job to deploy to test environment * Fix deploy to test job to deploy with any branch * Add missing install dependencies step in deployment job * Extract deployment job into a reusable workflow * Associate specific Github environment to the deployment * Split deployment workflow into multiple jobs
1 parent 529a662 commit ddb0182

15 files changed

+242
-63
lines changed

‎.devcontainer/devcontainer.json‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
{
2+
"name": "Python 3",
3+
"dockerComposeFile": "../docker-compose.yml",
4+
"service": "devcontainer",
5+
"workspaceFolder": "/workspaces/${localWorkspaceFolderBasename}",
6+
"postCreateCommand": "pip3 install --user -r requirements.txt",
7+
"customizations": {
8+
"vscode": {
9+
"settings": {
10+
"terminal.integrated.defaultProfile.linux": "zsh"
11+
},
12+
"extensions": [
13+
"GitHub.codespaces"
14+
]
15+
}
16+
}
17+
}

‎.github/workflows/cd-pipeline.yml‎

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
name: CD Pipeline
2+
3+
on:
4+
push:
5+
6+
jobs:
7+
test:
8+
name: Test
9+
runs-on: ubuntu-latest
10+
container:
11+
image: ghcr.io/dbt-labs/dbt-postgres:1.6.3
12+
env:
13+
POSTGRES_HOST: ${{ vars.POSTGRES_HOST }}
14+
POSTGRES_USER: ${{ vars.POSTGRES_USER }}
15+
POSTGRES_PASSWORD: ${{ secrets.POSTGRES_PASSWORD }}
16+
POSTGRES_DB: ${{ vars.POSTGRES_DB }}
17+
steps:
18+
- name: Checkout code
19+
uses: actions/checkout@v3
20+
- name: Install dependencies
21+
run: |
22+
dbt deps
23+
- name: Run unit tests
24+
run: |
25+
dbt test --target test --select tag:unit-test
26+
- name: Run component tests
27+
run: |
28+
dbt test --target test --select tag:unit-test
29+
deploy-test:
30+
name: Deploy to test
31+
needs: [test]
32+
uses: ./.github/workflows/deploy.yml
33+
with:
34+
environment-name: test
35+
secrets: inherit

‎.github/workflows/deploy.yml‎

Lines changed: 66 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,66 @@
1+
name: Deploy
2+
3+
on:
4+
workflow_call:
5+
inputs:
6+
environment-name:
7+
required: true
8+
type: string
9+
10+
env:
11+
POSTGRES_HOST: ${{ vars.POSTGRES_HOST }}
12+
POSTGRES_USER: ${{ vars.POSTGRES_USER }}
13+
POSTGRES_PASSWORD: ${{ secrets.POSTGRES_PASSWORD }}
14+
POSTGRES_DB: ${{ vars.POSTGRES_DB }}
15+
16+
jobs:
17+
seed-source-tables:
18+
name: Seed source tables
19+
runs-on: ubuntu-latest
20+
environment: ${{ inputs.environment-name }}
21+
container:
22+
image: ghcr.io/dbt-labs/dbt-postgres:1.6.3
23+
steps:
24+
- name: Checkout code
25+
uses: actions/checkout@v3
26+
- name: Install dependencies
27+
run: |
28+
dbt deps
29+
- name: Run seeds
30+
run: |
31+
dbt seed --target ${{ inputs.environment-name }}
32+
source-contract-tests:
33+
name: Run source contract tests
34+
needs: [seed-source-tables]
35+
runs-on: ubuntu-latest
36+
environment: ${{ inputs.environment-name }}
37+
container:
38+
image: ghcr.io/dbt-labs/dbt-postgres:1.6.3
39+
steps:
40+
- name: Checkout code
41+
uses: actions/checkout@v3
42+
- name: Install dependencies
43+
run: |
44+
dbt deps
45+
- name: Run seeds
46+
run: |
47+
dbt seed --target ${{ inputs.environment-name }}\
48+
- name: Run source contract tests
49+
run: |
50+
dbt test --target ${{ inputs.environment-name }} --select tag:contract-test-source
51+
deploy:
52+
name: Deploy
53+
needs: [source-contract-tests]
54+
runs-on: ubuntu-latest
55+
environment: ${{ inputs.environment-name }}
56+
container:
57+
image: ghcr.io/dbt-labs/dbt-postgres:1.6.3
58+
steps:
59+
- name: Checkout code
60+
uses: actions/checkout@v3
61+
- name: Install dependencies
62+
run: |
63+
dbt deps
64+
- name: Run data transformations
65+
run: |
66+
dbt run --target ${{ inputs.environment-name }}

‎.gitignore‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,4 +3,4 @@ target/
33
dbt_packages/
44
logs/
55
dbt
6-
profiles.yml
6+
.env

‎README.md‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -68,6 +68,12 @@ Check that everything works
6868
dbt debug
6969
```
7070

71+
Seed the database
72+
73+
```
74+
dbt seed
75+
```
76+
7177
## Running the tests
7278

7379
All tests
@@ -88,6 +94,12 @@ Component tests
8894
dbt test --select tag:component
8995
```
9096

97+
Contract tests
98+
99+
```
100+
dbt test --select tag:contract-test-source
101+
```
102+
91103
## Running data quality tests
92104

93105
```

‎dbt_project.yml‎

Lines changed: 2 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,13 +1,12 @@
1-
# Name your project! Project names should contain only lowercase characters
2-
# and underscores. A good package name should reflect your organization's
3-
# name or the intended use of these models
41
name: "dbt_testing_example"
52
version: "1.0.0"
63
config-version: 2
74

85
# This setting configures which "profile" dbt uses for this project.
96
profile: "dbt_testing_example"
107

8+
require-dbt-version: ">=1.5.0"
9+
1110
# These configurations specify where dbt should look for different types of files.
1211
# The `model-paths` config, for example, states that models in this project can be
1312
# found in the "models/" directory. You probably won't need to change these!

‎docker-compose.yml‎

Lines changed: 9 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,16 +1,20 @@
1-
version: '3.1'
1+
version: '3.8'
22

33
services:
4+
devcontainer:
5+
image: mcr.microsoft.com/devcontainers/python:0-3.10-bullseye
6+
volumes:
7+
- .:/workspaces:cached
8+
network_mode: service:db
9+
command: sleep infinity
10+
411
db:
512
image: postgres
613
restart: always
7-
ports:
8-
- 5432:5432
914
environment:
1015
POSTGRES_PASSWORD: example
1116

1217
adminer:
1318
image: adminer
1419
restart: always
15-
ports:
16-
- 8080:8080
20+
network_mode: service:db

‎models/marts/_body_mass_indexes__models.yml‎

Lines changed: 17 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,14 +2,30 @@ version: 2
22

33
models:
44
- name: body_mass_indexes
5+
config:
6+
materialized: table
7+
contract:
8+
enforced: true
59
columns:
610
- name: user_id
711
tags: ["data-quality"]
12+
data_type: int
13+
constraints:
14+
- type: not_null
15+
- type: check
16+
expression: "user_id > 0"
817
tests:
9-
- not_null
1018
- relationships:
1119
to: source('gym_app', 'raw_height')
1220
field: user_id
1321
- relationships:
1422
to: source('gym_app', 'raw_weight')
1523
field: user_id
24+
- name: created_date
25+
data_type: date
26+
- name: weight
27+
data_type: float
28+
- name: height
29+
data_type: float
30+
- name: bmi
31+
data_type: decimal

‎models/staging/gym_app/_gym_app__raw_height_schema.yml‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
version: 2
2+
3+
sources:
4+
- name: gym_app
5+
schema: dbt_testing_example
6+
tables:
7+
- name: raw_height
8+
tags: ["data-quality", "contract-test-source"]
9+
columns:
10+
- name: height_unit
11+
tests:
12+
- dbt_expectations.expect_column_values_to_be_of_type:
13+
column_type: text
14+
- accepted_values:
15+
values: ["cm", "inches"]
16+
- name: user_id
17+
tests:
18+
- not_null
19+
- dbt_expectations.expect_column_values_to_be_of_type:
20+
column_type: integer
21+
- name: height
22+
tests:
23+
- not_null
24+
- dbt_expectations.expect_column_values_to_be_of_type:
25+
column_type: double precision
26+
- dbt_utils.accepted_range:
27+
min_value: 0
28+
inclusive: false

‎models/staging/gym_app/_gym_app__raw_weight_schema.yml‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
version: 2
2+
3+
sources:
4+
- name: gym_app
5+
schema: dbt_testing_example
6+
tables:
7+
- name: raw_weight
8+
tags: ["data-quality", "contract-test-source"]
9+
columns:
10+
- name: measurement_unit
11+
tests:
12+
- dbt_expectations.expect_column_values_to_be_of_type:
13+
column_type: text
14+
- accepted_values:
15+
values: ["kg", "pounds"]
16+
- name: user_id
17+
tests:
18+
- not_null
19+
- dbt_expectations.expect_column_values_to_be_of_type:
20+
column_type: integer
21+
- name: weight
22+
tests:
23+
- not_null
24+
- dbt_expectations.expect_column_values_to_be_of_type:
25+
column_type: double precision
26+
- dbt_utils.accepted_range:
27+
min_value: 0
28+
inclusive: false

0 commit comments

Comments
 (0)