Skip to content

Commit aab1108

Browse files
georgesittasgeorgesittas
authored
Merge branch 'main' into rough_making_model_node_optional
2 parents ac68e4c + 8efbe5f commit aab1108

File tree

93 files changed

+6023
-1574
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

93 files changed

+6023
-1574
lines changed

‎.circleci/continue_config.yml‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -152,7 +152,6 @@ jobs:
152152
- image: cimg/node:20.19.0
153153
resource_class: small
154154
steps:
155-
- halt_unless_client
156155
- checkout
157156
- restore_cache:
158157
name: Restore pnpm Package Cache

‎.github/dependabot.yml‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,3 +4,7 @@ updates:
44
directory: '/'
55
schedule:
66
interval: 'weekly'
7+
- package-ecosystem: 'github-actions'
8+
directory: '/'
9+
schedule:
10+
interval: 'weekly'

‎.github/workflows/pr.yaml‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
on:
2+
push:
3+
branches:
4+
- main
5+
pull_request:
6+
branches:
7+
- main
8+
concurrency:
9+
group: 'pr-${{ github.event.pull_request.number }}'
10+
cancel-in-progress: true
11+
jobs:
12+
test-vscode:
13+
env:
14+
PLAYWRIGHT_SKIP_BROWSER_DOWNLOAD: 1
15+
runs-on: ubuntu-latest
16+
steps:
17+
- uses: actions/checkout@v4
18+
- uses: actions/setup-node@v4
19+
with:
20+
node-version: '20'
21+
- uses: pnpm/action-setup@v4
22+
with:
23+
version: latest
24+
- name: Install dependencies
25+
run: pnpm install
26+
- name: Run CI
27+
run: pnpm run ci

‎docs/guides/configuration.md‎

Lines changed: 111 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -381,6 +381,117 @@ Example showing default values:
381381
)
382382
```
383383

384+
385+
### Always comparing against production
386+
387+
By default, SQLMesh compares the current state of project files to the target `<env>` environment when `sqlmesh plan <env>` is run. However, a common expectation is that local changes should always be compared to the production environment.
388+
389+
The `always_recreate_environment` boolean plan option can alter this behavior. When enabled, SQLMesh will always attempt to compare against the production environment by recreating the target environment; If `prod` does not exist, SQLMesh will fall back to comparing against the target environment.
390+
391+
**NOTE:**: Upon succesfull plan application, changes are still promoted to the target `<env>` environment.
392+
393+
=== "YAML"
394+
395+
```yaml linenums="1"
396+
plan:
397+
always_recreate_environment: True
398+
```
399+
400+
=== "Python"
401+
402+
```python linenums="1"
403+
from sqlmesh.core.config import (
404+
Config,
405+
ModelDefaultsConfig,
406+
PlanConfig,
407+
)
408+
409+
config = Config(
410+
model_defaults=ModelDefaultsConfig(dialect=<dialect>),
411+
plan=PlanConfig(
412+
always_recreate_environment=True,
413+
),
414+
)
415+
```
416+
417+
#### Change Categorization Example
418+
419+
Consider this scenario with `always_recreate_environment` enabled:
420+
421+
1. Initial state in `prod`:
422+
```sql
423+
MODEL (name sqlmesh_example.test_model, kind FULL);
424+
SELECT 1 AS col
425+
```
426+
427+
1. First (breaking) change in `dev`:
428+
```sql
429+
MODEL (name sqlmesh_example__dev.test_model, kind FULL);
430+
SELECT 2 AS col
431+
```
432+
433+
??? "Output plan example #1"
434+
435+
```bash
436+
New environment `dev` will be created from `prod`
437+
438+
Differences from the `prod` environment:
439+
440+
Models:
441+
└── Directly Modified:
442+
└── sqlmesh_example__dev.test_model
443+
444+
---
445+
+++
446+
447+
448+
kind FULL
449+
)
450+
SELECT
451+
- 1 AS col
452+
+ 2 AS col
453+
```
454+
455+
3. Second (metadata) change in `dev`:
456+
```sql
457+
MODEL (name sqlmesh_example__dev.test_model, kind FULL, owner 'John Doe');
458+
SELECT 5 AS col
459+
```
460+
461+
??? "Output plan example #2"
462+
463+
```bash
464+
New environment `dev` will be created from `prod`
465+
466+
Differences from the `prod` environment:
467+
468+
Models:
469+
└── Directly Modified:
470+
└── sqlmesh_example__dev.test_model
471+
472+
---
473+
474+
+++
475+
476+
@@ -1,8 +1,9 @@
477+
478+
MODEL (
479+
name sqlmesh_example.test_model,
480+
+ owner "John Doe",
481+
kind FULL
482+
)
483+
SELECT
484+
- 1 AS col
485+
+ 2 AS col
486+
487+
Directly Modified: sqlmesh_example__dev.test_model (Breaking)
488+
Models needing backfill:
489+
└── sqlmesh_example__dev.test_model: [full refresh]
490+
```
491+
492+
Even though the second change should have been a metadata change (thus not requiring a backfill), it will still be classified as a breaking change because the comparison is against production instead of the previous development state. This is intentional and may cause additional backfills as more changes are accumulated.
493+
494+
384495
### Gateways
385496
386497
The `gateways` configuration defines how SQLMesh should connect to the data warehouse, state backend, and scheduler. These options are in the [gateway](../reference/configuration.md#gateway) section of the configuration reference page.

‎docs/guides/tablediff.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -253,12 +253,12 @@ Then, specify each table's gateway in the `table_diff` command with this syntax:
253253
For example, we could diff the `landing.table` table across `bigquery` and `snowflake` gateways like this:
254254

255255
```sh
256-
$ sqlmesh table_diff 'bigquery|landing.table:snowflake|lake.table'
256+
$ tcloud sqlmesh table_diff 'bigquery|landing.table:snowflake|lake.table'
257257
```
258258

259259
This syntax tells SQLMesh to use the cross-database diffing algorithm instead of the normal within-database diffing algorithm.
260260

261-
After adding gateways to the table names, use `table_diff` as described above - the same options apply for specifying the join keys, decimal precision, etc. See `sqlmesh table_diff --help` for a [full list of options](../reference/cli.md#table_diff).
261+
After adding gateways to the table names, use `table_diff` as described above - the same options apply for specifying the join keys, decimal precision, etc. See `tcloud sqlmesh table_diff --help` for a [full list of options](../reference/cli.md#table_diff).
262262

263263
!!! warning
264264

@@ -273,7 +273,7 @@ A cross-database diff is broken up into two stages.
273273
The first stage is a schema diff. This example shows that differences in column name case across the two tables are identified as schema differences:
274274

275275
```bash
276-
$ sqlmesh table_diff 'bigquery|sqlmesh_example.full_model:snowflake|sqlmesh_example.full_model' --on item_id --show-sample
276+
$ tcloud sqlmesh table_diff 'bigquery|sqlmesh_example.full_model:snowflake|sqlmesh_example.full_model' --on item_id --show-sample
277277

278278
Schema Diff Between 'BIGQUERY|SQLMESH_EXAMPLE.FULL_MODEL' and 'SNOWFLAKE|SQLMESH_EXAMPLE.FULL_MODEL':
279279
├── Added Columns:

‎docs/reference/cli.md‎

Lines changed: 8 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -343,6 +343,8 @@ Options:
343343
Default: prod.
344344
--skip-tests Skip tests prior to generating the plan if
345345
they are defined.
346+
--skip-linter Skip linting prior to generating the plan if
347+
the linter is enabled.
346348
-r, --restate-model TEXT Restate data for specified models and models
347349
downstream from the one specified. For
348350
production environment, all related model
@@ -383,9 +385,12 @@ Options:
383385
application (prod environment only).
384386
--enable-preview Enable preview for forward-only models when
385387
targeting a development environment.
386-
--diff-rendered Output text differences for rendered versions
387-
of models and standalone audits
388-
-v, --verbose Verbose output.
388+
--diff-rendered Output text differences for the rendered
389+
versions of the models and standalone
390+
audits.
391+
--explain Explain the plan instead of applying it.
392+
-v, --verbose Verbose output. Use -vv for very verbose
393+
output.
389394
--help Show this message and exit.
390395
```
391396

‎docs/reference/configuration.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -80,7 +80,7 @@ Configuration for the `sqlmesh plan` command.
8080
| `enable_preview` | Indicates whether to enable [data preview](../concepts/plans.md#data-preview) for forward-only models when targeting a development environment (Default: True, except for dbt projects where the target engine does not support cloning) | Boolean | N |
8181
| `no_diff` | Don't show diffs for changed models (Default: False) | boolean | N |
8282
| `no_prompts` | Disables interactive prompts in CLI (Default: True) | boolean | N |
83-
83+
| `always_recreate_environment` | Always recreates the target environment from the environment specified in `create_from` (by default `prod`) (Default: False) | boolean | N |
8484
## Run
8585

8686
Configuration for the `sqlmesh run` command. Please note that this is only applicable when configured with the [builtin](#builtin) scheduler.

‎examples/sushi/helper.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22
import typing as t
33
from datetime import datetime, timedelta
44

5-
import numpy as np
5+
import numpy as np # noqa: TID253
66

77

88
def set_seed(dt: datetime) -> None:

‎examples/sushi/models/items.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22
import typing as t
33
from datetime import datetime
44

5-
import numpy as np
5+
import numpy as np # noqa: TID253
66
import pandas as pd # noqa: TID253
77
from helper import iter_dates # type: ignore
88
from sqlglot.expressions import to_column

‎examples/sushi/models/order_items.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22
import typing as t
33
from datetime import datetime
44

5-
import numpy as np
5+
import numpy as np # noqa: TID253
66
import pandas as pd # noqa: TID253
77
from helper import iter_dates # type: ignore
88
from sqlglot import exp

0 commit comments

Comments
 (0)