Initial slate of deprecations for 1.10 (#11544)

* Begin basic jsonschema validations of dbt_project.yml (#11505)

* Add jsonschema for validation project file

* Add utility for helping to load jsonschema resources

Currently things are a bit hard coded. We should probably alter this
to be a bit more flexible.

* Begin validating the the `dbt_project.yml` via jsonschema

* Begin emitting deprecation warnings for generic jsonschema violations in dbt_project.yml

* Move from `DbtInternalError` to `DbtRuntimeError` to avoid circular imports

* Add tests for basic jsonschema validation of `dbt_project.yml`

* Add changie doc

* Add seralization test for new deprecation events

* Alter the project jsonschema to not require things that are optional

* Add datafiles to package egg

* Update inclusion of project jsonschema in setup.py to get files correctly

Using the glob spec returns a list of found files. Our previous spec was
raising the error
`error: can't copy 'dbt/resources/input_schemas/project/*.json': doesn't exist or not a regular file`

* Try another approach of adding jsonschema to egg

* Add input_schemas dir to MANIFEST.in spec

* Drop jsonschema inclusion spec from setup.py

* Begin using importlib.resources.files for loading project jsonschema

This doesn't currently work with editable installs :'(

* Use relative paths for loading jsonchemas instead of importlib

Using "importlib" is the blessed way to do this sort of thing. However,
that is failing for us on editable installs. This commit switches us
to using relative paths. Technically doing this has edge cases, however
this is also what we do for the `start_project` used in `dbt init`. So
we're going to do the same, for now. We should revisit this soon.

* Drop requirment of `__additional_properties__` specified by project jsonschema

* Drop requirement for `pre-hook` and `post-hook` specified by project jsonschema

* Reset `active_deprecations` global at the end of tests using `project` fixture

* Begin validation the jsonschema of YAML resource files (#11516)

* Add jsonschema for resources

* Begin jsonschema validating YAML resource files in dbt projects

* Drop `tests` and `data_tests` as required properties of `Columns` and `Models` for resources jsonschema

* Drop `__additional_properties__` as required for `_Metrics` in resources jsonschema

* Drop `post_hook` and `pre_hook` requirement for `__SnapshotsConfig` in resources jsonschema

* Update `_error_path_to_string` to handle empty paths

* Create + use custom Draft7Validator to ignore datetime and date classes

* Break `TestRetry` functional test class into multiple test classes

There was some overflow global state from one test to another which was
causing some of the tests to break.

* Refactor duplicate instances of `jsonschema_validate` to single definition

* Begin testing jsonschema validation of resource YAMLs

* Add changie doc

* Add Deprecation Warnings for Unexpected Jinja Blocks (#11514)

* Add deprecation warnings on unexpected jinja blocks.

* Add changelog entry.

* Add test event.

* Regen proto types.

* Fix event test.

* Add `UnexpectedJinjaBlockDeprecationSummary` and add file context to `UnexpectedJinjaBlockDeprecation` (#11517)

* Add summary event for UnexpectedJinjaBlockDeprecation

* Begin including file information in UnexpectedJinjaBlockDeprecation event

* Add UnexpectedJinjaBlockDeprecationSummary to test_events.py

* Deprecate Custom Top-Level Keys (#11518)

* Add specific deprecation for custom top level keys.

* Add changelog entry

* Add test events

* Add Check for Duplicate YAML Keys (#11510)

* Add functionality to check for duplicate yaml keys, working around PyYAML limitation.

* Fix up some ancient typing issues.

* Ignore typing issue, for now.

* Correct unit tests of `checked_load`

* Add event and deprecation types for duplicate yaml keys

* Begin validating `dbt_project.yml` for duplicate key violations

* Begin checking for duplicate key violations in schema files

* Add test to check duplicate keys are checked in schema files

* Refactor checked_yaml failure handling to reduce duplicate code

* Move `checked_load` utilities to separate file to avoid circular imports

* Handle yaml `start_mark` correctly for top level key errors

* Update changelog

* Fix test.

---------

Co-authored-by: Quigley Malcolm <[email protected]>

* Fix issue with YAML anchors in new CheckedLoader class.

* Deprecate having custom keys in config blocks (#11522)

* Add deprecation event for custom keys found in configs

* Begin checking schema files for custom keys found in configs

* Test new CustomConfigInConfigDeprecation event

* Add changie doc

* Add custom config key deprecation events to event serialization test

* Provide message to ValidationError in `SelectorConfig.from_path`

This typing error is unrelated to the changes in this PR. However,
it was failing CI, so I figured it'd be simple to just fix it.

* Add some extra guards around the ValidationFailure `path` and `instance`

* [TIDY-FRIST] Use new `deprecation_tag` (#11524)

* Tidy First: Update deprecation events to use the new `deprecation_tag`

Note did this for a majority of deprecations, but not _all_ deprecations.
That is because not all deprecations were following the pattern. As some
people do string parsing of our logs with regex, altering the deprecations
that weren't doing what `deprecation_tag` does to use `deprecation_tag`
would be a _breaking change_ for those events, thus we did not alter those
events

* Bump minimum dbt-common to `1.22.0`

* Fix tests

* Begin emitting deprecation events for custom properties found in objects (#11526)

* Fix CustomKeyInConfigDeprecationSummary

* Add deprecation type for custom properties in YAML objects

* Begin emitting deprecation events for custom properties found in objects

* Add changie doc

* Add `loaded_at_query` property to `_Sources` definition in jsonschema

This was breaking the test tests/unit/parser/test_parser.py::SchemaParserSourceTest::test_parse_source_custom_freshness_at_source

* Move validating jsonschema of schema files earlier in the process

Previously we were validating the jsonschema of schema files in
`SchemaParser.parse_file`. However, the file is originally loaded in
`yaml_from_file` (which happens before `SchemaParser.parse_file`), and
`yaml_from_file` _modifies_ the loaded dictionary to add some additional
properties. These additional properties violate the jsonschema unfortunately,
and thus we needed to start validating the schema against the jsonschema
before any such modifications.

* Skip parser tests for `model.freshness`

Model freshness never got fully implemented, won't be implemented nor
documented for 1.10. As such we're gonna consider the `model.freshness`
property an "unknown additional property". This is actually good as some
people have "accidentally" defined "freshness" on their models (likely due
to copy/paste of a source), and that property isn't doing anything.

* One single DeprecationsSummary event to rule them all (#11540)

* Begin emitting singular deprecations summary, instead of summary per deprecation type

* Remove concept of deprecation specific summary events in deprecations module

* Drop deprecation summary events that have been added to `feature-branch--11335-deprecations` but not `main`

These are save to drop with no notice because they only ever existed
on a feature branch, never main.

* Correct code numbers for new events on feature-branch that haven't made it to main yet

* Kill `PackageRedirectDeprecationSummary` event, and retire its event code

* add changie doc

* Update jsonschemas to versions 0.0.110 (#11541)

* Update jsonschems to 0.0.110

* Don't allow additional properties in configs

* Don't allow additional top level properties on objects

* Allow for 'loaded_at_query' on Sources and Tables

* Don't allow additional top level properties in schema files

---------

Co-authored-by: Peter Webb <[email protected]>

Author: QMalcolm

.changes/unreleased/Features-20250414-171209.yaml

@@ -0,0 +1,6 @@
+kind: Features
+body: Basic jsonschema validation of `dbt_project.yml`
+time: 2025-04-14T17:12:09.351572-07:00
+custom:
+  Author: QMalcolm
+  Issue: "11503"

.changes/unreleased/Features-20250415-133751.yaml

@@ -0,0 +1,6 @@
+kind: Freatures
+body: Begin checking YAML files for duplicate keys
+time: 2025-04-15T13:37:51.878256-04:00
+custom:
+    Author: peterallenwebb QMalcolm
+    Issue: "11296"

.changes/unreleased/Features-20250416-144427.yaml

@@ -0,0 +1,6 @@
+kind: Features
+body: Add deprecation warnings for unexpected blocks in jinja.
+time: 2025-04-16T14:44:27.136199-04:00
+custom:
+    Author: peterallenwebb
+    Issue: "11393"

.changes/unreleased/Features-20250416-151711.yaml

@@ -0,0 +1,6 @@
+kind: Features
+body: Begin validating the jsonschema of resource YAML files
+time: 2025-04-16T15:17:11.760509-07:00
+custom:
+  Author: QMalcolm
+  Issue: "11504"

.changes/unreleased/Features-20250417-001347.yaml

@@ -0,0 +1,6 @@
+kind: Features
+body: Add deprecation warning for custom top level keys in YAML files.
+time: 2025-04-17T00:13:47.744191-04:00
+custom:
+    Author: peterallenwebb
+    Issue: "11338"

.changes/unreleased/Features-20250417-114915.yaml

@@ -0,0 +1,6 @@
+kind: Features
+body: Begin emitting deprecationw warnings for custom keys in config blocks
+time: 2025-04-17T11:49:15.056242-07:00
+custom:
+  Author: QMalcolm
+  Issue: "11337"

.changes/unreleased/Features-20250418-104210.yaml

@@ -0,0 +1,6 @@
+kind: Features
+body: Begin emitting deprecation events for custom properties found in objects
+time: 2025-04-18T10:42:10.048839-07:00
+custom:
+  Author: QMalcolm
+  Issue: "11336"

.changes/unreleased/Features-20250424-153327.yaml

@@ -0,0 +1,6 @@
+kind: Features
+body: Create a singular deprecations summary event
+time: 2025-04-24T15:33:27.252763-05:00
+custom:
+  Author: QMalcolm
+  Issue: "11536"

core/MANIFEST.in

@@ -1,3 +1,3 @@
-recursive-include dbt/include *.py *.sql *.yml *.html *.md .gitkeep .gitignore
+recursive-include dbt/include *.py *.sql *.yml *.html *.md .gitkeep .gitignore *.json
 include dbt/py.typed
 recursive-include dbt/task/docs *.html

core/dbt/cli/requires.py

@@ -16,7 +16,7 @@
 from dbt.config.runtime import UnsetProfile, load_profile, load_project
 from dbt.context.providers import generate_runtime_macro_context
 from dbt.context.query_header import generate_query_header_context
-from dbt.deprecations import show_all_deprecation_summaries
+from dbt.deprecations import show_deprecations_summary
 from dbt.events.logging import setup_event_logger
 from dbt.events.types import (
     ArtifactUploadError,
@@ -181,7 +181,7 @@ def wrapper(*args, **kwargs):
             except Exception as e:
                 fire_event(ArtifactUploadError(msg=str(e)))
 
-            show_all_deprecation_summaries()
+            show_deprecations_summary()
 
             if importlib.util.find_spec("resource") is not None:
                 import resource
@@ -267,8 +267,10 @@ def wrapper(*args, **kwargs):
             raise DbtProjectError("profile required for project")
 
         flags = ctx.obj["flags"]
+        # TODO deprecations warnings fired from loading the project will lack
+        # the project_id in the snowplow event.
         project = load_project(
-            flags.PROJECT_DIR, flags.VERSION_CHECK, ctx.obj["profile"], flags.VARS
+            flags.PROJECT_DIR, flags.VERSION_CHECK, ctx.obj["profile"], flags.VARS, validate=True
         )
         ctx.obj["project"] = project