Assess semantic readiness

[matt-winkler]

Describe the feature

Enable additional checks from the project evaluator, geared towards assessing a project's preparedness for implementing a semantic layer. This would enable organizations to:

1. Assess how much work will go into creating clear / meaningful / delineated semantic models on top of their existing dbt DAG to avoid garbage-in garbage-out.
2. Provide metadata for LLMs to scaffold (more of) the semantic layer configurations themselves.

Did some hacking here before realizing the project evaluator is a better fit for this.

Here is the set of checks (and weightings, though these may be better weighted specific for each organization) I thought of:

1. Temporal Consistency (30%) - % of fact models with clear, consistent temporal columns
2. Column Documentation (20%) - % of columns with meaningful descriptions (already handled in project evaluator)
3. Relationship Documentation (20%) - % of mart models with at least one relationship test (I think project evaluator just checks for primary key tests at the moment)
4. Ownership Metadata (15%) - % of mart models with owner metadata defined
5. Source Freshness Configuration (15%) - % of sources with freshness checks configured (already handled in project evaluator)

Describe alternatives you've considered

Something scripted or LLM-based on top of the manifest.

The project evaluator is a superior approach for a few reasons:

• Checks can be evaluated using native dbt tooling
• They're easy to implement in CI
• Existing support for Fusion - generated manifests

Additional context

Could be implemented on all dbt projects. Mainly relevant for those adapters which are compatible with the semantic layer itself.

Who will this benefit?

Anyone

Are you interested in contributing this feature?

Yes

[b-per]

Column Documentation (20%) - % of columns with meaningful descriptions (already handled in project evaluator)

One of the key limitations of dbt-project-evaluator is that we can check what percentage of columns in the YAML have documentation, but we CAN'T check what percentage of total columns (listed in YAML and not listed in YAML) are described.

[matt-winkler]

Ahh - very good callout thanks @b-per.