[Epic] Implement Workspace Concept for Multi-Project Architecture and dbt Mesh

[ismailsimsek]

This Epic tracks the implementation of a core Workspace concept to enable a robust multi-project structure. This feature is essential for supporting large-scale data platforms, adhering to dbt Mesh architecture, and enabling reliable cross-model referencing.

Goal

Introduce a hierarchical configuration structure that allows users to manage multiple, interconnected dbt projects under a single workspace, facilitating large-scale data modeling and scalable configuration management.

Key Features and Concepts

1. Workspace (Root Level)

• Acts as the primary entry point for managing a collection of dbt Projects.
• The configuration should be loaded from a designated root file (e.g., workspace.yml).
• Holds the root/global Config object, defining defaults and global resources.

2. Project (Sub-Level)

• Represents a single, independent dbt project operating within the context of the parent Workspace.
• Has its own local Config object (from a file like project.yml).
• Configuration Inheritance: A Project's configuration should automatically inherit or fall back to the Workspace's Config for any un-specified values (e.g., default database connections, common variables).
• Connection Overrides: Allow Projects to define and prioritize their own database connections, while still being able to reference or inherit global connections defined at the Workspace level.

3. Cross-Model Reference & dbt Mesh Support

• The architecture must allow models in one Project to easily and reliably reference models/artifacts in other Projects within the same Workspace.
• Ensure the design is flexible and scalable to support the principles of the dbt Mesh organizational pattern.

Initial Task

The first step for this Epic is to formalize the plan:

• Create Design Document: Define and create a comprehensive technical design document outlining:
  • The proposed file structure and naming conventions (workspace.yml, project.yml).
  • The configuration parsing logic and the inheritance/fallback mechanism.
  • A technical proposal for enabling cross-project model referencing.