02 Architecture

Architecture Diagram

The accelerator uses a medallion architecture for both the batch and real-time workloads.

• Hot Path: For real-time/event-driven workloads.
• Cold Path: For batch workloads.

Hot Path

The medallion layers in the hot path consist of Kusto Query Language (KQL) database tables and materialized views. Streaming data first lands as raw Kusto tables in the bronze layer, is processed by update policies into silver tables, and is then served to the analytics gold layer via materialized views.

1. Event Streams ingest data from various sources, including organization-wide data and streaming data from Microsoft products and Fabric events, discovered through Real-Time Hub.
2. Event Streams filters and routes data to low latency destinations like KQL databases, where raw data forms the bronze layer.
3. Update policies on Kusto tables transform data from bronze tables to silver tables. The update policies are automatically triggered when new data is written to bronze table eliminating the need for special orchestration.
4. Data from silver Kusto tables are aggregated by materialized views to form the gold layer for analytics.
5. Real-Time Dashboards serve analytics using KQL queries.
6. Activator Alerts can be triggered for observability purposed from Event Streams, Real-Time Dashboards and Power BI Reports.

Cold Path

The medallion layers in the cold path can be configured as files, Lakehouse, or Data Warehouse in OneLake. To demonstrate Fabric's seamless integration between different storage types in OneLake, this accelerator uses files for the bronze layer, Lakehouse for the silver layer, and Data Warehouse for the gold layer. You can easily reconfigure the medallion layers in any combination of files, Lakehouses, and Data Warehouses.

1. The Data Factory pipelines ingest data from both cloud and on-premises sources into OneLake bronze layer. The on-premises sources need an OPDG.
2. Data lands in the bronze layer in OneLake as files, where possible in parquet format as-is, without any transformation.
3. The Spark notebooks then transform the raw data from the bronze layer. The curated data is then stored in silver layer of OneLake as Lakehouse tables. Here, the data is cleansed, flattened, and standardized while maintaining its grain. The bronze data can be transformed into one-to-one or one-to-many Lakehouse table(s).
4. The Data Warehouse stored procedures apply business rules to data from the Lakehouse tables in the silver layer. It lands the data as DW tables in the gold layer of OneLake. Here, typical activities include applying custom business rules, creating snapshots, merging data from multiple tables, and creating hub-spoke star schema. A Lakehouse table from the silver layer can be transformed into one-to-one, one-to-many, or many-to-one DW tables in the gold layer.
5. Semantic models built on the gold layer DW tables serve as the analytics layer. This analytics layer is sometimes referred as the diamond layer. Here the relationships between tables are established.
6. The orchestration of this fabric accelerator is underpinned by the ELT Framework, a metadata-driven orchestration tool that streamlines ingestion and transformation pipelines. The ELT framework uses a Fabric SQL database for metadata. Semantic models built from ELT metadata provide real-time reporting via direct lake Semantic Models.
7. Power BI serves as the analytics layer, supported by PBI Copilot for self-service capabilities.