address ci/cd issues and Dennis Feedback

Author: C00ldudeNoonan

examples/data-quality-patterns/README.md

@@ -6,15 +6,15 @@ A comprehensive Dagster project demonstrating data quality validation across all
 
 This project covers all 7 dimensions of data quality:
 
-| Dimension | Description | Implementation |
-|-----------|-------------|----------------|
-| **Accuracy** | Data correctly represents real-world entities | Python checks for name patterns, age ranges |
-| **Completeness** | All required data is present | Python + dbt `not_null` tests |
-| **Consistency** | Data is uniform across datasets | Python checks for region code formats |
-| **Timeliness** | Data is up-to-date and available | `FreshnessPolicy` on assets |
-| **Validity** | Data conforms to formats and rules | Python + Great Expectations checks |
-| **Uniqueness** | No unwanted duplicates | Python + dbt `unique` tests |
-| **Integrity** | Relationships between data maintained | Python + dbt `relationships` tests |
+| Dimension        | Description                                   | Implementation                              |
+| ---------------- | --------------------------------------------- | ------------------------------------------- |
+| **Accuracy**     | Data correctly represents real-world entities | Python checks for name patterns, age ranges |
+| **Completeness** | All required data is present                  | Python + dbt `not_null` tests               |
+| **Consistency**  | Data is uniform across datasets               | Python checks for region code formats       |
+| **Timeliness**   | Data is up-to-date and available              | `FreshnessPolicy` on assets                 |
+| **Validity**     | Data conforms to formats and rules            | Python + Great Expectations checks          |
+| **Uniqueness**   | No unwanted duplicates                        | Python + dbt `unique` tests                 |
+| **Integrity**    | Relationships between data maintained         | Python + dbt `relationships` tests          |
 
 ## Project Structure
 
@@ -94,6 +94,7 @@ Open http://localhost:3000 to view the Dagster UI.
 ### 4. Materialize Assets
 
 In the Dagster UI:
+
 1. Navigate to the **Assets** page
 2. Select `raw_customers`, `raw_orders`, and `raw_products`
 3. Click **Materialize** to generate data with quality issues
@@ -105,29 +106,29 @@ In the Dagster UI:
 
 Located in `src/data_quality_patterns/defs/assets/python_checks.py`:
 
-| Check | Dimension | Description |
-|-------|-----------|-------------|
-| `check_accuracy_names` | Accuracy | Detects placeholder/test names |
-| `check_accuracy_age` | Accuracy | Validates age ranges (0-120) |
-| `check_completeness_email` | Completeness | Requires 95% email coverage |
-| `check_completeness_amount` | Completeness | Requires 98% amount coverage |
-| `check_consistency_region` | Consistency | Validates region codes (US, EU, APAC, LATAM) |
-| `check_validity_email` | Validity | Validates email format |
-| `check_validity_amount` | Validity | Validates positive amounts |
-| `check_uniqueness_customer_id` | Uniqueness | Detects duplicate customers |
-| `check_uniqueness_order_id` | Uniqueness | Detects duplicate orders |
-| `check_integrity_customer_ref` | Integrity | Validates customer foreign keys |
+| Check                          | Dimension    | Description                                  |
+| ------------------------------ | ------------ | -------------------------------------------- |
+| `check_accuracy_names`         | Accuracy     | Detects placeholder/test names               |
+| `check_accuracy_age`           | Accuracy     | Validates age ranges (0-120)                 |
+| `check_completeness_email`     | Completeness | Requires 95% email coverage                  |
+| `check_completeness_amount`    | Completeness | Requires 98% amount coverage                 |
+| `check_consistency_region`     | Consistency  | Validates region codes (US, EU, APAC, LATAM) |
+| `check_validity_email`         | Validity     | Validates email format                       |
+| `check_validity_amount`        | Validity     | Validates positive amounts                   |
+| `check_uniqueness_customer_id` | Uniqueness   | Detects duplicate customers                  |
+| `check_uniqueness_order_id`    | Uniqueness   | Detects duplicate orders                     |
+| `check_integrity_customer_ref` | Integrity    | Validates customer foreign keys              |
 
 ### Great Expectations Checks
 
 Located in `src/data_quality_patterns/defs/assets/ge_checks.py`:
 
-| Check | Dimension | Description |
-|-------|-----------|-------------|
-| `ge_check_sku_unique` | Uniqueness | Product SKU uniqueness |
-| `ge_check_name_not_null` | Completeness | Product name presence |
-| `ge_check_price_positive` | Validity | Price range validation |
-| `ge_check_category_valid` | Consistency | Category value validation |
+| Check                     | Dimension    | Description               |
+| ------------------------- | ------------ | ------------------------- |
+| `ge_check_sku_unique`     | Uniqueness   | Product SKU uniqueness    |
+| `ge_check_name_not_null`  | Completeness | Product name presence     |
+| `ge_check_price_positive` | Validity     | Price range validation    |
+| `ge_check_category_valid` | Consistency  | Category value validation |
 
 ### dbt Tests as Asset Checks
 
@@ -141,6 +142,7 @@ Defined in `dbt_project/models/*/schema.yml`:
 ### Freshness Policies
 
 Raw data assets have `FreshnessPolicy` attached for timeliness monitoring:
+
 - Fail window: 24 hours
 - Warn window: 12 hours
 
@@ -156,6 +158,7 @@ df = generate_customers(n=100, failure_rate=0.15, seed=42)
 ```
 
 Quality issues introduced:
+
 - **Uniqueness**: Duplicate customer IDs
 - **Completeness**: Missing email addresses
 - **Validity**: Invalid email formats
@@ -175,6 +178,7 @@ pytest tests/test_asset_checks.py -v
 ```
 
 Tests verify that:
+
 - Quality checks correctly identify issues in random data
 - Clean data passes all checks
 - Validators work as expected
@@ -202,6 +206,7 @@ ruff check . --fix --isolated  # Auto-fix issues
 ### Project Configuration
 
 The project uses:
+
 - `pyproject.toml` for dependency management (uv/hatch)
 - `tool.dagster` for Dagster configuration
 - `tool.dg` for dg CLI configuration

examples/data-quality-patterns/src/data_quality_patterns/__init__.py

@@ -4,4 +4,3 @@
 covering native asset checks, Great Expectations, dbt tests, and freshness checks
 across all 7 data quality dimensions.
 """
-

examples/data-quality-patterns/src/data_quality_patterns/defs/__init__.py

@@ -1,2 +1 @@
 """Dagster definitions for data quality patterns."""
-

examples/data-quality-patterns/src/data_quality_patterns/defs/assets/__init__.py

@@ -1,2 +1 @@
 """Asset definitions for data quality patterns."""
-

examples/data-quality-patterns/src/data_quality_patterns/defs/assets/ge_checks.py

@@ -11,9 +11,9 @@
 
 import dagster as dg
 import pandas as pd
+from great_expectations.dataset import PandasDataset
 
 from data_quality_patterns.defs.assets.raw_data import raw_products
-from data_quality_patterns.lib.expectations import GE_AVAILABLE, create_ge_dataset
 
 
 @dg.asset_check(asset=raw_products, name="ge_check_sku_unique")
@@ -22,21 +22,7 @@ def ge_check_sku_unique(
     raw_products: pd.DataFrame,
 ) -> dg.AssetCheckResult:
     """Great Expectations check: SKU values are unique."""
-    if not GE_AVAILABLE:
-        return dg.AssetCheckResult(
-            passed=True,
-            severity=dg.AssetCheckSeverity.WARN,
-            description="Great Expectations not installed, skipping check",
-        )
-
-    ge_df = create_ge_dataset(raw_products)
-    if ge_df is None:
-        return dg.AssetCheckResult(
-            passed=False,
-            severity=dg.AssetCheckSeverity.WARN,
-            description="Failed to create Great Expectations dataset",
-        )
-
+    ge_df = PandasDataset(raw_products)
     result = ge_df.expect_column_values_to_be_unique("sku")
 
     if result["success"]:
@@ -62,21 +48,7 @@ def ge_check_name_not_null(
     raw_products: pd.DataFrame,
 ) -> dg.AssetCheckResult:
     """Great Expectations check: Product names are not null (completeness)."""
-    if not GE_AVAILABLE:
-        return dg.AssetCheckResult(
-            passed=True,
-            severity=dg.AssetCheckSeverity.WARN,
-            description="Great Expectations not installed, skipping check",
-        )
-
-    ge_df = create_ge_dataset(raw_products)
-    if ge_df is None:
-        return dg.AssetCheckResult(
-            passed=False,
-            severity=dg.AssetCheckSeverity.WARN,
-            description="Failed to create Great Expectations dataset",
-        )
-
+    ge_df = PandasDataset(raw_products)
     result = ge_df.expect_column_values_to_not_be_null("name")
 
     if result["success"]:
@@ -101,21 +73,7 @@ def ge_check_price_positive(
     raw_products: pd.DataFrame,
 ) -> dg.AssetCheckResult:
     """Great Expectations check: Prices are positive (validity/accuracy)."""
-    if not GE_AVAILABLE:
-        return dg.AssetCheckResult(
-            passed=True,
-            severity=dg.AssetCheckSeverity.WARN,
-            description="Great Expectations not installed, skipping check",
-        )
-
-    ge_df = create_ge_dataset(raw_products)
-    if ge_df is None:
-        return dg.AssetCheckResult(
-            passed=False,
-            severity=dg.AssetCheckSeverity.WARN,
-            description="Failed to create Great Expectations dataset",
-        )
-
+    ge_df = PandasDataset(raw_products)
     result = ge_df.expect_column_values_to_be_between(
         "price", min_value=0.01, max_value=10000, mostly=0.9
     )
@@ -142,21 +100,7 @@ def ge_check_category_valid(
     raw_products: pd.DataFrame,
 ) -> dg.AssetCheckResult:
     """Great Expectations check: Categories are from valid set (consistency)."""
-    if not GE_AVAILABLE:
-        return dg.AssetCheckResult(
-            passed=True,
-            severity=dg.AssetCheckSeverity.WARN,
-            description="Great Expectations not installed, skipping check",
-        )
-
-    ge_df = create_ge_dataset(raw_products)
-    if ge_df is None:
-        return dg.AssetCheckResult(
-            passed=False,
-            severity=dg.AssetCheckSeverity.WARN,
-            description="Failed to create Great Expectations dataset",
-        )
-
+    ge_df = PandasDataset(raw_products)
     valid_categories = ["Electronics", "Clothing", "Home", "Sports", "Books"]
     result = ge_df.expect_column_values_to_be_in_set("category", valid_categories)
 

examples/data-quality-patterns/src/data_quality_patterns/defs/assets/python_checks.py

@@ -89,7 +89,9 @@ def check_accuracy_age(
         return dg.AssetCheckResult(
             passed=False,
             severity=dg.AssetCheckSeverity.ERROR,
-            description=f"Found {len(invalid_records)} records with invalid ages",
+            description=warnings[0]
+            if warnings
+            else f"Found {len(invalid_records)} records with invalid ages",
             metadata={
                 "invalid_count": len(invalid_records),
                 "invalid_ages": invalid_records["age"].tolist(),
@@ -220,13 +222,15 @@ def check_validity_email(
     validity_rate = calculate_validity(raw_customers, "email")
     threshold = 0.90
 
-    invalid_records, _ = validate_email_format(raw_customers, "email")
+    invalid_records, errors = validate_email_format(raw_customers, "email")
 
     if validity_rate < threshold:
         return dg.AssetCheckResult(
             passed=False,
             severity=dg.AssetCheckSeverity.ERROR,
-            description=f"Email validity {validity_rate:.1%} below threshold {threshold:.0%}",
+            description=errors[0]
+            if errors
+            else f"Email validity {validity_rate:.1%} below threshold {threshold:.0%}",
             metadata={
                 "validity_rate": float(validity_rate),
                 "invalid_count": len(invalid_records),
@@ -281,14 +285,16 @@ def check_uniqueness_customer_id(
 ) -> dg.AssetCheckResult:
     """Check that customer IDs are unique (uniqueness)."""
     uniqueness_rate = calculate_uniqueness(raw_customers, "customer_id")
-    duplicate_records, _ = check_uniqueness(raw_customers, ["customer_id"])
+    duplicate_records, errors = check_uniqueness(raw_customers, ["customer_id"])
 
     if not duplicate_records.empty:
         duplicate_ids = duplicate_records["customer_id"].unique().tolist()
         return dg.AssetCheckResult(
             passed=False,
             severity=dg.AssetCheckSeverity.ERROR,
-            description=f"Found {len(duplicate_ids)} duplicate customer IDs",
+            description=errors[0]
+            if errors
+            else f"Found {len(duplicate_ids)} duplicate customer IDs",
             metadata={
                 "uniqueness_rate": float(uniqueness_rate),
                 "duplicate_ids": duplicate_ids[:10],  # First 10
@@ -310,13 +316,13 @@ def check_uniqueness_order_id(
 ) -> dg.AssetCheckResult:
     """Check that order IDs are unique (uniqueness)."""
     uniqueness_rate = calculate_uniqueness(raw_orders, "order_id")
-    duplicate_records, _ = check_uniqueness(raw_orders, ["order_id"])
+    duplicate_records, errors = check_uniqueness(raw_orders, ["order_id"])
 
     if not duplicate_records.empty:
         return dg.AssetCheckResult(
             passed=False,
             severity=dg.AssetCheckSeverity.ERROR,
-            description="Found duplicate order IDs",
+            description=errors[0] if errors else "Found duplicate order IDs",
             metadata={
                 "uniqueness_rate": float(uniqueness_rate),
                 "duplicate_count": len(duplicate_records),
@@ -335,7 +341,11 @@ def check_uniqueness_order_id(
 # ============================================================================
 
 
-@dg.asset_check(asset=raw_orders, name="check_integrity_customer_ref", additional_ins={"raw_customers": dg.AssetIn("raw_customers")})
+@dg.asset_check(
+    asset=raw_orders,
+    name="check_integrity_customer_ref",
+    additional_ins={"raw_customers": dg.AssetIn("raw_customers")},
+)
 def check_integrity_customer_ref(
     context: dg.AssetCheckExecutionContext,
     raw_orders: pd.DataFrame,

examples/data-quality-patterns/src/data_quality_patterns/defs/assets/raw_data.py

@@ -49,8 +49,7 @@ def raw_customers(
     # Store in DuckDB for dbt to access
     with duckdb.get_connection() as conn:
         conn.execute("CREATE SCHEMA IF NOT EXISTS raw")
-        conn.execute("DROP TABLE IF EXISTS raw.customers")
-        conn.execute("CREATE TABLE raw.customers AS SELECT * FROM df")
+        conn.execute("CREATE OR REPLACE TABLE raw.customers AS SELECT * FROM df")
 
     context.add_output_metadata(
         {
@@ -67,7 +66,6 @@ def raw_customers(
 @dg.asset(
     group_name="raw_data",
     compute_kind="python",
-    deps=[raw_customers],
     freshness_policy=daily_freshness_policy,
 )
 def raw_orders(
@@ -102,8 +100,7 @@ def raw_orders(
     # Store in DuckDB for dbt to access
     with duckdb.get_connection() as conn:
         conn.execute("CREATE SCHEMA IF NOT EXISTS raw")
-        conn.execute("DROP TABLE IF EXISTS raw.orders")
-        conn.execute("CREATE TABLE raw.orders AS SELECT * FROM df")
+        conn.execute("CREATE OR REPLACE TABLE raw.orders AS SELECT * FROM df")
 
     context.add_output_metadata(
         {
@@ -137,8 +134,7 @@ def raw_products(
     # Store in DuckDB
     with duckdb.get_connection() as conn:
         conn.execute("CREATE SCHEMA IF NOT EXISTS raw")
-        conn.execute("DROP TABLE IF EXISTS raw.products")
-        conn.execute("CREATE TABLE raw.products AS SELECT * FROM df")
+        conn.execute("CREATE OR REPLACE TABLE raw.products AS SELECT * FROM df")
 
     context.add_output_metadata(
         {

examples/data-quality-patterns/src/data_quality_patterns/defs/resources/__init__.py

@@ -11,4 +11,3 @@
     "generate_orders",
     "generate_products",
 ]
-

examples/data-quality-patterns/src/data_quality_patterns/defs/resources/data_generator.py

@@ -113,7 +113,7 @@ def generate_customers(
 
         # Age - may have accuracy issues
         if has_accuracy_issue:
-            age = random.choice([-5, 0, 150, 200, 999])  # Invalid ages
+            age = random.choice([-5, -1, 150, 200, 999])  # Invalid ages (outside 0-120 range)
         else:
             age = random.randint(18, 80)
 
@@ -264,4 +264,3 @@ def generate_products(
         data.append(record)
 
     return pd.DataFrame(data)
-

examples/data-quality-patterns/src/data_quality_patterns/lib/expectations.py

@@ -1,28 +1,16 @@
 """Great Expectations integration utilities."""
 
-from typing import Any, Optional
-
 import pandas as pd
+from great_expectations.dataset import PandasDataset
 
-try:
-    from great_expectations.dataset import PandasDataset
-
-    GE_AVAILABLE = True
-except ImportError:
-    GE_AVAILABLE = False
-    PandasDataset = None
 
-
-def create_ge_dataset(df: pd.DataFrame) -> Optional[Any]:
+def create_ge_dataset(df: pd.DataFrame) -> PandasDataset:
     """Create a Great Expectations dataset from a pandas DataFrame.
 
     Args:
         df: pandas DataFrame to convert
 
     Returns:
-        Great Expectations PandasDataset, or None if GE is not available
+        Great Expectations PandasDataset
     """
-    if not GE_AVAILABLE or PandasDataset is None:
-        return None
-
     return PandasDataset(df)