Initial commit: Databricks wind turbine pipeline with secure credential management

- End-to-end data pipeline: Sensor → SQLite → Bacalhau → S3 → Databricks
- Docker container: databricks-uploader:v1.16.0 with environment variable support
- Security-first approach with credential templates and comprehensive .gitignore
- EC2 spot instance deployment with Bacalhau orchestration
- Full pipeline tested and operational

Author: aronchick

.cspell/custom-dictionary.txt

@@ -0,0 +1,50 @@
+ADLS
+apprunner
+asctime
+bacalhau
+bitnami
+buildx
+citus
+CLOB
+cpus
+creds
+dapi
+databrickscfg
+databricksruntime
+dbfs
+deltaio
+deltalake
+dind
+DISABLEANALYTICS
+dotenv
+drltab
+duckdb
+fcecd
+ferretdb
+gethostname
+healthcheck
+iloc
+IRUSR
+IWUSR
+Lakehouse
+levelname
+linuxcontainers
+makedirs
+minioadmin
+multiarch
+Newtonsoft
+nofile
+psql
+psycopg
+pyspark
+PYTHONUNBUFFERED
+pyyaml
+readwrite
+REARCHITECTURE
+referer
+specstory
+Sproc
+tvoc
+ulimits
+UNLOGGED
+venv

.cursor/rules/unit_testing_sqlite_uploader.mdc

@@ -0,0 +1,65 @@
+---
+description: 
+globs: 
+alwaysApply: false
+---
+# Unit Testing Guide for sqlite_to_databricks_uploader.py
+
+This guide outlines how to create simple unit tests for the [databricks-uploader/sqlite_to_databricks_uploader.py](mdc:databricks-uploader/sqlite_to_databricks_uploader.py) script. The goal is basic functional verification, not exhaustive testing or complex mocking.
+
+## General Approach
+
+1.  Use the standard Python `unittest` module.
+2.  Create a new test file, e.g., `databricks-uploader/test_uploader.py`.
+3.  Focus on testing pure functions or functions with easily verifiable inputs and outputs.
+4.  Avoid mocking database connections (SQLite, Databricks) or file system operations if it becomes complex. If a function is heavily reliant on I/O, consider testing its helper components or skipping it for these basic tests.
+
+## Functions to Test & Example Cases
+
+Here are some functions from `sqlite_to_databricks_uploader.py` that are good candidates for simple unit tests:
+
+### 1. `quote_databricks_identifier(name: str) -> str`
+
+*   **Purpose**: Tests if identifiers are correctly quoted with backticks and if internal backticks are escaped.
+*   **Test Cases**:
+    *   Input: `"my_table"` -> Expected Output: `"`my_table`"`
+    *   Input: `"table_with_`_backtick"` -> Expected Output: `"`table_with_``_backtick`"`
+    *   Input: `123` (as non-string) -> Expected Output: `"`123`"` (ensure it handles and logs conversion)
+
+### 2. `get_qualified_table_name(db_name_for_use_stmt: str, table_name_from_config: str) -> str`
+
+*   **Purpose**: Tests the construction of fully qualified table names.
+*   **Test Cases**:
+    *   `db_name_for_use_stmt="mydb"`, `table_name_from_config="mytable"` -> Expected: `"`mydb`"."`mytable`"`
+    *   `db_name_for_use_stmt="mydb"`, `table_name_from_config="myschema.mytable"` -> Expected: `"`myschema`"."`mytable`"` (and check for potential warning log if `myschema` != `mydb`)
+    *   `db_name_for_use_stmt=""`, `table_name_from_config="mytable"` -> Expect `ValueError` as per function logic.
+    *   `db_name_for_use_stmt="mydb"`, `table_name_from_config="my_schema.my_table"` -> Expected: `"`my_schema`"."`my_table`"`
+
+### 3. `parse_args()`
+
+*   **Purpose**: Tests if command-line arguments are parsed correctly and defaults are applied.
+*   **Test Method**:
+    *   Import the `parse_args` function.
+    *   Call `parse_args()` with a list of strings representing command-line arguments (e.g., `parse_args(['--sqlite', 'test.db', '--interval', '60'])`).
+    *   Assert that the attributes of the returned `Namespace` object have the expected values.
+*   **Test Cases**:
+    *   Test with a minimal set of arguments.
+    *   Test overriding default values (e.g., `--interval`, `--sqlite-batch-size`).
+    *   Test boolean flags (e.g., `--once`, `--verbose`).
+    *   Test that default values are present when arguments are not supplied (e.g., `interval` should default to 30, `sqlite_batch_size` to 1000).
+
+### 4. Placeholder Data Processing Functions
+    *   `sanitize_data(df: pd.DataFrame, config: dict) -> pd.DataFrame`
+    *   `filter_data(df: pd.DataFrame, config: dict) -> pd.DataFrame`
+    *   `aggregate_data(df: pd.DataFrame, config: dict) -> pd.DataFrame`
+*   **Purpose**: Since these are placeholders, tests should just verify they return the input DataFrame.
+*   **Test Method**: Create a sample Pandas DataFrame, pass it to these functions, and assert that the returned DataFrame is the same as the input (or an identical copy).
+
+## What to Avoid (for these basic tests)
+
+*   Testing the `main()` function directly due to its complexity and heavy I/O.
+*   Testing functions that heavily rely on external services or file system state unless very simple to set up (e.g., `read_data` or the main loop's SQLite/Databricks interaction parts).
+*   Complex mocking of `sqlite3`, `databricks.sql`, `databricks.sdk`, or `yaml` modules.
+
+The tests should be straightforward and confirm that the core logic of these utility and argument parsing functions behaves as expected under simple conditions.
+When writing the tests, import the necessary functions directly from `databricks-uploader.sqlite_to_databricks_uploader`.

.cursorindexingignore

@@ -0,0 +1,2 @@
+# Don't index SpecStory auto-save files, but allow explicit context inclusion via @ references
+.specstory/**

.githooks/README.md

@@ -0,0 +1,58 @@
+# Git Hooks
+
+This directory contains git hooks to prevent common issues in Databricks notebooks.
+
+## Setup
+
+To enable these hooks in your local repository, run:
+
+```bash
+git config core.hooksPath .githooks
+```
+
+## Hooks Included
+
+### pre-commit
+
+Validates Databricks notebooks before commit to check for:
+
+1. **Unsupported Triggers**: Prevents use of `processingTime` or `continuous` triggers which don't work on Databricks serverless compute
+2. **Old Path Patterns**: Warns about nested directory structures when flat structure is preferred
+3. **Missing Error Handling**: Warns about streaming queries without try/except blocks
+4. **Hardcoded Values**: Warns about hardcoded S3 bucket names
+
+## Manual Validation
+
+You can manually run the validation script without committing:
+
+```bash
+uv run -s scripts/check-databricks-notebooks.py
+```
+
+## Bypassing Hooks (Emergency Only)
+
+If you absolutely need to bypass the pre-commit hook (not recommended):
+
+```bash
+git commit --no-verify -m "your message"
+```
+
+## Troubleshooting
+
+If the hooks aren't running:
+
+1. Check that hooks path is configured:
+   ```bash
+   git config core.hooksPath
+   ```
+   Should output: `.githooks`
+
+2. Ensure hooks are executable:
+   ```bash
+   chmod +x .githooks/pre-commit
+   ```
+
+3. Verify uv is installed for Python scripts:
+   ```bash
+   curl -LsSf https://astral.sh/uv/install.sh | sh
+   ```

.githooks/pre-commit

@@ -0,0 +1,31 @@
+#!/bin/bash
+# Pre-commit hook to validate Python code
+
+set -e
+
+# Check if we have any Python changes
+if git diff --cached --name-only | grep -q "\.py$"; then
+    echo "Running validation on Python files..."
+    
+    # Run comprehensive validation
+    if command -v uv &> /dev/null; then
+        uv run -s scripts/validate-all.py check
+        
+        if [ $? -ne 0 ]; then
+            echo ""
+            echo "❌ Validation failed!"
+            echo "Fix the issues above before committing."
+            echo ""
+            echo "To auto-fix some issues, run:"
+            echo "  uv run -s scripts/validate-all.py fix"
+            exit 1
+        fi
+    else
+        echo "⚠️  uv not found. Install it with:"
+        echo "  curl -LsSf https://astral.sh/uv/install.sh | sh"
+        exit 1
+    fi
+fi
+
+echo "✅ All pre-commit checks passed!"
+exit 0

.githooks/pre-commit-databricks

@@ -0,0 +1,56 @@
+#!/bin/bash
+
+# Pre-commit hook to validate Databricks notebooks before committing
+# Install by running: ln -sf ../../.githooks/pre-commit-databricks .git/hooks/pre-commit
+
+set -e
+
+# Colors for output
+RED='\033[0;31m'
+GREEN='\033[0;32m'
+YELLOW='\033[1;33m'
+NC='\033[0m' # No Color
+
+echo -e "${YELLOW}Running Databricks notebook validation...${NC}"
+
+# Find all changed .py files in databricks-notebooks directory
+NOTEBOOKS=$(git diff --cached --name-only --diff-filter=ACM | \
+            grep -E '^databricks-notebooks/.*\.py$' || true)
+
+if [ -z "$NOTEBOOKS" ]; then
+    echo -e "${GREEN}No Databricks notebooks to validate${NC}"
+    exit 0
+fi
+
+# Check if validation script exists
+VALIDATOR="scripts/validate-databricks-notebook.py"
+if [ ! -f "$VALIDATOR" ]; then
+    echo -e "${YELLOW}Warning: Validator script not found at $VALIDATOR${NC}"
+    echo -e "${YELLOW}Skipping notebook validation${NC}"
+    exit 0
+fi
+
+# Validate each notebook
+FAILED=0
+for notebook in $NOTEBOOKS; do
+    echo -n "Validating $notebook... "
+    
+    if uv run -s "$VALIDATOR" "$notebook" --quiet; then
+        echo -e "${GREEN}✓${NC}"
+    else
+        echo -e "${RED}✗${NC}"
+        echo -e "${RED}Validation failed for $notebook${NC}"
+        echo "Run the following command for details:"
+        echo "  uv run -s $VALIDATOR $notebook --show-context"
+        FAILED=1
+    fi
+done
+
+if [ $FAILED -eq 1 ]; then
+    echo -e "${RED}Commit aborted due to validation errors${NC}"
+    echo "Fix the errors or use 'git commit --no-verify' to skip validation"
+    exit 1
+fi
+
+echo -e "${GREEN}All notebooks validated successfully${NC}"
+exit 0

.github/workflows/validate-notebooks.yml

@@ -0,0 +1,53 @@
+name: Validate Databricks Notebooks
+
+on:
+  push:
+    paths:
+      - 'databricks-notebooks/**/*.py'
+      - 'scripts/validate-databricks-notebook.py'
+  pull_request:
+    paths:
+      - 'databricks-notebooks/**/*.py'
+      - 'scripts/validate-databricks-notebook.py'
+
+jobs:
+  validate:
+    runs-on: ubuntu-latest
+    
+    steps:
+    - uses: actions/checkout@v4
+    
+    - name: Set up Python
+      uses: actions/setup-python@v4
+      with:
+        python-version: '3.11'
+    
+    - name: Install uv
+      run: |
+        curl -LsSf https://astral.sh/uv/install.sh | sh
+        echo "$HOME/.cargo/bin" >> $GITHUB_PATH
+    
+    - name: Find Databricks notebooks
+      id: find-notebooks
+      run: |
+        notebooks=$(find databricks-notebooks -name "*.py" -type f | tr '\n' ' ')
+        echo "notebooks=$notebooks" >> $GITHUB_OUTPUT
+        echo "Found notebooks: $notebooks"
+    
+    - name: Validate notebooks
+      if: steps.find-notebooks.outputs.notebooks != ''
+      run: |
+        for notebook in ${{ steps.find-notebooks.outputs.notebooks }}; do
+          echo "Validating $notebook..."
+          uv run -s scripts/validate-databricks-notebook.py "$notebook" \
+            --show-context || exit 1
+        done
+    
+    - name: Run strict validation
+      if: github.event_name == 'pull_request'
+      run: |
+        for notebook in ${{ steps.find-notebooks.outputs.notebooks }}; do
+          echo "Running strict validation on $notebook..."
+          uv run -s scripts/validate-databricks-notebook.py "$notebook" \
+            --strict --show-context || exit 1
+        done