[recipes] Brain health monitoring views

recipes/brain-health-monitoring/README.md

@@ -0,0 +1,134 @@
+# Brain Health Monitoring
+
+<div align="center">
+
+![Community Contribution](https://img.shields.io/badge/OB1_COMMUNITY-Approved_Contribution-2ea44f?style=for-the-badge&logo=github)
+
+**Created by [@alanshurafa](https://github.com/alanshurafa)**
+
+</div>
+
+> SQL views and runbook for monitoring source volumes, enrichment gaps, ingestion pipeline health, stalled queues, and knowledge graph coverage.
+
+## What It Does
+
+Adds 8 monitoring views to your Open Brain database that answer the most common operational questions:
+
+| View | What It Shows |
+|------|---------------|
+| `ops_source_volume_24h` | Thought counts per source in the last 24 hours |
+| `ops_recent_thoughts` | Latest thoughts with type, source, enrichment status, and preview |
+| `ops_enrichment_gaps` | Thoughts that haven't been enriched yet |
+| `ops_type_distribution` | Type breakdown (all-time, 7-day, 24-hour windows) |
+| `ops_sensitivity_distribution` | Sensitivity tier breakdown |
+| `ops_ingestion_summary` | Ingestion job status and counts (requires `schemas/smart-ingest`) |
+| `ops_stalled_entity_queue` | Queue items stuck or permanently failed (requires `schemas/entity-extraction`) |
+| `ops_graph_coverage` | Entity extraction progress and coverage percentage (requires `schemas/entity-extraction`) |
+
+Views 1-5 work with the base enhanced thoughts schema. Views 6-8 are wrapped in `to_regclass` guards, so the SQL file runs cleanly on any shape of install — missing optional tables produce a `NOTICE` and the corresponding view is skipped rather than failing.
+
+## Prerequisites
+
+- Working Open Brain setup ([guide](../../docs/01-getting-started.md))
+- **Enhanced thoughts schema** applied — install `schemas/enhanced-thoughts` (required for all views)
+- Optional: `schemas/smart-ingest` for the ingestion summary view (view 6)
+- Optional: `schemas/entity-extraction` for the stalled queue and graph coverage views (views 7-8)
+
+## Steps
+
+1. Review which monitoring views apply to your installed schemas.
+2. Run `ops-views.sql` in the Supabase SQL Editor.
+3. Verify the `ops_*` views were created successfully.
+4. Query the views to establish a baseline health check.
+
+### 1. Review the SQL File
+
+Open `ops-views.sql` and check which views apply to your setup:
+
+- **Views 1-5** (source volume, recent thoughts, enrichment gaps, type/sensitivity distribution): Work with any Open Brain install that has the enhanced thoughts schema.
+- **View 6** (ingestion summary): Requires the `ingestion_jobs` table from `schemas/smart-ingest`.
+- **Views 7-8** (stalled queue, graph coverage): Require the `entity_extraction_queue` table from `schemas/entity-extraction`.
+
+You do not need to comment anything out. Views 6-8 are wrapped in `to_regclass` guards; if the underlying tables are missing, the DO blocks emit a `NOTICE` and skip the view without aborting the file.
+
+### 2. Run the SQL
+
+In the Supabase SQL Editor, paste the contents of `ops-views.sql` and execute. All statements use `CREATE OR REPLACE VIEW`, so running multiple times is safe.
+
+```bash
+# Or via psql:
+psql "$DATABASE_URL" -f ops-views.sql
+```
+
+### 3. Verify Views Exist
+
+```sql
+SELECT table_name
+FROM information_schema.views
+WHERE table_schema = 'public'
+  AND table_name LIKE 'ops_%'
+ORDER BY table_name;
+```
+
+You should see between 5 and 8 views depending on which schemas are installed.
+
+### 4. Run Your First Health Check
+
+```sql
+-- How many thoughts arrived in the last 24 hours, by source?
+SELECT * FROM ops_source_volume_24h;
+
+-- How many thoughts are waiting for enrichment?
+SELECT count(*) AS unenriched FROM ops_enrichment_gaps;
+
+-- What's the type distribution?
+SELECT * FROM ops_type_distribution;
+```
+
+## Runbook: What "Healthy" Looks Like
+
+### Fresh Install (< 100 thoughts)
+
+- `ops_source_volume_24h`: 0-10 thoughts, mostly from `mcp` or `rest_api`
+- `ops_enrichment_gaps`: May show all thoughts if enrichment hasn't run yet — this is normal
+- `ops_type_distribution`: Mostly `idea` (default type before enrichment)
+- `ops_sensitivity_distribution`: All `standard` unless you've captured sensitive content
+
+### Established Brain (1000+ thoughts)
+
+- `ops_source_volume_24h`: Regular flow from expected sources. If a source drops to 0, check the capture pipeline.
+- `ops_enrichment_gaps`: Should be near 0 if the enrichment pipeline is active. A growing backlog means enrichment is stalled.
+- `ops_type_distribution`: Diverse types across `idea`, `decision`, `lesson`, `reference`, `person_note`, etc. If everything is `idea`, the classifier may not be running.
+- `ops_sensitivity_distribution`: Mostly `standard` with some `personal`. A spike in `restricted` is worth investigating.
+- `ops_ingestion_summary`: Mostly `complete` jobs. `failed` jobs need error investigation.
+- `ops_graph_coverage`: `coverage_pct` should climb toward 100% over time. Stalled at a low percentage means the entity worker isn't running.
+- `ops_stalled_entity_queue`: Should be empty. Items here need manual intervention (reset `processing` items, investigate `failed` items).
+
+### Common Remediation Actions
+
+| Symptom | Action |
+|---------|--------|
+| Source volume dropped to 0 | Check the capture integration (MCP server, REST API, webhook) |
+| Large enrichment gap | Run the thought enrichment pipeline (`recipes/thought-enrichment`) |
+| All types are "idea" | Verify the LLM classifier is configured (`OPENROUTER_API_KEY` set) |
+| Stalled queue items | Reset with: `UPDATE entity_extraction_queue SET status = 'pending' WHERE status = 'processing' AND started_at < now() - interval '10 minutes'` |
+| Failed queue items | Check `last_error` column. Common: LLM rate limits, empty content |
+| Low graph coverage | Run the entity extraction worker (`integrations/entity-extraction-worker`) |
+
+## Expected Outcome
+
+After running the SQL, you should be able to query any `ops_*` view from the Supabase SQL Editor, your dashboard, or the REST API to get a real-time picture of your brain's health. These views are also available through PostgREST if you need to query them programmatically.
+
+## Troubleshooting
+
+**"relation ops_ingestion_summary does not exist"**
+The `ingestion_jobs` table isn't installed, so the guarded DO block skipped view 6 and emitted a `NOTICE`. Install `schemas/smart-ingest` and re-run `ops-views.sql` to create the view.
+
+**"relation ops_stalled_entity_queue does not exist" or "relation ops_graph_coverage does not exist"**
+The `entity_extraction_queue` table isn't installed, so views 7-8 were skipped. Install `schemas/entity-extraction` and re-run `ops-views.sql`.
+
+**Views return empty results**
+This is normal for a fresh install with no thoughts. Capture a few thoughts first, then query the views.
+
+**Permission denied on a view**
+Ensure the GRANT statements at the end of the SQL file executed successfully. Re-run them if needed.

recipes/brain-health-monitoring/metadata.json

@@ -0,0 +1,17 @@
+{
+  "name": "Brain Health Monitoring",
+  "description": "SQL views and runbook for monitoring source volumes, enrichment gaps, ingestion pipeline health, stalled queues, and knowledge graph coverage.",
+  "category": "recipes",
+  "author": {
+    "name": "Alan Shurafa",
+    "github": "alanshurafa"
+  },
+  "version": "1.0.0",
+  "requires": {
+    "open_brain": true,
+    "tools": ["Supabase SQL Editor or psql"]
+  },
+  "tags": ["monitoring", "ops", "health", "observability", "views"],
+  "difficulty": "beginner",
+  "estimated_time": "15 minutes"
+}

recipes/brain-health-monitoring/ops-views.sql

@@ -0,0 +1,198 @@
+-- Operational Monitoring and Brain Health Views
+-- Provides SQL views for monitoring source volumes, enrichment gaps,
+-- ingestion pipeline health, entity extraction queue, and graph coverage.
+-- Safe to run multiple times (CREATE OR REPLACE).
+--
+-- Required: Enhanced thoughts schema (schemas/enhanced-thoughts)
+-- Optional: Smart ingest schema (schemas/smart-ingest) for the ingestion summary view
+-- Optional: Entity extraction schema (schemas/entity-extraction) for queue and
+--           graph coverage views
+-- Views 6-8 are wrapped in `to_regclass` guards, so running this file on a
+-- stock install (only enhanced-thoughts) creates views 1-5 cleanly and emits
+-- a NOTICE for each skipped optional view.
+
+-- ============================================================
+-- 1. SOURCE VOLUME (24h)
+--    How many thoughts arrived from each source in the last day.
+--    Quick pulse check — if a source goes silent, investigate.
+-- ============================================================
+
+CREATE OR REPLACE VIEW public.ops_source_volume_24h AS
+SELECT
+  coalesce(source_type, 'unknown') AS source,
+  count(*)::bigint AS thoughts_24h
+FROM public.thoughts
+WHERE created_at >= now() - interval '24 hours'
+GROUP BY 1
+ORDER BY thoughts_24h DESC;
+
+-- ============================================================
+-- 2. RECENT THOUGHTS WITH SOURCE
+--    Last N thoughts with source, type, topics, and preview.
+--    Useful for spot-checking what's flowing in.
+-- ============================================================
+
+CREATE OR REPLACE VIEW public.ops_recent_thoughts AS
+SELECT
+  id,
+  created_at,
+  coalesce(type, 'unknown') AS type,
+  coalesce(source_type, 'unknown') AS source,
+  importance,
+  sensitivity_tier,
+  enriched,
+  left(content, 180) AS preview
+FROM public.thoughts
+WHERE sensitivity_tier IS DISTINCT FROM 'restricted'
+ORDER BY created_at DESC;
+
+-- ============================================================
+-- 3. ENRICHMENT GAPS
+--    Thoughts that haven't been enriched yet. If this grows,
+--    the enrichment pipeline may be stalled or misconfigured.
+-- ============================================================
+
+CREATE OR REPLACE VIEW public.ops_enrichment_gaps AS
+SELECT
+  id,
+  created_at,
+  coalesce(type, 'unknown') AS type,
+  coalesce(source_type, 'unknown') AS source,
+  left(content, 180) AS preview
+FROM public.thoughts
+WHERE enriched IS NOT TRUE
+  AND sensitivity_tier IS DISTINCT FROM 'restricted'
+ORDER BY created_at DESC;
+
+-- ============================================================
+-- 4. TYPE DISTRIBUTION
+--    How thoughts are distributed across types.
+--    Helps spot classification drift or misconfigured sources.
+-- ============================================================
+
+CREATE OR REPLACE VIEW public.ops_type_distribution AS
+SELECT
+  coalesce(type, 'unclassified') AS type,
+  count(*)::bigint AS total,
+  count(*) FILTER (WHERE created_at >= now() - interval '7 days')::bigint AS last_7d,
+  count(*) FILTER (WHERE created_at >= now() - interval '24 hours')::bigint AS last_24h
+FROM public.thoughts
+GROUP BY 1
+ORDER BY total DESC;
+
+-- ============================================================
+-- 5. SENSITIVITY DISTRIBUTION
+--    How thoughts break down by sensitivity tier.
+--    A sudden spike in "restricted" warrants investigation.
+-- ============================================================
+
+CREATE OR REPLACE VIEW public.ops_sensitivity_distribution AS
+SELECT
+  coalesce(sensitivity_tier, 'standard') AS tier,
+  count(*)::bigint AS total
+FROM public.thoughts
+GROUP BY 1
+ORDER BY total DESC;
+
+-- ============================================================
+-- 6. INGESTION JOB SUMMARY (requires smart-ingest schema)
+--    Status breakdown of ingestion jobs. Healthy brains should
+--    show mostly "complete" with few "failed".
+--    Guarded: only installed if public.ingestion_jobs exists.
+-- ============================================================
+
+DO $$
+BEGIN
+  IF to_regclass('public.ingestion_jobs') IS NOT NULL THEN
+    EXECUTE $v$
+      CREATE OR REPLACE VIEW public.ops_ingestion_summary AS
+      SELECT
+        status,
+        count(*)::bigint AS job_count,
+        sum(added_count)::bigint AS total_added,
+        sum(skipped_count)::bigint AS total_skipped,
+        max(completed_at) AS last_completed
+      FROM public.ingestion_jobs
+      GROUP BY status
+      ORDER BY job_count DESC
+    $v$;
+    EXECUTE 'GRANT SELECT ON public.ops_ingestion_summary TO service_role';
+  ELSE
+    RAISE NOTICE 'skipping ops_ingestion_summary -- public.ingestion_jobs not found (install schemas/smart-ingest)';
+  END IF;
+END$$;
+
+-- ============================================================
+-- 7. STALLED ENTITY QUEUE (requires entity-extraction schema)
+--    Queue items stuck in "processing" for more than 10 minutes,
+--    or items that have failed repeatedly.
+--    Guarded: only installed if public.entity_extraction_queue exists.
+-- ============================================================
+
+DO $$
+BEGIN
+  IF to_regclass('public.entity_extraction_queue') IS NOT NULL THEN
+    EXECUTE $v$
+      CREATE OR REPLACE VIEW public.ops_stalled_entity_queue AS
+      SELECT
+        thought_id,
+        status,
+        attempt_count,
+        last_error,
+        started_at,
+        queued_at
+      FROM public.entity_extraction_queue
+      WHERE (status = 'processing' AND started_at < now() - interval '10 minutes')
+         OR (status = 'failed')
+      ORDER BY queued_at DESC
+    $v$;
+    EXECUTE 'GRANT SELECT ON public.ops_stalled_entity_queue TO service_role';
+  ELSE
+    RAISE NOTICE 'skipping ops_stalled_entity_queue -- public.entity_extraction_queue not found (install schemas/entity-extraction)';
+  END IF;
+END$$;
+
+-- ============================================================
+-- 8. GRAPH COVERAGE (requires entity-extraction schema)
+--    How many thoughts have been processed for entity extraction
+--    vs how many are still pending.
+--    Guarded: only installed if public.entity_extraction_queue exists.
+-- ============================================================
+
+DO $$
+BEGIN
+  IF to_regclass('public.entity_extraction_queue') IS NOT NULL THEN
+    EXECUTE $v$
+      CREATE OR REPLACE VIEW public.ops_graph_coverage AS
+      SELECT
+        count(*) FILTER (WHERE status = 'complete')::bigint AS extracted,
+        count(*) FILTER (WHERE status = 'pending')::bigint AS pending,
+        count(*) FILTER (WHERE status = 'processing')::bigint AS processing,
+        count(*) FILTER (WHERE status = 'failed')::bigint AS failed,
+        count(*)::bigint AS total_queued,
+        CASE
+          WHEN count(*) > 0
+          THEN round(100.0 * count(*) FILTER (WHERE status = 'complete') / count(*), 1)
+          ELSE 0
+        END AS coverage_pct
+      FROM public.entity_extraction_queue
+    $v$;
+    EXECUTE 'GRANT SELECT ON public.ops_graph_coverage TO service_role';
+  ELSE
+    RAISE NOTICE 'skipping ops_graph_coverage -- public.entity_extraction_queue not found (install schemas/entity-extraction)';
+  END IF;
+END$$;
+
+-- ============================================================
+-- 9. GRANTS (for always-installed views 1-5)
+--    Views 6-8 are granted inside their guarded DO blocks above
+--    so grants only run when the view was actually created.
+-- ============================================================
+
+GRANT SELECT ON public.ops_source_volume_24h TO service_role;
+GRANT SELECT ON public.ops_recent_thoughts TO service_role;
+GRANT SELECT ON public.ops_enrichment_gaps TO service_role;
+GRANT SELECT ON public.ops_type_distribution TO service_role;
+GRANT SELECT ON public.ops_sensitivity_distribution TO service_role;
+
+NOTIFY pgrst, 'reload schema';