Skip to content

bug-missing-sources-in-newsletter-prompt.md

Latest commit

 

History

History
301 lines (246 loc) · 15.6 KB

bug-missing-sources-in-newsletter-prompt.md

File metadata and controls

301 lines (246 loc) · 15.6 KB
Raw

Bug: Missing Sources in Generated Newsletter Prompt

Bug Description

When generating a newsletter, the generated prompt shows "## Active Sources (0 total)" with no source URLs listed below it, even though the user has selected sources in the dashboard. This results in the AI agent receiving an empty source list and being unable to generate a meaningful newsletter because it has no sources to search or analyze.

The bug manifests in the prompt template where:

## Active Sources (0 total)

Should instead show something like:

## Active Sources (5 total)
- NICE - HTA - https://www.nice.org.uk (Important)
- FDA - Regulatory - https://www.fda.gov (Important)
...

Problem Statement

The newsletter generation process creates a prompt for the Claude AI agent, but the sources array used to populate the "Active Sources" section is empty (sources.length === 0), even when the user has selected sources in the UI. This prevents the AI from knowing which sources to search and analyze, resulting in either a failed generation or a generic newsletter without actual data from the intended sources.

The root cause is that either:

  1. The selected source IDs are not being properly passed from the frontend to the backend
  2. The database query to fetch sources is failing or returning no results
  3. The source data is lost between the /api/generate POST endpoint and the /api/generate/stream GET endpoint

Solution Statement

Investigate and fix the data flow from source selection in the frontend through to the prompt generation in the streaming endpoint. Ensure that:

  1. Selected source IDs are correctly passed in the POST request to /api/generate
  2. Source IDs are properly stored in the newsletterGeneration record's config field
  3. The streaming endpoint correctly retrieves source IDs from the generation config
  4. The database query successfully fetches the source records
  5. Add validation and error handling to catch and report when sources are missing at each step
  6. Add a check to prevent newsletter generation when no sources are selected or found

Steps to Reproduce

  1. Navigate to https://catalyst-newsletter-1078437195105.europe-west1.run.app/login
  2. Log in with admin credentials
  3. Navigate to /dashboard
  4. Verify that sources are displayed and selected (checkboxes should be checked)
  5. Click "Generate Newsletter" button
  6. Observe that generation proceeds but the prompt shows "## Active Sources (0 total)"
  7. The generated newsletter is generic and does not include real data from the sources

Root Cause Analysis

After analyzing the code flow, the issue can occur at multiple points:

Data Flow:

  1. Frontend (dashboard/page.tsx:98-106): User clicks "Generate Newsletter" → POST to /api/generate with selectedSources: config.selectedSources
  2. Backend /api/generate (route.ts:11): Receives selectedSources array → Queries database for sources (line 14-21) → Creates generation record with config (line 48-55)
  3. Backend /api/generate/stream (route.ts:133-145): Retrieves generation record → Extracts selectedSources from config (line 134) → Queries database for sources (line 136-145)
  4. Prompt Generation (route.ts:150-152): Creates source list string → Builds prompt with sources (line 168-169)

Potential Root Causes:

  1. Empty selectedSources array: If config.selectedSources in the frontend is empty [], no sources will be queried

    • Location: /app/dashboard/page.tsx:43 - Initial state
    • Location: /app/dashboard/page.tsx:74-77 - Sources are set after fetching
    • Risk: If fetchSources() fails or returns empty array, selectedSources remains []

  2. Database query failure: The Prisma query at /app/api/generate/route.ts:14-21 might return 0 results if:

    • No sources have active: true in the database
    • The source IDs don't exist in the database
    • Database connection issues

  3. Config not properly stored: The agentConfig object might not be properly serialized when stored in the database at line 48-55 of /app/api/generate/route.ts

  4. Config not properly retrieved: The config extraction at /app/api/generate/stream/route.ts:133-134 might fail:

    const config = generation.config as any
const selectedSourceIds = config.selectedSources || []

    If config.selectedSources is undefined or null, it defaults to []

  5. Second database query fails: The query at /app/api/generate/stream/route.ts:136-145 might fail even if the first query succeeded

Most Likely Cause: Based on the bug report showing "Active Sources (0 total)" appearing in the generated prompt, the sources list is reaching the prompt generation step but is empty. This suggests the issue is likely in one of these areas:

  • The config.selectedSources array is empty when passed to /api/generate
  • The database query is failing to find matching sources (possibly due to active: false on sources)
  • The config is not being properly stored/retrieved from the database

Relevant Files

Use these files to fix the bug:

Backend Files - API Routes

  • /Users/clazz/Projects/Inovintell/catalyst-newsletter/app/api/generate/route.ts:8-28 - POST endpoint that receives selectedSources and creates generation record. This is where we first query for sources and store the config.

    • Why relevant: This is where sources are first fetched from the database and where the generation record is created with the config. If sources are empty here, they'll be empty downstream.
    • Key logic: Lines 14-21 query sources, lines 48-55 create generation record with config

  • /Users/clazz/Projects/Inovintell/catalyst-newsletter/app/api/generate/stream/route.ts:84-354 - GET endpoint that generates the newsletter. This is where the prompt is built with sources.

    • Why relevant: This is where the prompt showing "Active Sources (0 total)" is generated. Lines 133-145 retrieve sources from database, lines 150-169 build the prompt.
    • Key logic: Line 134 extracts selectedSourceIds from config, line 136-145 queries sources, line 168-169 shows source count in prompt

Frontend Files - Dashboard

  • /Users/clazz/Projects/Inovintell/catalyst-newsletter/app/dashboard/page.tsx:65-83 - fetchSources() function that loads available sources

    • Why relevant: This populates the sources list and sets config.selectedSources. If this fails, no sources are selected.
    • Key logic: Lines 71-77 set selectedSources to all fetched source IDs

  • /Users/clazz/Projects/Inovintell/catalyst-newsletter/app/dashboard/page.tsx:85-175 - handleGenerateNewsletter() function that sends generation request

    • Why relevant: This sends the selectedSources array to the backend. If the array is empty here, no sources will be used.
    • Key logic: Line 100 includes selectedSources: config.selectedSources in POST body

Database Schema (for context)

  • /Users/clazz/Projects/Inovintell/catalyst-newsletter/prisma/schema.prisma - Database schema for NewsSource and NewsletterGeneration models
    • Why relevant: Understanding the schema helps verify what fields are available and how data is stored
    • Key fields: active boolean on NewsSource, config JSON field on NewsletterGeneration

Step by Step Tasks

IMPORTANT: Execute every step in order, top to bottom.

1. Investigate Current State - Check Database for Active Sources

  • Run a query to check if there are any active sources in the database
  • Verify that sources exist and have active: true
  • Document the current count of active sources
  • If no active sources exist, this explains the bug immediately

2. Add Logging to /api/generate POST Endpoint

  • Add console.log to show received selectedSources array before database query
  • Add console.log to show query results (source count and IDs) after database query
  • Add console.log to show the config object before creating generation record
  • This will help us trace exactly where sources are lost

3. Add Validation to /api/generate POST Endpoint

  • After querying sources (line 21), check if sources.length === 0
  • If zero sources found, return a clear error response explaining why (no active sources, invalid IDs, etc.)
  • This prevents creating a generation record with no sources
  • Return HTTP 400 with detailed error message for frontend to display

4. Add Logging to /api/generate/stream GET Endpoint

  • Add console.log to show generation config after retrieving from database
  • Add console.log to show selectedSourceIds extracted from config
  • Add console.log to show query results (source count and IDs) after second database query
  • This will show if sources are being lost between endpoints

5. Add Validation to /api/generate/stream GET Endpoint

  • After querying sources (line 145), check if sources.length === 0
  • If zero sources found, send an 'error' event explaining that no sources were found
  • Include diagnostic information: generationId, selectedSourceIds from config
  • Do not proceed with newsletter generation if no sources available

6. Improve Frontend Error Handling

  • Update handleGenerateNewsletter() to check if config.selectedSources.length === 0 before making API call
  • Display user-friendly error message if no sources are selected
  • Prevent API call when selectedSources is empty to avoid wasting resources

7. Add Defensive Checks in Frontend Source Loading

  • In fetchSources(), add error logging if response is not ok
  • Add console.log to show how many sources were fetched and selected
  • Display error message to user if source loading fails
  • This helps diagnose if the issue is in source loading vs source querying

8. Test the Fix with Database Query

  • Use Prisma Studio or direct database access to verify active sources exist
  • Test newsletter generation with valid source selection
  • Test newsletter generation with no sources selected (should show validation error)
  • Test newsletter generation after deactivating all sources (should show validation error)
  • Verify error messages are clear and actionable for the user

9. Run Validation Commands

  • Execute all validation commands to ensure the bug is fixed with zero regressions
  • Verify that meaningful error messages appear when sources are missing
  • Verify that newsletter generation works when sources are properly selected

Validation Commands

Execute every command to validate the bug is fixed with zero regressions.

Database Validation

# Check if active sources exist in the database
cd /Users/clazz/Projects/Inovintell/catalyst-newsletter
npx prisma studio
# Manually verify: Open NewsSource table and check for records with active=true

Manual Testing Steps

  1. Test Case: No Active Sources in Database

    • Deactivate all sources via Prisma Studio or admin interface
    • Navigate to /dashboard
    • Try to generate newsletter
    • Expected: Clear error message indicating no active sources are available
    • Actual: Should NOT show "Active Sources (0 total)" in generated prompt

  2. Test Case: Sources Selected but Not Sent to Backend

    • Activate at least 3 sources in database
    • Navigate to /dashboard
    • Verify sources appear and are selected
    • Open browser DevTools → Network tab
    • Click "Generate Newsletter"
    • Check POST request to /api/generate
    • Verify: Request body includes selectedSources array with numeric IDs
    • Verify: Response includes sources count matching selection

  3. Test Case: Successful Newsletter Generation

    • Ensure at least 3 sources are active in database
    • Navigate to /dashboard
    • Verify sources are selected (should auto-select all active sources)
    • Click "Generate Newsletter"
    • Open browser DevTools → Console
    • Check for logging output showing source count at each step
    • Verify: Newsletter generation starts successfully
    • Verify: Prompt shows "Active Sources (X total)" where X > 0
    • Verify: Source list appears below the header with actual source URLs

  4. Test Case: Frontend Validation

    • Navigate to /dashboard
    • Manually deselect all sources
    • Click "Generate Newsletter"
    • Expected: Error message appears without making API call
    • Actual: User-friendly message explaining that sources must be selected

Server-Side Log Validation

# Check Cloud Run logs for debugging output
cd /Users/clazz/Projects/Inovintell/catalyst-newsletter
gcloud logging read "resource.type=cloud_run_revision AND resource.labels.service_name=catalyst-newsletter" --limit 50 --format json | grep -E "(selectedSources|sources.length|Active Sources)"

# Or check local logs if running via docker-compose
docker-compose logs app | grep -E "(selectedSources|sources.length|Active Sources)"

Build and Type Check

# Ensure TypeScript compilation succeeds
cd /Users/clazz/Projects/Inovintell/catalyst-newsletter
npm run build

Deployment Test

# After fixing, deploy and test on production
cd /Users/clazz/Projects/Inovintell/catalyst-newsletter
./deploy/gcp-deploy.sh
# Wait for deployment to complete, then test newsletter generation on production URL

Notes

Why This Bug is Critical

This bug completely breaks the core functionality of the newsletter generation system. Without sources, the AI agent cannot:

  • Search for relevant news and updates
  • Aggregate information from HTA agencies
  • Provide meaningful market access intelligence
  • Generate valuable content for the newsletter

The generated newsletter would be generic and useless to users who expect timely, relevant HTA and market access information.

Data Flow Architecture

Understanding the data flow is critical to fixing this bug:

Frontend (dashboard)
  └─> POST /api/generate
      ├─> Query DB for sources
      ├─> Store config with selectedSources
      └─> Return generationId

Frontend (dashboard)
  └─> GET /api/generate/stream?id={generationId}
      ├─> Retrieve generation record
      ├─> Extract selectedSources from config
      ├─> Query DB for sources AGAIN
      ├─> Build prompt with sources
      └─> Stream newsletter generation

Key Insight: Sources are queried TWICE - once in POST endpoint, once in GET endpoint. The bug could occur at either step.

Possible Database Issues

  • Sources might exist but have active: false
  • Source IDs in config might not match actual database IDs
  • Prisma Client might not be properly initialized
  • Database connection might be failing silently

Frontend State Management Issues

  • config.selectedSources might not be updating when sources are fetched
  • React state updates might be asynchronous and not completing before generation
  • User might be clicking "Generate" before sources are loaded

Why Validation is Important

Adding validation at multiple layers prevents this bug from manifesting silently:

  1. Frontend validation: Prevents wasting API calls and provides immediate feedback
  2. POST endpoint validation: Catches issues early and returns clear errors
  3. Stream endpoint validation: Last line of defense before generating with no sources

Best Practices for Fix

  • Always validate inputs at API boundaries
  • Log diagnostic information at key decision points
  • Return specific, actionable error messages
  • Fail fast and loud rather than proceeding with invalid state
  • Add defensive coding to handle edge cases (empty arrays, null values)

Future Improvements

After fixing this bug, consider:

  1. Adding integration tests that verify end-to-end source flow
  2. Creating a health check endpoint that verifies active sources exist
  3. Adding metrics/monitoring for source selection counts
  4. Implementing source caching to reduce database queries
  5. Adding Prisma query logging in development to debug database issues
  6. Creating a sources dashboard that shows source status and statistics