Skip to content

LLM_TESTS_DEBUG.md

Latest commit

 

History

History
87 lines (69 loc) · 2.61 KB

LLM_TESTS_DEBUG.md

File metadata and controls

87 lines (69 loc) · 2.61 KB

LLM Tests Debugging Summary

GitHub Workflow Issue Fixed ✅

Problem Identified

The .github/workflows/copilot-setup-steps.yml had a critical step ordering issue:

Before (BROKEN):

  1. Create .env.llm-tests file
  2. Checkout code ← This overwrites the directory, losing the .env file!
  3. Run tests

After (FIXED):

  1. Checkout code
  2. Create .env.llm-tests file ← Now persists correctly
  3. Run tests

Changes Made

  • Moved "Persist Secrets to Agent Environment" step AFTER "Checkout code"
  • Added export prefix to environment variables for proper shell sourcing
  • Added RUN_LLM_TESTS=true to enable tests automatically
  • Added chmod 600 for security
  • Added verification logging to confirm file creation

Environment Setup Verification

API Key Sourcing ✅

The .env.llm-tests sourcing mechanism works correctly:

# Create .env file
echo 'export OPENAI_API_KEY=sk-your-key' > .env.llm-tests
echo 'export RUN_LLM_TESTS=true' >> .env.llm-tests

# Source and verify
source .env.llm-tests
echo $OPENAI_API_KEY  # Shows the key

Test Detection ✅

When the environment is properly sourced, tests correctly:

  • Detect the API key presence
  • Enable live test execution (not skipped)
  • Show provider detection: "Running LLM integration tests with provider: openai"

Current Limitation ⚠️

Tests fail in the jsdom environment with network errors:

Error: Cross origin null forbidden
Error: LLM API call failed: Network Error

This is expected because:

  1. Tests run in a jsdom environment (not a real browser)
  2. axios HTTP requests fail due to CORS restrictions in jsdom
  3. Live API tests need a proper Node.js environment or network mocking

Recommendations

For Local Development

Run tests with a real API key in a Node environment:

source .env.llm-tests
cd app && yarn test

For CI/CD

The workflow now correctly:

  1. Checks out the repository first
  2. Creates .env.llm-tests in the workspace
  3. Makes the API key available to subsequent steps

Consider:

  1. Running tests in a Node environment (not jsdom)
  2. Using nock or msw to mock HTTP requests in tests
  3. Running live tests only in scheduled jobs with proper network access

Verified Working

  • ✅ .env.llm-tests creation via workflow (step order fixed)
  • ✅ .env.llm-tests creation via setup-llm-env.sh
  • ✅ Environment variable sourcing
  • ✅ Test detection of API keys
  • ✅ Provider auto-detection (OpenAI/Gemini)
  • ✅ Proper skip behavior when no API key

Status

The infrastructure is now working correctly. The workflow step order has been fixed to ensure .env.llm-tests persists after checkout.