Full Pipeline Test Implementation with History Tracking

[jeremymanning]

Overview

This issue tracks the implementation of a comprehensive pipeline test system for llmXive, including:

• Mock model system for deterministic testing
• Complete validation framework for artifacts and scoring
• Project history tracking system
• End-to-end test of the entire scientific discovery pipeline

Related Issues

• Parent Issue: llmXive Automation System: Fully Autonomous Scientific Discovery Pipeline #21 (llmXive Automation System)

Documentation

The following documentation has been created for this implementation:

1. Full Pipeline Test Plan - Comprehensive overview of the testing approach
2. Pipeline Validation Framework - Detailed validation system for artifacts and scoring
3. Project History Tracking - System for tracking complete project evolution
4. Implementation Roadmap - Step-by-step implementation plan

Goals

• Validate the complete pipeline from project creation to publication
• Test all project state transitions and review mechanisms
• Ensure proper GitHub integration and web interface updates
• Track complete history of every project with timestamps and attribution
• Create a repeatable test framework for future development

Implementation Timeline

• Week 1: Mock models, scoring system, validation framework
• Week 2: Pipeline orchestrator, execution environment, integration
• Week 3: History tracking, bug fixes, final testing

Total estimated time: 16 days

[jeremymanning]

Full Pipeline Test Implementation Plan

This document outlines the implementation plan for testing the complete llmXive project pipeline from ideation to publication.

Architecture Overview

1. Mock Model System

   • Create deterministic, controllable model responses for testing
   • Pre-defined responses for each task type
   • Configurable review outcomes (approval, rejection, revision requests)

2. Enhanced Task Executor

   • Code execution in sandboxed environment
   • Real data generation and analysis
   • Figure generation and validation
   • LaTeX compilation to PDF

3. Pipeline Test Orchestrator

   • Step-by-step execution with checkpoints
   • State validation after each step
   • Error recovery and retry logic
   • Progress tracking and reporting

Test Execution Plan (38 Steps)

Stage 1: Project Creation

1. Generate test idea
2. Create GitHub issue
3. Add to project board (Backlog)
4. Generate initial labels

Stage 2: Technical Design
5-11. Create design, reviews, address feedback, advance to Ready

Stage 3: Implementation Planning
12-17. Create plan, reviews, address feedback, advance to In Progress

Stage 4: Code Development
18-25. Generate code, test, debug, analyze, create figures, iterate

Stage 5: Paper Writing
26-33. Write all sections, reviews, revisions

Stage 6: Publication
34-38. Compile PDF, organize code, move to Done, verify web update

Implementation Timeline

• Phase 1: 3 days (Infrastructure)
• Phase 2: 4 days (Task Implementation)
• Phase 3: 3 days (Test Scenarios)
• Phase 4: 4 days (Integration & Testing)
• Total: 14 days

[Full document in code/llmxive-automation/docs/full_pipeline_test_plan.md]

[jeremymanning]

Pipeline Validation Framework

This document defines the comprehensive validation framework for testing the llmXive pipeline.

Key Components

1. Artifact Validation System

   • File validators for all artifact types (designs, plans, code, data, papers)
   • GitHub validators for issues, labels, comments, project board
   • Comprehensive checks for structure, content, and references

2. Score System Validation

   • Score values: LLM +0.5/-0.5, Human +1.0/-1.0
   • Critical reviews reset score to 0
   • Score floor at 0, advancement at ≥5
   • Comprehensive test scenarios for all edge cases

3. Stage Transition Validation

   • Validates transitions follow defined rules
   • Checks requirements met before advancement
   • Ensures proper artifact creation at each stage

Scoring System Details

class ScoreSystem:
    LLM_POSITIVE = 0.5
    LLM_NEGATIVE = -0.5
    HUMAN_POSITIVE = 1.0
    HUMAN_NEGATIVE = -1.0
    CRITICAL_RESET = 0
    ADVANCEMENT_THRESHOLD = 5.0

Test Scenarios

1. Basic advancement: 10 LLM reviews → score 5.0 → advance
2. Mixed reviews: Handle positive and negative reviews correctly
3. Critical reset: Score resets to 0 on critical issue
4. Score floor: Negative reviews can't push score below 0

Integration Test Framework

• Validates each pipeline step
• Checks artifact creation
• Verifies GitHub state changes
• Tracks score progression
• Generates detailed validation reports

[Full document in code/llmxive-automation/docs/pipeline_validation_framework.md]

[jeremymanning]

Project History Tracking System

A comprehensive history tracking system that records every action, decision, and contribution throughout a project's lifecycle.

Features

1. Dual Format Storage

   • history.md - Human-readable timeline with emojis and links
   • history.json - Machine-readable detailed event log

2. Event Types Tracked

   • Project creation and ideation
   • Document creation and updates
   • All reviews (positive, negative, critical)
   • Code generation and testing
   • Data analysis and figure creation
   • Paper writing and compilation
   • Stage transitions
   • Human interactions (upvotes, comments)

3. History Entry Format

   10:35:22 | `tinyllama-1.1b` | 📋 [Reviewed](link) technical design ([v1.0](commit-link))
   - Score: +0.5 (Total: 0.5)
   - Highlighted strong theoretical foundation
   

4. Paper Integration

   • Auto-generated "Project History" section
   • Summary statistics and timeline visualization
   • Complete chronological event list with links
   • Contribution analysis by model/human

Implementation

• ProjectHistoryTracker class for recording events
• Integration with all task executions
• Automatic commit tracking and linking
• Real-time score and stage tracking
• HTML timeline visualization for web

Benefits

• Complete audit trail with attribution
• Research insights into discovery process
• Reproducibility through detailed logs
• Transparent contribution tracking
• Enhanced paper documentation

[Full document in code/llmxive-automation/docs/project_history_tracking.md]

[jeremymanning]

Implementation Roadmap

Step-by-step implementation plan for the full pipeline test with validation and history tracking.

Schedule Overview

Week 1: Foundation

• Days 1-2: Mock Model Infrastructure
• Days 3-4: Scoring System Implementation
• Days 5-6: Artifact Validation System

Week 2: Core Development

• Days 7-8: Pipeline Test Orchestrator
• Days 9-10: Execution Environment
• Days 11-12: Integration Testing

Week 3: Features & Polish

• Days 13-14: History Tracking System
• Days 15-16: Bug fixes and final testing

Key Implementation Details

1. Mock Model System

   • Deterministic responses via scenario files
   • Multiple model personalities
   • Scripted review outcomes

2. Scoring Implementation

   • Accurate point calculations
   • Label management
   • Stage transition validation

3. Execution Environment

   • Docker sandbox for code execution
   • Real data/figure generation
   • LaTeX compilation support

4. History Integration

   • Automatic event recording
   • Markdown/JSON dual format
   • Paper section generation

Success Metrics

1. All 38 pipeline steps execute successfully
2. Score calculations match expected values
3. All artifacts created correctly
4. Stage transitions follow rules
5. Complete history tracking with attribution
6. Auto-generated paper history section
7. Full test completes in <30 minutes

Next Steps

1. Set up development environment
2. Create mock model infrastructure
3. Implement scoring system with tests
4. Build validation framework
5. Integrate all components

[Full document in code/llmxive-automation/docs/implementation_roadmap.md]