Migrate background processing to supervisord

[durapensa]

Migrate Background Processing to Supervisord

Overview

Replace current background processing system with supervisord for better process management, monitoring, and reliability.

Motivation

Current background processing has limitations:

• Manual process tracking and cleanup
• Limited monitoring capabilities
• No automatic restart on failure
• Difficult to manage multiple concurrent processes

Supervisord provides:

• Robust process supervision
• Web interface and programmatic control
• Automatic restart capabilities
• Better logging and monitoring
• Industry-standard process management

Implementation Plan

This migration will be piloted through the logex system (Issue #24), then extended system-wide.

Phase 1: Logex Pilot

• Implement supervisord integration in logex dialogue composer
• Create tools/logex/supervisor for process monitoring
• Establish patterns for supervisord configuration and management

Phase 2: System Migration

• Replace existing background processing in analysis tools
• Update tools/plumbing/monitor-background-processes
• Migrate queue management to supervisord
• Update documentation and examples

Benefits

• More reliable background processing
• Better process monitoring and control
• Simplified debugging and troubleshooting
• Foundation for future scaling

Dependencies

• Issue Implement configuration-driven conversation harness for testing and experiments #24 (logex implementation as pilot)
• Review of existing background processing tools