Skip to content

[FEATURE] #880 - Add batch run summary with telemetry to CliConsoleManager and listeners #913

@stephanj

Description

@stephanj

Parent Issue

Part of #880 — Introduce Batch-Level Learning and Strategy Memory

Context

CliConsoleManager currently prints per-task headers and process output but no run-level summary. SpecTaskRunnerListener.onRunFinished() receives only completed, skipped, total, and finalState counts. There is no telemetry breakdown at the end of a batch run.

Task

Add a comprehensive batch run summary to the console and listener interface.

Key Changes

  1. Create BatchRunSummary model:

    @Data
    public class BatchRunSummary {
        String batchId;                      // UUID for this run
        Instant startedAt;
        Instant finishedAt;
        long totalExecutionTimeMs;
        int totalTasks;
        int completedTasks;
        int skippedTasks;
        int failedTasks;
        long totalInputTokens;
        long totalOutputTokens;
        double totalCost;
        RunnerState finalState;
        List<TaskExecutionMetrics> taskMetrics;  // per-task breakdown
        Map<TaskResultStatus, Integer> statusBreakdown;
    }
  2. Build summary in SpecTaskRunnerService.finish():

    • Aggregate all TaskExecutionMetrics collected during run
    • Calculate totals and breakdowns
    • Pass to listener via extended onRunFinished()
  3. Print to CliConsoleManager:

    ═══════════════════════════════════════════════
    Batch Run Summary
    ═══════════════════════════════════════════════
    Total tasks:    8  (6 completed, 1 skipped, 1 failed)
    Total time:     4m 32s
    Total tokens:   145,230 in / 67,890 out
    Total cost:     $0.47
    
    Per-task breakdown:
      ✓ TASK-1  Set up schema         45s    $0.05
      ✓ TASK-2  Create User model     32s    $0.04
      ✗ TASK-3  Email validation      —      (timeout)
      ✓ TASK-4  REST endpoints        68s    $0.12
      ...
    ═══════════════════════════════════════════════
    
  4. Extend SpecTaskRunnerListener:

    default void onRunFinished(BatchRunSummary summary) {
        onRunFinished(summary.getCompletedTasks(), summary.getSkippedTasks(),
                      summary.getTotalTasks(), summary.getFinalState());
    }

Files to Create / Modify

  • src/main/java/com/devoxx/genie/model/spec/BatchRunSummary.java (new)
  • src/main/java/com/devoxx/genie/service/spec/SpecTaskRunnerService.java
  • src/main/java/com/devoxx/genie/service/spec/SpecTaskRunnerListener.java
  • src/main/java/com/devoxx/genie/service/cli/CliConsoleManager.java

Dependencies

Acceptance Criteria

  • BatchRunSummary aggregates all per-task metrics
  • Console prints formatted summary at end of every batch run
  • Per-task breakdown shows status, timing, and cost
  • Listener receives full summary (backward compatible default method)
  • Summary printed for both CLI and LLM mode runs

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions