Skip to content

Add summary logging with progress counters and aggregation #57

@CybotTM

Description

@CybotTM

Problem

The migration process generates ~31M log lines for 68K work packages (~450 lines per WP). There's no periodic progress reporting or final summary aggregation.

Current State

  • Individual INFO messages for every operation
  • 155K+ warnings logged individually (e.g., BUG32 user not found)
  • No periodic progress counters (e.g., "Completed 1000/68741 WPs")
  • No final summary of warnings/errors

Requirements

  1. Periodic progress logging (every N work packages, e.g., 1000):

    [INFO] Migration progress: 1000/68741 WPs (1.5%) - 10.2 WPs/sec
    
  2. Warning aggregation - instead of 155K individual warnings:

    [INFO] Warning summary: 155182 user lookups used fallback (top: caroline.kuhn=24627, gitlab-ci=24084, Anonymous=17759)
    
  3. Final migration summary:

    [SUCCESS] Migration complete:
    - Work packages: 68741 created, 0 failed
    - Journals: 850000 created
    - Warnings: 155182 (user fallbacks)
    - Errors: 2 (non-critical)
    - Duration: 11.3 hours (101 WPs/min)
    

Related

  • ADR analysis identified excessive log verbosity
  • 31M lines for 68K WPs is not sustainable for observability

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions