Performance cost optimization

[tekrajchhetri]

Summary

This PR introduces several major improvements to the pipeline:

Changes

Faster Pipeline Execution

• Fixes the issue of tasks taking excessively long to complete (previously 6+ hours in some cases)
• Reduces token usage and overall cost
• The full pipeline now completes in 5–10 minutes for a 35+ page document (based on testing)
• Longer documents may still require additional processing time

Staged Output Persistence

• Adds support for saving outputs at each stage (e.g., extraction, alignment)
• Previously, failures (e.g., invalid OpenRouter key or insufficient credit) required restarting from scratch
• Now allows reuse of completed stages, avoiding redundant work and saving time

Selective Agent/Task Execution

• Introduces the ability to skip specific agents or tasks
• Enables running only required stages (e.g., extraction + alignment) without invoking others like judge or human feedback agents

LLM Call Logging

• Logs all LLM calls made:
  • Direct calls
  • Calls via CrewAI

[tekrajchhetri]

Merging based on discussion over slack.