Goal
Analyze the AI analysis pipeline for latency, throughput, and cost opportunities before changing chunking or concurrency behavior.
Why this matters
The current analysis path appears to process oversized inputs in serial chunks and then run a combine pass, which likely makes AI-backed features one of the most expensive backend paths.
Scope
Investigate AI analysis behavior used by direct analysis endpoints and report-generation flows.
Primary questions
- How much latency is spent in Azure OpenAI calls versus local prep work?
- What is the current chunking strategy doing to total request time?
- Where are we paying for duplicate or unnecessary synthesis passes?
- Can chunk analysis be parallelized safely with bounded concurrency?
- Can we reuse or cache intermediate summaries for repeated report requests?
- What rate-limit, token, and cost controls would be required before parallelizing?
Relevant code areas
- shareddump\AI\FeedbackAnalyzerService.cs
- shareddump\AI\IFeedbackAnalyzerService.cs
- feedbackfunctions\FeedbackAnalysis\FeedbackFunctions.cs
- feedbackfunctions\Utils\ReportGenerator.cs
Investigation tasks
- Trace the end-to-end call path for AnalyzeComments and report-level analysis.
- Measure single-pass versus chunked analysis behavior.
- Review chunk sizing, prompt size, and combine-pass behavior.
- Compare streaming and non-streaming code paths for duplicated work or missed reuse.
- Evaluate bounded concurrency patterns and failure handling options.
- Evaluate memoization or caching opportunities for repeated inputs or report-level summaries.
Deliverables
- A performance profile of the AI pipeline.
- A risk/benefit analysis for bounded parallel chunk processing.
- A shortlist of low-risk and high-impact improvements.
- Clear guardrails for any future AI parallelism or caching work.
Acceptance criteria
- We understand the main latency and cost drivers in AI-backed backend flows.
- We have a concrete recommendation for what to change first, and what to avoid.
Goal
Analyze the AI analysis pipeline for latency, throughput, and cost opportunities before changing chunking or concurrency behavior.
Why this matters
The current analysis path appears to process oversized inputs in serial chunks and then run a combine pass, which likely makes AI-backed features one of the most expensive backend paths.
Scope
Investigate AI analysis behavior used by direct analysis endpoints and report-generation flows.
Primary questions
Relevant code areas
Investigation tasks
Deliverables
Acceptance criteria