Skip to content

Commit 8c76817

Browse files
gobrandoclaude
gobrando
and
claude
committed
docs: Root cause analysis of web search spike from 8% to 43%
Investigates and documents why web search invocation increased 5.6x. Three reinforcing changes identified: model switch to gpt-5.1/reasoning=none (primary), Keystone PA region with no RAG resources (amplifier), and RAG pipeline replacing full DB injection (contributor). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
1 parent a63e0cd commit 8c76817

1 file changed

Lines changed: 153 additions & 0 deletions

File tree

‎app/WEB_SEARCH_SPIKE_ROOT_CAUSE_ANALYSIS.md‎

Lines changed: 153 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,153 @@
1+
# Web Search Spike Root Cause Analysis
2+
3+
**Date:** February 20, 2026
4+
**Branch:** performance-testing
5+
**Environment:** Production (pilot-prod)
6+
7+
## Summary
8+
9+
Web search invocation in production increased from **8% historically to 43% recently** — a 5.6x increase. This report identifies the root causes by correlating the spike with specific code changes deployed between December 2025 and January 2026.
10+
11+
**The spike is caused by three reinforcing changes, not a single event:**
12+
13+
| Rank | Change | Date | Impact |
14+
|------|--------|------|--------|
15+
| 1 | Model switched to `gpt-5.1` with `reasoning="none"` | Jan 5, 2026 | **Primary cause** — this model invokes web search ~100% of the time |
16+
| 2 | Keystone PA region added with no RAG resources | Jan 15-20, 2026 | **Major amplifier** — every Keystone query forces web search |
17+
| 3 | RAG pipeline replaced full DB injection | Dec 5-8, 2025 | **Contributing factor** — fewer resources in context |
18+
19+
## Detailed Timeline
20+
21+
### Phase 1: Baseline (Oct 30 – Dec 4, 2025) — ~8% web search rate
22+
23+
Web search was added on Oct 30 (`8e2e214`) with the `OpenAIWebSearchGenerator` component. The generate_referrals pipeline used:
24+
25+
- **Model:** `gpt-5-mini` with `reasoning_effort="low"`
26+
- **Resources:** All supports loaded from PostgreSQL via `LoadSupports()` and injected directly into the prompt
27+
- **Web search:** Available but rarely used — the model had full resource context and was conservative about tool use
28+
29+
This is the period reflected in the **8% historical rate** (447/5,554 traces).
30+
31+
### Phase 2: RAG Introduction (Dec 5-8, 2025) — Moderate increase
32+
33+
| Commit | Date | Change |
34+
|--------|------|--------|
35+
| `c6623cb` | Dec 5 | Added `generate_referrals_rag` endpoint with ChromaDB vector retrieval (`top_k=10`) |
36+
| `d976d51` | Dec 8 | Frontend switched to use RAG endpoint |
37+
38+
**Why this increased web search:**
39+
- The old pipeline dumped **all** resources from PostgreSQL into the prompt. The RAG pipeline retrieves only the **top 10** most relevant documents from ChromaDB.
40+
- Fewer resources in context means the LLM has less to work with and is more likely to supplement via web search.
41+
- The Central TX prompt explicitly encourages this: *"Supplement with trusted resources you can find through web search."*
42+
43+
### Phase 3: Model Switch to gpt-5.1 (Jan 5, 2026) — PRIMARY CAUSE
44+
45+
| Commit | Date | Change |
46+
|--------|------|--------|
47+
| `d1b7388` | Jan 5 | All models centralized to `gpt-5.1` / `reasoning="none"` |
48+
49+
**Before:**
50+
```
51+
Non-RAG pipeline: gpt-5-mini / reasoning="low"
52+
RAG pipeline: gpt-5 / reasoning="high" (component defaults)
53+
```
54+
55+
**After:**
56+
```
57+
All pipelines: gpt-5.1 / reasoning="none"
58+
```
59+
60+
**Why this is the biggest factor:**
61+
62+
Benchmark testing (`reasoning_level_comparison_results.json`) shows `gpt-5.1` with `reasoning="none"` has a **100% web search invocation rate** (30/30 tests). The previous `gpt-5-mini` with `reasoning="low"` was far less aggressive about invoking the web search tool.
63+
64+
The combination of a more capable model (`gpt-5.1`) and disabling reasoning (`"none"`) fundamentally changed how the LLM interacts with the web search tool. Without reasoning overhead, the model defaults to using every tool available to it, including web search.
65+
66+
### Phase 4: Keystone PA Region (Jan 14-20, 2026) — MAJOR AMPLIFIER
67+
68+
| Commit | Date | Change |
69+
|--------|------|--------|
70+
| `e0c3541` | Jan 14 | Added prompt suffix handling |
71+
| `1e8b9aa` | Jan 15 | Frontend added Keystone PA location suffix |
72+
| `e581e25` | Jan 20 | Keystone prompt version updated |
73+
| `e9cbcfd` | Jan 21 | RAG docs distinguished by region |
74+
75+
**Why this amplified the spike:**
76+
77+
The Keystone prompt has **no RAG resources section** — no `{% for s in supports %}` block. Instead, it explicitly instructs: *"Your response should include resources you find searching the web."*
78+
79+
This means **100% of Keystone queries force web search**, regardless of model behavior. As Keystone traffic grew after launch on Jan 15, the aggregate web search rate climbed proportionally.
80+
81+
### Phase 5: Cleanup & Temperature (Jan 23 – Feb 5, 2026)
82+
83+
| Commit | Date | Change |
84+
|--------|------|--------|
85+
| `003669f` | Jan 23 | Removed `Support`, `SupportListing`, `CrawlJob`, and other non-RAG code |
86+
| `548929e` | Jan 26 | Removed old `generate_referrals` pipeline entirely |
87+
| `8a8bf66` | Jan 26 | Hardcoded `suffix="centraltx"` as default |
88+
| `540481a` | Feb 5 | Added temperature setting (`0.9` in production) |
89+
90+
These changes cemented the new architecture. The high temperature (0.9) may marginally influence the model's tendency to invoke web search, but this is not a primary driver.
91+
92+
### Phase 6: Improved Tracing (Feb 9, 2026)
93+
94+
| Commit | Date | Change |
95+
|--------|------|--------|
96+
| `b10c02a` | Feb 9 | Logged web search source URLs and context in Phoenix traces |
97+
98+
This improved **visibility** of web search calls in traces. Some of the measured increase in web search rate may be attributable to better detection, not just more actual usage.
99+
100+
## Current Production Configuration
101+
102+
```python
103+
# app/src/app_config.py (origin/main)
104+
generate_referrals_rag_model_version: str = "gpt-5.1"
105+
generate_referrals_rag_reasoning_level: str = "none"
106+
generate_referrals_rag_temperature: float = 0.9
107+
```
108+
109+
```python
110+
# app/src/common/components.py — web search is always enabled
111+
"tools": [{"type": "web_search"}]
112+
```
113+
114+
| Region | RAG Resources | Web Search Instruction | Expected Web Search Rate |
115+
|--------|--------------|----------------------|--------------------------|
116+
| Central TX | Yes (top_k=10 from ChromaDB) | "Supplement with trusted resources you can find through web search" | High (model + prompt both encourage it) |
117+
| Keystone PA | None | "Your response should include resources you find searching the web" | ~100% (no other resource source) |
118+
119+
## Benchmark Evidence
120+
121+
From `reasoning_level_comparison_results.json` (30 test prompts):
122+
123+
| Model | Reasoning | Web Search Rate |
124+
|-------|-----------|----------------|
125+
| `gpt-5.1` | `none` | **100%** (30/30) |
126+
| `gpt-5.1` | `low` | 0% (all failed) |
127+
128+
From `PRODUCTION_WEBSEARCH_ANALYSIS.md`:
129+
130+
| Period | Web Search Rate | Traces |
131+
|--------|----------------|--------|
132+
| Historical (all time) | 8.0% | 447 / 5,554 |
133+
| Recent (last 1,000 spans) | 45.0% | 50 / 111 |
134+
135+
## Conclusions
136+
137+
1. **The model change is the primary cause.** Switching from `gpt-5-mini`/`reasoning="low"` to `gpt-5.1`/`reasoning="none"` on Jan 5 fundamentally changed web search behavior. The new model is dramatically more aggressive about using web search when it's available.
138+
139+
2. **The Keystone region is the secondary cause.** Its prompt has no RAG resources and explicitly requests web search, guaranteeing 100% web search usage for all Keystone traffic since Jan 15.
140+
141+
3. **The RAG migration is a contributing factor.** Providing top-k=10 vector results instead of all DB resources gives the LLM less context, making it more likely to supplement via web search.
142+
143+
4. **This is not a bug.** All three changes were intentional. The higher web search rate is a natural consequence of the new architecture. Whether this is desirable depends on whether the web search results improve output quality and whether the added latency is acceptable.
144+
145+
## Recommendations
146+
147+
1. **If web search rate needs to decrease:** Test `gpt-5.1` with `reasoning="low"` or `reasoning="medium"` — reasoning effort appears to be the strongest lever for controlling web search behavior.
148+
149+
2. **If Keystone web search rate needs to decrease:** Add RAG resources for the Keystone PA region so the prompt can include local resources alongside web search.
150+
151+
3. **Monitor quality:** Compare output quality for traces with and without web search to determine if the higher rate is beneficial.
152+
153+
4. **Consider prompt tuning:** The Central TX prompt's instruction to *"Supplement with trusted resources"* could be softened to *"Optionally supplement"* if web search rate is too high.

0 commit comments

Comments
 (0)