LLM_CONTEXT_WINDOW.md

LLM Context Window Configuration

Overview

The Analify platform now features an enhanced LLM context window for the analytics assistant, providing:

• Larger context capacity: 8192 tokens (doubled from default 4096)
• Conversation history tracking: Maintains context across multiple questions
• Extended model persistence: 30-minute keep-alive for faster responses
• GPU acceleration: RTX 4060 GPU support for improved inference speed

Configuration

Ollama Settings

Located in backAnalify/src/main/resources/application.properties:

# Spring AI Configuration (Ollama - Local LLM)
spring.ai.ollama.base-url=http://localhost:11434
spring.ai.ollama.chat.model=llama3.2:3b

# Model Configuration
spring.ai.ollama.chat.options.temperature=0.3
# Context window: 8192 tokens for larger conversation history and analytics data
spring.ai.ollama.chat.options.num-ctx=8192
# Keep model in memory for 30 minutes (faster response times)
spring.ai.ollama.chat.options.keep-alive=30m

GPU Support

For optimal performance, Ollama should run with GPU acceleration:

# Start Ollama with sudo for GPU access
sudo ollama serve

# Verify GPU is detected
ollama ps
# Should show: PROCESSOR: 100% GPU (not CPU)

API Usage

Endpoint

POST /api/assistant/analytics/query

Request Format

Simple Query (No History)

{
  "question": "What is my total revenue this month?"
}

Query with Conversation History

{
  "question": "How does that compare to last month?",
  "conversationHistory": [
    {
      "role": "user",
      "content": "What is my total revenue this month?"
    },
    {
      "role": "assistant",
      "content": "Your total revenue for this month is $45,230. This includes sales from all your stores."
    }
  ]
}

Response Format

{
  "answer": "Compared to last month, your revenue increased by 12.5%. Last month's revenue was $40,200, and this month is $45,230, showing a growth of $5,030.",
  "metadata": {
    "role": "INVESTOR",
    "usedEnhancedDashboard": true,
    "usedBasicDashboard": true,
    "lowStockAlertsCount": 3
  }
}

Context Window Benefits

1. Larger Context Capacity (8192 tokens)

• Handles extensive analytics data summaries
• Processes multiple dashboard metrics simultaneously
• Accommodates detailed conversation history
• Supports complex multi-part questions

Token Estimation:

• Average question: ~50-100 tokens
• Analytics context: ~1000-2000 tokens
• Conversation history (5 exchanges): ~500-1000 tokens
• Total usage: typically 1500-3000 tokens (well within 8192 limit)

2. Conversation History Tracking

• Maintains context from previous questions
• Enables follow-up questions without repeating context
• Supports natural conversation flow
• Stores last 5 message exchanges automatically

Example Conversation:

User: "Show me my top-selling products"
Assistant: "Your top products are: 1) Product A ($12,500), 2) Product B ($9,800)..."

User: "What about their profit margins?" (uses context from previous answer)
Assistant: "Based on the products we just discussed: Product A has a 35% margin..."

3. Extended Model Persistence (30 minutes)

• Model stays loaded in GPU memory for 30 minutes after last request
• First request: ~2-3 seconds (model loading)
• Subsequent requests: ~200-500ms (GPU inference only)
• Significantly improves user experience for active sessions

4. GPU Acceleration

• RTX 4060 Laptop GPU: 8GB VRAM, CUDA Compute 8.9
• Inference speed: ~50-100 tokens/second (vs ~10-20 on CPU)
• Memory: Model uses ~2.8GB VRAM, leaves ~5GB for context
• Concurrent requests: Supports 2-3 simultaneous queries

Frontend Integration

Basic Query (TypeScript/React)

import { assistantApi } from '@/services/api';

const answer = await assistantApi.askQuestion("What is my revenue?");
console.log(answer.answer);

With Conversation History

interface ConversationMessage {
  role: 'user' | 'assistant';
  content: string;
}

const [history, setHistory] = useState<ConversationMessage[]>([]);

const askWithContext = async (question: string) => {
  // Send question with history
  const response = await fetch('/api/assistant/analytics/query', {
    method: 'POST',
    headers: {
      'Content-Type': 'application/json',
      'Authorization': `Bearer ${token}`
    },
    body: JSON.stringify({
      question,
      conversationHistory: history
    })
  });
  
  const { answer } = await response.json();
  
  // Update conversation history
  setHistory([
    ...history,
    { role: 'user', content: question },
    { role: 'assistant', content: answer }
  ]);
  
  return answer;
};

Performance Metrics

Token Usage

Scenario	Tokens Used	% of Context
Simple question (no history)	~1,500	18%
With 3 message history	~2,200	27%
With 5 message history (max)	~3,000	37%
Complex analytics query	~4,500	55%

Response Times (with GPU)

Operation	Cold Start	Warm (cached)
Model loading	2-3 seconds	N/A
Simple query	0.5-1 second	0.2-0.5 seconds
With history	1-2 seconds	0.5-1 second
Complex query	2-3 seconds	1-2 seconds

Monitoring

Check Context Window Configuration

# View current Ollama model settings
curl -s http://localhost:11434/api/show -d '{"name":"llama3.2:3b"}' | jq '.model_info.num_ctx'
# Should return: 8192

Monitor GPU Usage

# Real-time GPU monitoring
nvidia-smi -l 1

# Check Ollama GPU usage
ollama ps
# Look for: PROCESSOR: 100% GPU

Backend Logs

# Check Spring AI configuration on startup
tail -f /tmp/backend_run.log | grep -i "ollama\|spring.ai"

# Monitor LLM requests
tail -f /tmp/backend_run.log | grep "AnalyticsAssistantService"

Troubleshooting

Issue: Context window not applied

Symptom: Model still uses default 4096 tokens

Solution:

# Restart backend to apply configuration
cd backAnalify
./mvnw spring-boot:run

# Verify in logs
grep "num-ctx" /tmp/backend_run.log

Issue: GPU not detected

Symptom: ollama ps shows "100% CPU" instead of "100% GPU"

Solution:

# Start Ollama with sudo for GPU permissions
sudo pkill ollama
sudo ollama serve &

# Verify GPU detection
ollama ps  # Should show GPU
nvidia-smi  # Should show ollama process using GPU

Issue: Model not staying in memory

Symptom: Every request takes 2-3 seconds (model reloading)

Solution:

# Verify keep-alive setting
grep "keep-alive" backAnalify/src/main/resources/application.properties
# Should show: 30m

# Check Ollama process
ps aux | grep ollama
# Should be running continuously

Issue: Out of memory errors

Symptom: "Failed to allocate memory" or slow responses

Solution:

# Check available VRAM
nvidia-smi

# If low, reduce context window
# Edit application.properties:
spring.ai.ollama.chat.options.num-ctx=4096

# Or close other GPU applications

Best Practices

1. Conversation History Management

• Store only last 5 exchanges (already implemented)
• Clear history for new topics
• Don't send sensitive data in history

2. Token Optimization

• Analytics context is already summarized (1000-2000 tokens)
• Avoid sending raw database dumps
• Use concise questions for better responses

3. Caching Strategy

• Keep Ollama running continuously
• Use 30-minute keep-alive for active hours
• Consider shorter keep-alive (5m) during off-hours to save memory

4. GPU Resource Management

• Monitor VRAM usage with nvidia-smi
• Allow 2-3GB buffer for context processing
• Limit concurrent requests to 2-3 for optimal performance

Future Enhancements

Potential Improvements

1. RAG (Retrieval-Augmented Generation)

   • Index historical analytics data
   • Retrieve relevant past insights
   • Enhance context with business knowledge base

2. Streaming Responses

   • Real-time token streaming
   • Progressive answer display
   • Better UX for long responses

3. Multi-turn Planning

   • Break complex questions into sub-queries
   • Execute multiple analytics operations
   • Synthesize comprehensive answers

4. Context Pruning

   • Intelligently summarize old conversation turns
   • Keep most relevant parts of history
   • Extend practical conversation length

References

• Ollama API Documentation
• Spring AI Reference
• LLaMA 3.2 Model Card
• NVIDIA CUDA Documentation