[FEATURE] Implement Example Question Resolver & Incremental Streaming Support

[AndlerRL]

Context:
This issue is created based on this comment by @AndlerRL regarding a new approach for handling example questions and streamlining the user experience when starting conversations with pre-selected questions.

---

Description

We need to restructure how we handle example questions and how we serve answers for them when the user does not modify the question. The main idea is:

• Pre-load 16–24 example questions per bot (50% deep expertise, 50% regular) into constants to avoid unnecessary chain requests on load.

• When the user selects an example question without modifying it, the backend should detect that and serve the answer immediately, skipping the standard AI processing flow.

• If feasible, reuse the same api/chat route but adapt its logic to:

  • Detect these pre-defined example question IDs or content hash.
  • Return pre-stored answers instantly or through a custom stream resolver.
  • Optionally build this using Vercel AI SDK and Next.js routes to improve UI responsiveness.

---

Tasks

Immediate Tasks (Simplified flow for this PR)

• Grab the selected example question (even if slightly modified).
• Start a conversation using that question via the regular api/chat route.
• Continue the usual flow for now (skip optimization).

Future Tasks (Create new PR later)

• Research feasibility of using Vercel AI SDK to:

  • Override api/chat with a custom stream response.
  • Render UI incrementally as response is streamed.

• Design a stream resolver that can detect when an example question is selected.

• Serve pre-defined answers quickly from constant object/cache if question is recognized.

• Document structure of pre-selected constant questions and their associated metadata (e.g., deepExpertise flag, score, etc).

---

Technical Considerations

• Ensure that a custom api/chat resolver can support:

  • Distinction between real user input vs selected template questions.
  • Return a mock/constant answer fast.

• Might need a dedicated streamer function to reflect real-time changes in UI.

• Use ICL (In-Context Learning) examples based on scoring logic (e.g., top 50%).

---

Follow-up

This task can remain unassigned until the base infrastructure decision is made (Vercel AI SDK vs custom route middleware). Meanwhile, implement simplified logic in current PR.

[AndlerRL]

@coderabbitai can you verify if it is better to use Vercel AI SDK (if feasible too) or create a custom stream to return our data instead?

[coderabbitai]

Tip

For best results, initiate chat on the files or code changes.

🧩 Analysis chain

I'll help you analyze whether to use Vercel AI SDK or custom streaming for your data. Let me first examine the GitHub issue and current implementation to understand the context.

---

🏁 Script executed:

#!/bin/bash
# First, let's understand what GitHub issue #524 is about
gh issue view 524 --repo bitcashorg/masterbots

# Look for any current streaming implementations in API routes
fd -e ts -e js "route" apps/masterbots.ai/app/api/ --exec cat {}

# Search for existing streaming patterns
rg -A 10 -B 5 "stream|Stream" apps/masterbots.ai/app/api/

Length of output: 41495

---

Let me research the Vercel AI SDK streaming capabilities to provide you with a comprehensive comparison for your specific use case.

---

🏁 Script executed:

#!/bin/bash
# Let's examine the current streaming implementations more closely
echo "=== Current createResponseStream function ==="
fd -e ts -e js "actions" apps/masterbots.ai/app/ --exec rg -l "createResponseStream" {}

# Look for the actual implementation
rg -A 20 -B 5 "createResponseStream" apps/masterbots.ai/app/

# Check the ai-streams helper
echo "=== AI Streams Helper ==="
cat apps/masterbots.ai/lib/helpers/ai-streams.ts

# Look for any existing AI SDK usage patterns
echo "=== Existing AI SDK Usage ==="
rg -A 10 -B 5 "@ai-sdk|ai-sdk" apps/masterbots.ai/

Length of output: 31058

---

🌐 Web query:

Vercel AI SDK streaming capabilities 2024 custom stream responses Next.js API routes

💡 Result:

The Vercel AI SDK in 2024 offers robust and flexible streaming capabilities for generating custom stream responses in Next.js API routes, going well beyond basic text streaming. Here’s how its key streaming features work and how you can implement custom streaming logic:

1. Rich Streaming Protocol and UI Message Structure

• Vercel AI SDK v5 introduced a new streaming protocol, designed to efficiently handle rich, structured, multi-part responses (UI messages) that go beyond simple string outputs.
• A single UIMessage can consist of multiple parts (e.g., TextUIPart, ToolInvocationUIPart, FileUIPart, SourceUIPart, and others), allowing granular and dynamic updates in the UI as data streams from the backend[1].

2. Streaming Standard LLM Responses

• The most common pattern is streaming text from LLMs as soon as chunks are received, to provide a responsive chat or generation experience.
• Example code using streamText in a Next.js API route:

import { streamText } from 'ai'
import { openai } from '@ai-sdk/openai'

export async function GET() {
  const response = streamText({
    model: openai('gpt-4o-mini'),
    messages: [{ role: 'user', content: 'Say this is a test.' }],
  })

  return response.toTextStreamResponse({
    headers: {
      'Content-Type': 'text/event-stream',
    },
  })
}

• This streams the model’s response token-by-token as SSE (Server-Sent Events) to the client[2][5].

3. Custom Streaming Data and Annotations

• For more advanced scenarios—such as RAG, citation streaming, or multi-source data—you can stream custom data or sources alongside LLM output.
• The SDK provides lower-level helpers like createDataStreamResponse and DataStreamWriter:
  • You can write custom messages, sources, or annotations to the response stream in real time.
  • Example pattern:

import { openai } from '@ai-sdk/openai'
import { createDataStreamResponse, streamText } from 'ai'

export async function POST(req) {
  const { messages } = await req.json();

  return createDataStreamResponse({
    execute: dataStream => {
      dataStream.writeData('stream initialized');

      const result = streamText({
        model: openai('gpt-4o'),
        messages,
        onChunk() {
          dataStream.writeMessageAnnotation({ chunk: 'example' });
        },
        onFinish() {
          dataStream.writeData('stream completed');
        }
      });

      result.mergeIntoDataStream(dataStream);
    },
    onError: err => err instanceof Error ? err.message : String(err)
  });
}

• This approach lets you send custom chunks (e.g., progress, citations, metadata, dynamic sources) live with the LLM response[8].

4. Custom AI Process Streaming (Advanced)

• For custom or non-standard streams, you can implement your own streaming logic and manually write to the stream writer.
• Key best practice: Do not await your handler if you want true streaming—let it write to the stream asynchronously, then return a Response with correct SSE headers:

handleCustomAIProcess(writer, input)
  .catch(err => {
    if (!writer.closed) {
      writer.writeError(err.message || "Unknown error");
      writer.close();
    }
  });

return new NextResponse(stream, {
  headers: {
    'Content-Type': 'text/event-stream; charset=utf-8',
    'x-vercel-ai-ui-message-stream': 'v1',
    'Cache-Control': 'no-cache',
    'Connection': 'keep-alive',
  }
});

• This ensures data streams live as events occur, and enables custom error handling[1].

5. Streaming Arbitrary Objects and Data

• The SDK supports streaming structured objects (not just text) that match a schema, using streamObject or similar utilities.
• This is useful for advanced interactive UIs needing structured data updates, like real-time dashboards or tool invocations[4][3].

6. Client Integration

• The AI SDK offers React hooks (useChat, useCompletion, useAssistant) that consume these streams seamlessly and update UI state as data arrives in real time[4].

7. Caveats and Troubleshooting

• Make sure your API routes use correct headers for SSE, and note that streaming issues are often due to client/network settings rather than the SDK itself[6].

Summary Table: Streaming Features in Vercel AI SDK (2024)

Feature/Pattern	Use Case	Key Method/Helper
Basic LLM text streaming	Chat, completions	streamText, .toTextStreamResponse
Structured object streaming	Advanced UIs, multi-part data	streamObject
Custom data, sources, annotations	RAG, citations, meta-info	createDataStreamResponse, DataStreamWriter
Custom background process streaming	Non-LLM, custom AI pipelines	Manual writer, NextResponse

The SDK is well-documented and routinely updates patterns for handling sophisticated streaming use cases in Next.js API routes, with strong support for both “quick start” and fully custom streaming solutions[1][2][4][8].

Citations:

• 1: https://dev.to/yigit-konur/vercel-ai-sdk-v5-internals-part-2-streaming-the-richness-inside-the-ui-message-protocol--2o34
• 2: https://vercel.com/guides/streaming-from-llm
• 3: How to use streamObject in a NextJS API route? vercel/ai#1962
• 4: https://www.telerik.com/blogs/practical-guide-using-vercel-ai-sdk-next-js-applications
• 5: https://vercel.com/docs/functions/streaming-functions
• 6: Streaming not working. vercel/ai#2009
• 7: https://sdk.vercel.ai/providers/ai-sdk-providers/google-generative-ai
• 8: https://ai-sdk.dev/docs/ai-sdk-ui/streaming-data
• 9: https://vercel.com/guides/fine-tuning-openai-nextjs
• 10: https://community.vercel.com/t/ai-vercel-sdk-rsc-streaming-for-the-assistants-api/1467

---

@AndlerRL Based on my analysis of issue #524 and your current codebase, I strongly recommend using Vercel AI SDK for your example question resolver and incremental streaming implementation. Here's why and how:

Recommendation: Vercel AI SDK

Why Vercel AI SDK is the better choice:

1. Architectural Consistency: Your codebase already extensively uses Vercel AI SDK with multiple providers and the existing createResponseStream function
2. Advanced Streaming Capabilities: AI SDK 2024 supports exactly what you need - custom data streaming with createDataStreamResponse
3. Maintenance: Single streaming paradigm reduces complexity vs. maintaining both AI SDK and custom streaming implementations

Implementation Approach

For your example question resolver, you can enhance your existing /api/chat route using AI SDK's custom streaming:

import { createDataStreamResponse, streamText } from 'ai'
import { openai } from '@ai-sdk/openai'

export async function POST(req: Request) {
  const { messages, exampleQuestionId } = await req.json()

  return createDataStreamResponse({
    execute: async (dataStream) => {
      // Step 1: Check if this is a pre-defined example question
      const preDefinedAnswer = await checkExampleQuestion(exampleQuestionId, messages)
      
      if (preDefinedAnswer) {
        // Step 2: Stream pre-stored answer immediately
        dataStream.writeData('example-question-detected')
        
        // Simulate incremental streaming of pre-stored content
        const chunks = preDefinedAnswer.split(' ')
        for (const chunk of chunks) {
          dataStream.writeMessageAnnotation({ 
            type: 'chunk', 
            content: chunk + ' ',
            source: 'example-repository' 
          })
          await new Promise(resolve => setTimeout(resolve, 50)) // Realistic streaming delay
        }
        
        dataStream.writeData('example-answer-complete')
      } else {
        // Step 3: Fall back to normal AI processing
        const result = await streamText({
          model: openai('gpt-4'),
          messages,
          onChunk() {
            dataStream.writeMessageAnnotation({ source: 'ai-generated' })
          }
        })
        
        result.mergeIntoDataStream(dataStream)
      }
    },
    onError: (error) => error.message
  })
}

Key Advantages Over Custom Streaming

1. Rich Data Types: Stream metadata, sources, and annotations alongside content
2. Client Integration: Seamless integration with your existing useMBChat hook using useChat
3. Error Handling: Built-in error handling and recovery mechanisms
4. Performance: Optimized streaming protocol designed for real-time UI updates

Migration Path

1. Phase 1: Extend your existing createResponseStream function to detect example questions
2. Phase 2: Use createDataStreamResponse for custom streaming logic
3. Phase 3: Maintain your current Wordware custom streaming (it's working well for that specific use case)

This approach leverages your existing AI SDK infrastructure while adding the custom streaming capabilities you need for the example question feature. The consistency with your current architecture will make maintenance much easier long-term.

Would you like me to provide more specific implementation details for any part of this approach?

🐰✨
thoughtfully considering the architectural implications

---

🧠 Learnings used

Learnt from: Bran18
PR: bitcashorg/masterbots#504
File: apps/masterbots.ai/app/api/generate-images/route.ts:58-80
Timestamp: 2025-06-10T05:48:25.508Z
Learning: In apps/masterbots.ai/app/api/generate-images/route.ts, the conditional check for modelId === 'gpt-image-1' is correct and uses the OpenAI client directly to handle this advanced model, while other models like 'dall-e-2' and 'dall-e-3' use the AI SDK. This implementation pattern is appropriate for handling different model capabilities.