集成 AWS Bedrock 作为新的 LLM 服务提供商 #6430
Conversation
Adds support for using models hosted on AWS Bedrock, specifically Anthropic Claude models. Key changes: - Added '@aws-sdk/client-bedrock-runtime' dependency. - Updated constants, server config, and auth logic for Bedrock. - Implemented backend API handler () to communicate with the Bedrock API, handling streaming and non-streaming responses, and formatting output to be OpenAI compatible. - Updated dynamic API router () to dispatch requests to the Bedrock handler. - Created frontend client () and updated client factory (). - Updated with necessary Bedrock environment variables (AWS keys, region, enable flag) and an example for using to alias Bedrock models.
- Added logging to `getClientApi` for better debugging of provider input and standardized provider names. - Updated `ChatActions` to handle lowercase provider IDs, converting them to TitleCase for consistency with the ServiceProvider enum. - Implemented defaulting to OpenAI when provider ID is missing and added relevant logging for session updates. - Enhanced logging in `useChatStore` to track API call preparations and provider configurations.
- Enhanced error messages in `getAwsCredentials` to provide clearer feedback when AWS Bedrock is not configured properly. - Added checks for missing Bedrock Access Key ID and Secret Access Key, with specific error messages for each case.
- Updated logging in the Bedrock API handler to include detailed information about the request model, stream status, and message count for improved debugging and monitoring.
|
Someone is attempting to deploy a commit to the NextChat Team on Vercel. A member of the Team first needs to authorize it. |
WalkthroughThis pull request integrates AWS Bedrock support into the application. Configuration changes in the Changes
Sequence Diagram(s)sequenceDiagram
participant C as Client
participant AP as API Route Handler
participant BH as Bedrock Handler
participant AWS as AWS Bedrock Service
participant CL as BedrockApi (Client)
C->>AP: Send request to Bedrock endpoint
AP->>BH: Forward request to bedrockHandler
BH->>AWS: Invoke model command (streaming or non-streaming)
AWS-->>BH: Return response (or stream data)
BH-->>AP: Format and relay response
AP-->>C: Deliver final response
Poem
✨ Finishing Touches
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
🧹 Nitpick comments (11)
app/store/chat.ts (1)
459-478: Enhanced API call logging for debugging Bedrock integration.The added logging helps debug provider resolution, particularly checking if the provider is Bedrock. This is valuable during integration but consider cleaning up excessive logging in production code.
Consider:
- Using structured logging instead of multiple concatenated values
- Removing or conditionalizing verbose logging in production
- Standardizing language (mixing Chinese comments with English logs may confuse non-Chinese speakers)
const providerNameFromConfig = modelConfig.providerName; -console.log( - "[onUserInput] Preparing API call. Provider from config:", - providerNameFromConfig, - "| Type:", - typeof providerNameFromConfig, - "| Is Enum value (Bedrock)?:", - providerNameFromConfig === ServiceProvider.Bedrock, // 与枚举比较 - "| Is 'Bedrock' string?:", - providerNameFromConfig === "Bedrock", // 与字符串比较 - "| Model:", - modelConfig.model, -); +if (process.env.NODE_ENV !== 'production') { + console.log("[onUserInput] API call details:", { + provider: providerNameFromConfig, + type: typeof providerNameFromConfig, + isBedrockEnum: providerNameFromConfig === ServiceProvider.Bedrock, + isBedrockString: providerNameFromConfig === "Bedrock", + model: modelConfig.model, + }); +}app/components/chat.tsx (1)
696-709: Consider configurable log levels.
Lines 697-709 add multiple console logs that may be too verbose for production environments. Consider using a logging framework or adjusting log verbosity levels to avoid clutter.app/api/auth.ts (1)
117-120: Evaluate sensitive logging.
Displaying the provider name in logs may be helpful for debugging but can be unnecessary in multi-tenant or production setups. Assess whether to remove or downgrade this log to reduce potential noise or confusion.app/api/bedrock/index.ts (5)
14-32: Add fallback or improved error messages.
Presently,getAwsCredentials()throws generic errors if Bedrock is misconfigured. For a more user-friendly experience, consider including instructions or next steps in the error messages to help resolve the missing or incorrect environment variables.
34-58: Streamline subpath authorization logic.
Currently, only one path fromALLOWED_PATHis authorized. If future expansions include multiple Bedrock endpoints, consider using a more scalable approach (e.g., partially matching known subpath patterns) to reduce manual maintenance of theALLOWED_PATHset.
106-127: Plan for broader model support.
Currently, only Anthropic Claude-based models are supported. If you plan to integrate other Bedrock models (e.g., Amazon Titan), building a more generic payload construction mechanism here will reduce refactoring later.
207-248: Return "finish_reason" carefully.
The non-streaming path always uses"stop"forfinish_reason. In some edge cases, the model may reach a max token limit or encounter an error mid-generation. Differentiate these scenarios for a more accurate representation of why the generation ended.
250-253: Log details on errors.
While the error is returned to the client, consider logging additional context (e.g., command parameters, partial responses) for troubleshooting, ensuring sensitive data (like AWS keys) is excluded.app/config/server.ts (1)
177-177: Consider grouping Bedrock config under a single object.Currently, you return separate properties for
bedrockRegion,bedrockAccessKeyId,bedrockSecretAccessKey, etc. Grouping them into a nested object (e.g.,bedrockConfig) within the returned config may improve clarity and detect misconfigurations more easily.export const getServerSideConfig = () => { ... + const bedrockConfig = { + isEnabled: isBedrock, + region: process.env.AWS_REGION, + accessKeyId: process.env.AWS_ACCESS_KEY_ID, + secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY, + endpoint: process.env.AWS_BEDROCK_ENDPOINT, + }; const config = { ... - isBedrock, - bedrockRegion: process.env.AWS_REGION, - bedrockAccessKeyId: process.env.AWS_ACCESS_KEY_ID, - bedrockSecretAccessKey: process.env.AWS_SECRET_ACCESS_KEY, - bedrockEndpoint: process.env.AWS_BEDROCK_ENDPOINT, + bedrockConfig, ... }; return config; };Also applies to: 248-253
app/client/platforms/bedrock.ts (2)
16-120: Check streaming chunk parsing reliability.
- Splitting lines by “\n” could fail if partial JSON lines span multiple chunks. This might lead to JSON parse errors or data loss.
- Consider buffering partial lines across chunks to handle SSE streaming more robustly.
Everything else, including comfort calls for
onUpdate,onFinish, andonError, looks consistent.+ // Potential partial line buffer let leftover = ""; while (true) { const { done, value } = await reader.read(); if (done) break; - const text = decoder.decode(value, { stream: true }); - const lines = text.split("\n"); + const chunk = leftover + decoder.decode(value, { stream: true }); + const lines = chunk.split("\n"); + leftover = lines.pop() ?? ""; // store incomplete line for (const line of lines) { ... } }
136-140: No direct TTS support.Throwing an error in
speechis reasonable if AWS Bedrock TTS is not yet implemented. Consider returning a user-friendly error message or a specialized exception if you decide to add partial TTS handling later.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
⛔ Files ignored due to path filters (1)
yarn.lockis excluded by!**/yarn.lock,!**/*.lock
📒 Files selected for processing (11)
.env.template(2 hunks)app/api/[provider]/[...path]/route.ts(2 hunks)app/api/auth.ts(2 hunks)app/api/bedrock/index.ts(1 hunks)app/client/api.ts(4 hunks)app/client/platforms/bedrock.ts(1 hunks)app/components/chat.tsx(1 hunks)app/config/server.ts(3 hunks)app/constant.ts(4 hunks)app/store/chat.ts(1 hunks)package.json(1 hunks)
🧰 Additional context used
🧬 Code Definitions (4)
app/api/bedrock/index.ts (2)
app/config/server.ts (1)
getServerSideConfig(128-275)app/api/auth.ts (1)
auth(27-129)
app/store/chat.ts (1)
app/client/api.ts (2)
ClientApi(136-229)getClientApi(363-436)
app/client/api.ts (1)
app/client/platforms/bedrock.ts (1)
BedrockApi(8-140)
app/client/platforms/bedrock.ts (2)
app/client/api.ts (5)
ChatOptions(76-86)getHeaders(244-361)LLMUsage(88-91)LLMModel(93-99)SpeechOptions(67-74)app/constant.ts (1)
Bedrock(272-274)
🔇 Additional comments (18)
package.json (1)
24-24: Added AWS Bedrock SDK dependency.The addition of
@aws-sdk/client-bedrock-runtimeis necessary for integrating AWS Bedrock as a new LLM provider. This aligns with the PR objective of supporting AWS Bedrock-hosted models.app/api/[provider]/[...path]/route.ts (2)
17-17: Added Bedrock handler import.The import statement for the Bedrock handler follows the same pattern as other provider handlers. This is a clean implementation.
54-55: Added Bedrock API path routing.The case statement for
ApiPath.Bedrockcorrectly routes requests to the Bedrock handler, maintaining consistency with the existing API routing pattern.app/constant.ts (4)
75-75: Added Bedrock to ApiPath enum.The addition follows the consistent naming pattern used for other providers.
134-134: Added Bedrock to ServiceProvider enum.This is necessary for client-side provider identification and follows the established pattern.
161-161: Added Bedrock to ModelProvider enum.Properly integrates Bedrock into the model provider ecosystem for use in model selection logic.
272-274: Added Bedrock constant with ChatPath.The Bedrock constant correctly defines the API path for chat completions. Note that unlike some other providers (e.g., OpenAI), this only defines ChatPath without other capabilities like speech or image generation.
Is this intentional or are you planning to add additional capabilities for Bedrock in future updates? Some AWS Bedrock models support image generation and other multimodal capabilities that could be added later.
app/store/chat.ts (1)
475-478: Improved API client instantiation with fallback.The modification correctly handles cases where
providerNameFromConfigmight be undefined by providing a default value ofServiceProvider.OpenAI. This increases robustness.app/client/api.ts (3)
27-27: Use consistent import grouping.While this import correctly references the new
BedrockApi, ensure consistent import grouping or ordering if your coding guidelines require alphabetical or categorized ordering. Otherwise, this is fine.
177-179: Good extension for Bedrock provider.Adding a
ModelProvider.Bedrockcase is straightforward and consistent with the existing pattern. No issues detected.
363-434: Check for fallback correctness and future expansions.
- The conversion logic for string-based providers to
ServiceProviderenum is well-structured. Logging the standardized provider aids debugging.- The final
defaultcase falling back to GPT is appropriate, but ensure that your team is aligned on defaulting to GPT if an unknown provider string is received. This behavior might hide configuration errors.No immediate issues, but consider adding additional known lowercase mappings (e.g., "anthropic", "baidu") to unify your usage across the codebase.
Would you like me to scan for other occurrences of string-based providers in the repository to suggest more mappings?
app/config/server.ts (2)
167-171: Validate AWS credentials for Bedrock.The
isBedrocklogic correctly checks environment variables for AWS credentials. Ensure that you handle scenarios where credentials are invalid or missing at runtime, especially ifENABLE_AWS_BEDROCKis set to"true"but credentials are incorrect.Would you like a script to search for usage of these environment variables and confirm they are validated properly?
274-274: Clear separation of concerns.Wrapping the configuration in a
configobject prior to returning is a neat approach — it decouples environment variable loading from usage. Code looks good..env.template (2)
60-60: Verify custom models correctness.You have removed the
-alloption and added new models, including abedrock-based model namedsonnet. Ensure these custom tokens are recognized by the back end. Missing or mismatched model references could lead to unexpected errors.Please confirm that “us.anthropic.claude-3-5-sonnet-20241022-v2:0@bedrock=sonnet” is a valid internal or Bedrock model reference. If needed, I can provide a script to locate all references to “sonnet” in the codebase.
85-103: Follow best practices for secrets in templates.Defining placeholders for AWS credentials indicates good security hygiene. Be sure to remove or comment these lines out if you do not want new developers to accidentally commit real credentials.
app/client/platforms/bedrock.ts (3)
1-14: Ensure bedrock path constants are validated.The
pathmethod concatenatesApiPath.Bedrockwith a subpath. The final endpoint depends on your backend route definitions. Validate thatApiPath.Bedrockis correct and that backend routes match this prefix.
122-128: Usage reporting placeholder.Returning static usage data is acceptable as a placeholder. Ensure you revisit this method if future usage/billing metrics for Bedrock become relevant.
130-134: Model fetch is intentionally unimplemented.Acknowledging that dynamic model retrieval from Bedrock is complex, the empty array fallback is fine for now. No issues.
| case ModelProvider.Bedrock: | ||
| console.log( | ||
| "[Auth] Using AWS credentials for Bedrock, no API key override.", | ||
| ); | ||
| return { error: false }; | ||
| case ModelProvider.GPT: |
There was a problem hiding this comment.
🛠️ Refactor suggestion
Verify AWS credentials before returning.
This block immediately returns { error: false } for Bedrock, skipping an explicit credential validation step. Consider ensuring that AWS credential checks (e.g., presence of required environment variables) have definitively passed before returning success.
🧰 Tools
🪛 Biome (1.9.4)
[error] 104-104: Useless case clause.
because the default clause is present:
Unsafe fix: Remove the useless case.
(lint/complexity/noUselessSwitchCase)
- Modified the assignment of `providerName` in `ChatActions` to assert non-null, ensuring type safety and preventing potential runtime errors.
There was a problem hiding this comment.
Actionable comments posted: 0
♻️ Duplicate comments (1)
app/components/chat.tsx (1)
710-733: 🛠️ Refactor suggestionImproved provider ID handling with case normalization.
The code now properly handles the conversion from lowercase provider IDs (e.g., "bedrock") to TitleCase format required by the ServiceProvider enum. This is a good improvement that will help maintain consistency when new providers are added.
However, when an unknown provider ID is encountered, the code only logs an error without setting a fallback provider, which could lead to undefined behavior.
Consider adding a fallback to OpenAI for unknown providers, similar to how you handle missing providers:
} else { console.error( `[ChatActions] Unknown provider ID: ${providerId}`, ); - // Handle error or fallback if needed + // Fallback to OpenAI for unknown providers + targetProvider = ServiceProvider.OpenAI; + console.warn( + `[ChatActions] Unknown provider ID: ${providerId}, defaulting to OpenAI.`, + ); }
🧹 Nitpick comments (1)
app/components/chat.tsx (1)
740-750: Non-null assertion on targetProvider could be improved.Line 744 uses a non-null assertion (
targetProvider!) which may cause runtime errors if targetProvider is somehow null despite the enclosing if check.Consider a safer approach:
if (targetProvider) { // Only update if we found a valid provider chatStore.updateTargetSession(session, (session) => { session.mask.modelConfig.model = model as ModelType; - session.mask.modelConfig.providerName = targetProvider!; // Use the Enum value (Assert non-null) + session.mask.modelConfig.providerName = targetProvider; // Already checked in if condition session.mask.syncGlobalConfig = false;
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (1)
app/components/chat.tsx(1 hunks)
🔇 Additional comments (2)
app/components/chat.tsx (2)
697-708: Good addition of detailed logging for debugging provider selection.Adding logging for both raw and parsed values will help with troubleshooting model selection issues that may arise when implementing a new provider like Bedrock.
751-760: Improved provider-specific toast message display.The code now correctly displays different toast messages based on the provider type, which improves the user experience by showing more relevant information.
- Removed redundant Bedrock configuration check in the API handler for cleaner code. - Enhanced logging in `ChatActions` to provide clearer feedback when an unknown provider ID is encountered, defaulting to OpenAI when necessary.
There was a problem hiding this comment.
Actionable comments posted: 0
♻️ Duplicate comments (2)
app/api/bedrock/index.ts (2)
60-70: 🛠️ Refactor suggestionConsolidate configuration checks.
The server-side config is retrieved twice - once at line 61 and again inside
getAwsCredentials()called at line 68. This creates redundant code paths and potential inconsistencies.try { - const config = getServerSideConfig(); + const credentials = getAwsCredentials(); + const config = getServerSideConfig(); const bedrockRegion = config.bedrockRegion as string; const bedrockEndpoint = config.bedrockEndpoint; const client = new BedrockRuntimeClient({ region: bedrockRegion, - credentials: getAwsCredentials(), + credentials: credentials, endpoint: bedrockEndpoint || undefined, });Alternatively, modify
getAwsCredentials()to return both credentials and the config:function getAwsCredentials() { const config = getServerSideConfig(); if (!config.isBedrock) { throw new Error( "AWS Bedrock is not configured properly (ENABLE_AWS_BEDROCK is not true)", ); } if (!config.bedrockAccessKeyId) { throw new Error("AWS Bedrock Access Key ID is missing or empty."); } if (!config.bedrockSecretAccessKey) { throw new Error("AWS Bedrock Secret Access Key is missing or empty."); } return { + config, accessKeyId: config.bedrockAccessKeyId as string, secretAccessKey: config.bedrockSecretAccessKey as string, }; }
140-142: 🛠️ Refactor suggestionHandle partial chunk scenarios.
When streaming data from Bedrock, a partial chunk may be returned if the chunk boundaries don't align with complete JSON structures. Direct parsing could lead to JSON parse errors.
Implement a buffer for incomplete JSON:
try { + let buffer = ''; for await (const event of responseBody) { if (event.chunk?.bytes) { - const chunkData = JSON.parse(decoder.decode(event.chunk.bytes)); + buffer += decoder.decode(event.chunk.bytes); + + // Try to parse complete JSON objects from the buffer + try { + const chunkData = JSON.parse(buffer); + // Reset buffer after successful parse + buffer = '';
🧹 Nitpick comments (5)
app/api/bedrock/index.ts (5)
248-249: Move import statement to the top of the file.The import for
nanoidshould be moved to the top of the file with other imports to follow standard import organization practices.-// Need nanoid for unique IDs -import { nanoid } from "nanoid";Add this to the imports section at the top:
import { BedrockRuntimeClient, InvokeModelWithResponseStreamCommand, InvokeModelCommand, } from "@aws-sdk/client-bedrock-runtime"; +import { nanoid } from "nanoid";
152-157: Add documentation for finish reason logic.The determination of finish reason is complex and would benefit from clearer documentation explaining why different finish reasons are assigned based on token counts.
} else if (chunkData.type === "message_stop") { + // Determine finish reason based on output token count: + // - "stop" when tokens were generated (normal completion) + // - "length" when no tokens were generated (likely hit max token limit) finishReason = chunkData["amazon-bedrock-invocationMetrics"] ?.outputTokenCount > 0 ? "stop" : "length"; // Example logic }
90-97: Add support for non-Claude Bedrock models.The implementation currently only supports Claude models from Anthropic on Bedrock, rejecting other models. This limits the flexibility of the Bedrock integration.
Consider implementing adapters for other Bedrock models like Llama, Titan, or Cohere to expand the available options:
// --- Payload formatting for Claude on Bedrock --- const isClaudeModel = model.includes("anthropic.claude"); -if (!isClaudeModel) { - return NextResponse.json( - { error: true, msg: "Unsupported Bedrock model: " + model }, - { status: 400 }, - ); -} + +let payload; +if (isClaudeModel) { + // Claude payload formatting + const systemPrompts = messages.filter((msg: any) => msg.role === "system"); + const userAssistantMessages = messages.filter( + (msg: any) => msg.role !== "system", + ); + + payload = { + anthropic_version: "bedrock-2023-05-31", + max_tokens: max_tokens || 4096, + temperature: temperature, + messages: userAssistantMessages.map((msg: any) => ({ + role: msg.role, // 'user' or 'assistant' + content: + typeof msg.content === "string" + ? [{ type: "text", text: msg.content }] + : msg.content, + })), + ...(systemPrompts.length > 0 && { + system: systemPrompts.map((msg: any) => msg.content).join("\n"), + }), + }; +} else if (model.includes("llama")) { + // Add Llama model support + // Format payload according to Llama requirements +} else if (model.includes("amazon.titan")) { + // Add Titan model support + // Format payload according to Titan requirements +} else { + return NextResponse.json( + { error: true, msg: "Unsupported Bedrock model: " + model }, + { status: 400 }, + ); +}
108-118: Improve content type validation for messages.The current implementation assumes the message content is either a string or already in the expected format, but doesn't validate the structure for non-string content.
Add explicit validation for non-string content to ensure it matches Claude's expected format:
messages: userAssistantMessages.map((msg: any) => ({ role: msg.role, // 'user' or 'assistant' content: typeof msg.content === "string" ? [{ type: "text", text: msg.content }] - : msg.content, // Assuming MultimodalContent format is compatible + : validateMessageContent(msg.content), // Validate non-string content })), +// Add this validation function elsewhere in the file +function validateMessageContent(content: any) { + // Validate that content is an array + if (!Array.isArray(content)) { + throw new Error("Non-string message content must be an array"); + } + + // Validate each content item has required properties + for (const item of content) { + if (!item.type) { + throw new Error("Each content item must have a 'type' property"); + } + + if (item.type === "text" && typeof item.text !== "string") { + throw new Error("Text content items must have a valid 'text' property"); + } + // Add additional validation for other content types as needed + } + + return content; +}
227-238: Simplify token usage calculation.The current token usage calculation logic is unnecessarily complex with multiple nullish coalescing operators and conditional expressions.
usage: { - prompt_tokens: - responseBody["amazon-bedrock-invocationMetrics"]?.inputTokenCount ?? - -1, - completion_tokens: - responseBody["amazon-bedrock-invocationMetrics"] - ?.outputTokenCount ?? -1, - total_tokens: - (responseBody["amazon-bedrock-invocationMetrics"] - ?.inputTokenCount ?? 0) + - (responseBody["amazon-bedrock-invocationMetrics"] - ?.outputTokenCount ?? 0) || -1, + prompt_tokens: responseBody["amazon-bedrock-invocationMetrics"]?.inputTokenCount ?? -1, + completion_tokens: responseBody["amazon-bedrock-invocationMetrics"]?.outputTokenCount ?? -1, + total_tokens: calculateTotalTokens(responseBody), }, +// Add this helper function +function calculateTotalTokens(responseBody: any) { + const metrics = responseBody["amazon-bedrock-invocationMetrics"]; + if (!metrics) return -1; + + const inputTokens = metrics.inputTokenCount ?? 0; + const outputTokens = metrics.outputTokenCount ?? 0; + + return inputTokens + outputTokens || -1; +}
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (2)
app/api/bedrock/index.ts(1 hunks)app/components/chat.tsx(1 hunks)
🚧 Files skipped from review as they are similar to previous changes (1)
- app/components/chat.tsx
🧰 Additional context used
🧬 Code Definitions (1)
app/api/bedrock/index.ts (2)
app/config/server.ts (1)
getServerSideConfig(128-275)app/api/auth.ts (1)
auth(27-129)
…prompts by bypassing cache system, added Bedrock models with vision detection, enhanced image processing for URLs, fixed Edge routing to Bedrock, added error handling and debugging
There was a problem hiding this comment.
Actionable comments posted: 2
♻️ Duplicate comments (1)
app/client/platforms/bedrock.ts (1)
56-95: Streaming parser still vulnerable to split-JSON chunksChunks are only split on
\nand each line is parsed immediately.
If Bedrock splits JSON in the middle of an object (very common)JSON.parsewill throw.A past review already flagged this (“Handle partial chunk scenarios.”) – consider buffering until a full JSON object is received:
-let lines = text.split("\n"); -for (const line of lines) { - if (!line.startsWith("data:")) continue; - const jsonData = line.substring("data:".length).trim(); - … - const data = JSON.parse(jsonData); +buffer += text; +let boundary; +while ((boundary = buffer.indexOf("\n")) !== -1) { + const raw = buffer.slice(0, boundary).trim(); + buffer = buffer.slice(boundary + 1); + if (!raw.startsWith("data:")) continue; + const jsonData = raw.substring(5).trim(); + if (!jsonData) continue; + const data = JSON.parse(jsonData);Without this, random customer streams will intermittently fail.
🧹 Nitpick comments (1)
app/client/platforms/bedrock.ts (1)
74-87:breakonly exits the innerforloop – stream keeps reading after FINISHAfter receiving
finishReasonthe parserbreaks out of thefor (const line …)loop, but the outerwhilecontinues reading further data.
Use a flag orcontroller.abort()to terminate the read loop once a stop signal is observed to avoid wasting bandwidth and CPU.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (6)
app/api/bedrock/index.ts(1 hunks)app/client/api.ts(4 hunks)app/client/platforms/bedrock.ts(1 hunks)app/components/chat.tsx(5 hunks)app/constant.ts(7 hunks)app/store/chat.ts(2 hunks)
🚧 Files skipped from review as they are similar to previous changes (3)
- app/components/chat.tsx
- app/client/api.ts
- app/constant.ts
🔇 Additional comments (2)
app/store/chat.ts (1)
479-506: Excessive client-side logging risks PII exposure and performance degradation
console.logdumps the fullmodelConfig, session mask, browser UA, etc. every time a user sends a message.
In production this can:
- Leak potentially sensitive user / model details into the browser console (or remote log collectors).
- Impair performance when large objects are stringified on every keystroke.
-console.log( - "[onUserInput] Full modelConfig:", - JSON.stringify(modelConfig, null, 2), -); -… -console.log( - "[onUserInput] Session mask:", - JSON.stringify(session.mask, null, 2), -); +if (process.env.NEXT_PUBLIC_DEBUG === "true") { + // eslint-disable-next-line no-console + console.debug("[onUserInput] provider =", modelConfig.providerName); +}Consider gating these logs behind an explicit debug flag or stripping them entirely in production builds.
[ suggest_essential_refactor ]app/api/bedrock/index.ts (1)
590-597:Bufferusage breaks when this route is deployed to the Edge runtime
Bufferis Node-only; in a Next.js Edge function it is undefined.
Either:
- Switch the route to
runtime: "nodejs"; or- Replace
Buffer.from(arrayBuffer)withbtoa/atoborArrayBuffer→ base64 helpers that work in Edge.Add an explicit comment or runtime config to prevent accidental deployment to the Edge tier.
| const api: ClientApi = getClientApi( | ||
| providerNameFromConfig ?? ServiceProvider.OpenAI, | ||
| ); | ||
|
|
||
| // Edge browser workaround: if we're using a Bedrock model but got wrong API, force Bedrock | ||
| if ( | ||
| modelConfig.model?.includes("anthropic.claude") && | ||
| !api.llm.constructor.name.includes("Bedrock") | ||
| ) { | ||
| console.warn( | ||
| "[onUserInput] Edge workaround: Detected Bedrock model but wrong API class:", | ||
| api.llm.constructor.name, | ||
| "- forcing Bedrock", | ||
| ); | ||
| const bedrockApi = getClientApi(ServiceProvider.Bedrock); | ||
| bedrockApi.llm.chat({ | ||
| messages: sendMessages, | ||
| config: { ...modelConfig, stream: true }, | ||
| onUpdate(message) { | ||
| botMessage.streaming = true; | ||
| if (message) { | ||
| botMessage.content = message; | ||
| } | ||
| get().updateTargetSession(session, (session) => { | ||
| session.messages = session.messages.concat(); | ||
| }); | ||
| }, | ||
| async onFinish(message) { | ||
| botMessage.streaming = false; | ||
| if (message) { | ||
| botMessage.content = message; | ||
| botMessage.date = new Date().toLocaleString(); | ||
| get().onNewMessage(botMessage, session); | ||
| } | ||
| ChatControllerPool.remove(session.id, botMessage.id); | ||
| }, | ||
| onBeforeTool(tool: ChatMessageTool) { | ||
| (botMessage.tools = botMessage?.tools || []).push(tool); | ||
| get().updateTargetSession(session, (session) => { | ||
| session.messages = session.messages.concat(); | ||
| }); | ||
| }, | ||
| onAfterTool(tool: ChatMessageTool) { | ||
| botMessage?.tools?.forEach((t, i, tools) => { | ||
| if (tool.id == t.id) { | ||
| tools[i] = { ...tool }; | ||
| } | ||
| }); | ||
| get().updateTargetSession(session, (session) => { | ||
| session.messages = session.messages.concat(); | ||
| }); | ||
| }, | ||
| onError(error) { | ||
| const isAborted = error.message?.includes?.("aborted"); | ||
| botMessage.content += | ||
| "\n\n" + | ||
| prettyObject({ | ||
| error: true, | ||
| message: error.message, | ||
| }); | ||
| botMessage.streaming = false; | ||
| userMessage.isError = !isAborted; | ||
| botMessage.isError = !isAborted; | ||
| get().updateTargetSession(session, (session) => { | ||
| session.messages = session.messages.concat(); | ||
| }); | ||
| ChatControllerPool.remove( | ||
| session.id, | ||
| botMessage.id ?? messageIndex, | ||
| ); | ||
|
|
||
| console.error("[Chat] failed ", error); | ||
| }, | ||
| onController(controller) { | ||
| // collect controller for stop/retry | ||
| ChatControllerPool.addController( | ||
| session.id, | ||
| botMessage.id ?? messageIndex, | ||
| controller, | ||
| ); | ||
| }, | ||
| }); | ||
| return; | ||
| } |
There was a problem hiding this comment.
🛠️ Refactor suggestion
Edge-workaround duplicates 120+ lines of chat logic – extract to a helper
The fallback block re-implements the full llm.chat invocation (callbacks, pool handling, error paths).
Maintenance cost will double if the main path evolves.
Refactor:
- Move the common
invokeChat(api, sendMessages, …)logic to a small internal util. - Call it from both the normal branch and the Edge-override branch, passing the proper
apiinstance.
This preserves behaviour while eliminating duplication and potential divergence.
🤖 Prompt for AI Agents
In app/store/chat.ts between lines 508 and 591, the Edge browser workaround
duplicates over 120 lines of the llm.chat invocation logic, including callbacks
and error handling, which increases maintenance burden. Refactor by extracting
the common llm.chat call and its associated callbacks into a helper function
that accepts the api instance and other necessary parameters. Then replace both
the normal and Edge-override branches to call this helper with the appropriate
api, ensuring behavior remains the same but duplication is eliminated.
| console.log( | ||
| "[Bedrock] HTTP URL detected, fetching directly:", | ||
| url.substring(0, 50) + "...", | ||
| ); | ||
|
|
||
| try { | ||
| const response = await fetch(url); | ||
| console.log( | ||
| "[Bedrock] Fetch response status:", | ||
| response.status, | ||
| response.statusText, | ||
| ); | ||
|
|
||
| if (!response.ok) { | ||
| console.error( | ||
| "[Bedrock] Failed to fetch image:", | ||
| response.status, | ||
| response.statusText, | ||
| ); | ||
| return null; | ||
| } | ||
|
|
||
| const blob = await response.blob(); | ||
| console.log("[Bedrock] Blob info:", { | ||
| size: blob.size, | ||
| type: blob.type, | ||
| }); | ||
|
|
||
| if (blob.size === 0) { | ||
| console.error( | ||
| "[Bedrock] Fetched blob is empty - cache endpoint may not be working", | ||
| ); | ||
| console.log( | ||
| "[Bedrock] This might be a service worker cache issue - image was uploaded but cache retrieval failed", | ||
| ); | ||
| return null; | ||
| } | ||
|
|
||
| const arrayBuffer = await blob.arrayBuffer(); | ||
| console.log( | ||
| "[Bedrock] ArrayBuffer size:", | ||
| arrayBuffer.byteLength, | ||
| ); | ||
|
|
||
| if (arrayBuffer.byteLength === 0) { | ||
| console.error("[Bedrock] ArrayBuffer is empty"); | ||
| return null; | ||
| } | ||
|
|
||
| const base64Data = | ||
| Buffer.from(arrayBuffer).toString("base64"); | ||
| console.log("[Bedrock] Base64 conversion:", { |
There was a problem hiding this comment.
Unbounded server-side fetch of arbitrary image URLs => SSRF / DoS
The handler blindly fetches any http(s) image URL supplied by the client, converts it to base-64 and forwards it to Bedrock.
Attackers can:
- Trigger internal-network requests (SSRF).
- Download very large files → excessive memory (Buffer) and latency.
- Force long-lived connections that block the Lambda / Edge worker.
Mitigations:
- Whitelist allowed host patterns or reject non-data URLs outright.
- Impose hard byte limits and timeouts on the fetch (e.g.
AbortControllerwith ≤5 MB cap). - Stream-convert instead of buffering entire blobs into memory.
Failing to guard this path is a critical security hole.
🤖 Prompt for AI Agents
In app/api/bedrock/index.ts between lines 220 and 271, the code fetches
arbitrary HTTP URLs without restrictions, exposing the server to SSRF, DoS, and
resource exhaustion risks. To fix this, implement a whitelist of allowed
hostnames or reject any URLs that are not data URLs. Add an AbortController to
enforce a timeout on the fetch request and limit the maximum response size to 5
MB or less. Replace the full blob buffering with a streaming approach to convert
the image data to base64 incrementally, preventing excessive memory usage.
There was a problem hiding this comment.
Actionable comments posted: 1
♻️ Duplicate comments (1)
app/api/bedrock/index.ts (1)
220-263:⚠️ Potential issueUnbounded external image fetch enables SSRF / DoS
fetch(url)is performed on any user-suppliedhttp(s)URL and the entire response is buffered (blob()➜arrayBuffer()➜Buffer).
Attackers can:
- make the server reach arbitrary internal hosts (SSRF);
- request very large files → memory & latency explosion (DoS);
- keep connections open → worker exhaustion.
No host allow-list, size cap, or timeout/abort logic is in place and the full body is materialised in memory.
- const response = await fetch(url); + const ac = new AbortController(); + const timeout = setTimeout(() => ac.abort(), 5_000); // 5 s hard timeout + const response = await fetch(url, { signal: ac.signal, method: "GET" }); + clearTimeout(timeout); + + // Impose a 5 MB limit + const maxBytes = 5 * 1024 * 1024; + let bytesRead = 0; + const chunks: Uint8Array[] = []; + for await (const chunk of response.body!) { + bytesRead += chunk.length; + if (bytesRead > maxBytes) { + throw new Error("Image exceeds 5 MB limit"); + } + chunks.push(chunk); + } + const arrayBuffer = Buffer.concat(chunks).buffer;Also reject non-whitelisted hosts (e.g., only object-storage domains) before fetching.
Failing to add these guards is a critical security hole.
🧹 Nitpick comments (3)
app/api/bedrock/index.ts (3)
459-476:TextDecodercreated per chunk – move outside loopA new
TextDecoderis instantiated for every chunk iteration. This is unnecessary object churn in a hot path.
Instantiate once outside theReadableStreamor reuse a module-level decoder.
497-508:nanoid()for every SSE delta inflates trafficEach tiny delta chunk receives a unique
id. The ID is optional for SSE and identical IDs within one response are acceptable.
Generating a random ID per chunk increases payload size and CPU.
Recommend using a single ID for the entire completion or omit the field.
630-633: Error surfaces raw stack – potential information leak
prettyObject(e)serialises the full error, including stack traces. In production, expose only safe messages to prevent leakage of internal paths.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (5)
app/api/bedrock/index.ts(1 hunks)app/client/platforms/bedrock.ts(1 hunks)app/client/platforms/openai.ts(1 hunks)app/store/chat.ts(3 hunks)app/store/config.ts(1 hunks)
✅ Files skipped from review due to trivial changes (2)
- app/store/config.ts
- app/client/platforms/openai.ts
🚧 Files skipped from review as they are similar to previous changes (2)
- app/store/chat.ts
- app/client/platforms/bedrock.ts
🔇 Additional comments (1)
app/api/bedrock/index.ts (1)
374-381:max_tokensdefaults to 8000 – may exceed model limits
const max_tokens = 8000is hard-coded, but Bedrock’s Claude variants currently cap at 4096 (7 K context).
Sending an oversized value yieldsValidationException. Consider:- max_tokens: typeof max_tokens === "number" && max_tokens > 0 ? max_tokens : 8000, + const DEFAULT_MAX_TOKENS = 4096; + max_tokens: + typeof max_tokens === "number" && max_tokens > 0 + ? Math.min(max_tokens, DEFAULT_MAX_TOKENS) + : DEFAULT_MAX_TOKENS,Either derive limits from model metadata or clamp to a safe maximum.
| const arrayBuffer = await blob.arrayBuffer(); | ||
| console.log( | ||
| "[Bedrock] ArrayBuffer size:", | ||
| arrayBuffer.byteLength, | ||
| ); | ||
|
|
||
| if (arrayBuffer.byteLength === 0) { | ||
| console.error("[Bedrock] ArrayBuffer is empty"); | ||
| return null; | ||
| } | ||
|
|
||
| const base64Data = | ||
| Buffer.from(arrayBuffer).toString("base64"); | ||
| console.log("[Bedrock] Base64 conversion:", { | ||
| originalSize: arrayBuffer.byteLength, | ||
| base64Length: base64Data.length, | ||
| isEmpty: !base64Data || base64Data.length === 0, |
There was a problem hiding this comment.
🛠️ Refactor suggestion
Inefficient memory usage – avoid double buffering
blob.arrayBuffer() followed by Buffer.from(arrayBuffer) duplicates the entire payload in RAM.
Since Bedrock only needs base-64, stream-encode directly while reading (or at least avoid the intermediate blob).
This will halve peak memory and improve latency for large images.
🤖 Prompt for AI Agents
In app/api/bedrock/index.ts around lines 258 to 274, the code currently converts
a blob to an ArrayBuffer and then to a Buffer for base64 encoding, causing
double buffering and high memory usage. To fix this, refactor the code to stream
the blob data directly into a base64 encoder without creating an intermediate
ArrayBuffer, thereby reducing memory consumption and improving performance for
large payloads.
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (5)
app/locales/index.ts (1)
99-105: Guard against missingIntl.Localefor broader runtime safetyThe new
navigatorcheck is great, but if the code is executed in an older browser or a non-browser environment that injects a fakenavigatorwithout shippingIntl.Locale(Node 18, some WebViews, Safari ≤ 14),new Intl.Localewill still throw at line 103. You can add a lightweight fallback to avoid a hard crash:- const locale = new Intl.Locale(navigator.language).maximize(); + const localeCtor = (Intl as any).Locale; + if (typeof localeCtor !== "function") { + return DEFAULT_LANG; + } + const locale = new localeCtor(navigator.language).maximize();This keeps the fast path intact while gracefully falling back when
Intl.Localeis unavailable.app/utils.ts (1)
22-25: Regex is overly verbose – duplicates hamper readability
["""*]contains the same"character three times and the*needs no escaping inside a character class. You can simplify (and slightly speed up) the expression without changing behaviour:- .replace(/^["""*]+|["""*]+$/g, "") - .replace(/[，。！？""""、,.!?*]*$/, "") + .replace(/^["*]+|["*]+$/g, "") + .replace(/[，。！？"、,.!?*]*$/, "")Clearer intent, same effect.
app/client/api.ts (3)
27-27: Import order inconsistency
BedrockApiis appended after a block that appears alphabetically sorted. Consider inserting the import in its canonical position to keep the section tidy and reduce merge-conflict noise.-import { BedrockApi } from "./platforms/bedrock"; +import { BedrockApi } from "./platforms/bedrock"; // ⇐ move up to keep imports alphabetised
363-377: Very chatty console logs – gate behind an environment checkSeven separate
console.logcalls will ship to every browser in production, spamming the console and slightly bloating bundle size. Wrap them in a development guard (or remove once verified).-console.log("[getClientApi] Received Provider (raw):", provider, ...); +if (process.env.NODE_ENV !== "production") { + console.log("[getClientApi] Received Provider (raw):", provider, ...); +}
463-472: Edge-specific Bedrock fallback is brittleHard-coding
provider.includes("anthropic.claude")couples client logic to a single model naming scheme and adds yet another log. If future Bedrock models have different prefixes this will silently fail. Prefer:
- A regular expression for all Bedrock Anthropic model IDs, or
- Reading the mapping from a shared constant that already exists on the server side.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (7)
app/client/api.ts(4 hunks)app/components/chat.tsx(6 hunks)app/locales/index.ts(1 hunks)app/page.tsx(1 hunks)app/store/chat.ts(3 hunks)app/utils.ts(2 hunks)app/utils/stream.ts(1 hunks)
✅ Files skipped from review due to trivial changes (1)
- app/utils/stream.ts
🚧 Files skipped from review as they are similar to previous changes (2)
- app/components/chat.tsx
- app/store/chat.ts
🧰 Additional context used
🧬 Code Graph Analysis (1)
app/client/api.ts (1)
app/client/platforms/bedrock.ts (1)
BedrockApi(8-140)
🔇 Additional comments (4)
app/utils.ts (1)
117-120: Nice defensive checkEarly-returning
falsewhennavigatoris absent prevents server-side crashes. Solid fix.app/client/api.ts (2)
177-179: Bedrock provider correctly wired
ModelProvider.Bedrockis now mapped to the concreteBedrockApiimplementation – looks good and keeps the constructor pattern consistent.
380-410:providercan be empty or unrecognised – add early fallbackIf
provideris"",null, or an unexpected string (e.g. user typo), we’ll fall through to the default GPT case but still emit the large log block above. A simple guard keeps the branch-storm cleaner.-if (typeof provider === "string") { +if (typeof provider === "string" && provider.trim().length) {Optionally default to
ServiceProvider.OpenAIearly when the value is falsy.app/page.tsx (1)
3-10: SSR disabled forHome– verify SEO / first-paint requirementsTurning
Homeinto a purely client-side component (ssr: false) avoids hydration issues but also removes it from the initial HTML payload, impacting crawlers and LCP. Double-check this is acceptable for your landing page; if not, consider a partial client component or suspense boundary instead.
- Updated hardcoded max limit in CompressThreshold input to match the 8000 token configuration - Resolves issue where UI was limiting users to 4000 tokens despite backend supporting 8000 - Now allows users to set compression thresholds up to the full 8000 token limit 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (1)
app/components/model-config.tsx (1)
217-230: Increase to 8 000 is sensible, but verify backend alignment & harden the input handler
- Confirm no hard-coded
4000limits remain in summarisation or server-side validation; otherwise users may hit silent truncation or errors.- Clearing the number field makes
e.currentTarget.valueAsNumberbecomeNaN, which would overwrite the setting withNaN.
Adding a guard and an explicitstepprevents accidental corruption.- max={8000} + max={8000} + step={100} ... - (config.compressMessageLengthThreshold = - e.currentTarget.valueAsNumber), + (config.compressMessageLengthThreshold = + Number.isNaN(e.currentTarget.valueAsNumber) + ? config.compressMessageLengthThreshold + : e.currentTarget.valueAsNumber),
💻 变更类型 | Change Type
🔀 变更说明 | Description of Change
集成 AWS Bedrock 作为新的 LLM 服务提供商。用户现在可以通过配置 AWS 凭证和区域，在 NextChat 中使用 Bedrock 托管的模型。
主要变更包括：
@aws-sdk/client-bedrock-runtime依赖。app/constant.ts中添加了 Bedrock 相关的枚举和模型定义。app/config/server.ts以读取和处理 Bedrock 的环境变量配置。app/api/bedrock/index.ts，负责：app/api/[provider]/[...path]/route.ts以将/api/bedrock/...请求分发给 Bedrock 处理器。app/api/auth.ts以跳过为 Bedrock 请求添加 Authorization 头。app/client/platforms/bedrock.ts。app/client/api.ts(ClientApi构造函数和getClientApi) 以支持 Bedrock 客户端。.env.template添加 Bedrock 相关配置项，并提供了一个使用CUSTOM_MODELS重命名 Bedrock 模型名称的示例。📝 补充信息 | Additional Information
us.anthropic.claude-3-5-sonnet-20241022-v2:0），因为其 API 结构与本项目中的其他 Anthropic 实现类似。理论上支持其他需要类似输入/输出格式的 Bedrock 模型，但可能需要调整 Payload 格式。.env文件中提供有效的AWS_ACCESS_KEY_ID、AWS_SECRET_ACCESS_KEY和AWS_REGION，并将ENABLE_AWS_BEDROCK设置为true来启用此功能。CUSTOM_MODELS环境变量来添加或重命名 Bedrock 模型，例如：+us.anthropic.claude-3-5-sonnet-20241022-v2:0@bedrock=sonnet。Summary by CodeRabbit
New Features
Refactor
Chores