poc(chat): improve prompt to guide LLM toward JSON Patch format #1769
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
|
Updates to Preview Branch (hoshino-poc-chat-op) ↗︎
Tasks are run on every commit but only new migration files are pushed.
View logs for this Workflow Run ↗︎. |
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
Contributor
CI Feedback 🧐A test triggered by this PR failed. Here is an AI-generated analysis of the failure:
|
* Replaced the previous agent-based approach with a LangGraph workflow (`runChat`) for the `'build'` mode * Refined the system prompt to consistently guide the LLM toward proper RFC 6902 JSON Patch output, wrapped in \`\`\`json fences * Improved the prompt structure to allow informative commentary outside the JSON Patch block * Added fallback mechanism for non-schema-editing messages (i.e., omit JSON Patch) * Introduced unit tests for LangGraph flow to ensure format correctness and schema alignment * Updated dependencies to include `@langchain/langgraph`
hoshinotsuyoshi
force-pushed
the
hoshino-poc-chat-op
branch
from
8e0fa37 to
deed8e5
Compare
hoshinotsuyoshi
requested review from
FunamaYukina,
MH4GF,
NoritakaIkeda and
junkisai
and removed request for
a team
Contributor
PR Reviewer Guide 🔍Here are some key observations to aid the review process:
|
Contributor
PR Code Suggestions ✨Explore these optional code suggestions:
|
||||||||||||||||
FunamaYukina
approved these changes
May 28, 2025
Member
FunamaYukina
left a comment
FunamaYukina
left a comment
There was a problem hiding this comment.
Sorry for the late review.
frontend/apps/app/lib/chat/langGraph.ts and others were very helpful in considering the implementation of LangGraph.Thank you for creating the PoC!
| } | ||
|
|
||
| const draft = async (s: ChatState): Promise<Partial<ChatState>> => { | ||
| const agent = mastra.getAgent('databaseSchemaBuildAgent') |
Member
There was a problem hiding this comment.
📝
I found that I can use mastra as is! (I thought I had to go back to LangChain...)
Comment on lines
+91
to
+99
| const check = async (s: ChatState): Promise<Partial<ChatState>> => { | ||
| const m = s.draft?.match(/```json\s+([\s\S]+?)\s*```/i) | ||
| if (!m) return { valid: false } | ||
| try { | ||
| return { valid: true, patch: JSON.parse(m[1]) } | ||
| } catch { | ||
| return { valid: false } | ||
| } | ||
| } |
Member
There was a problem hiding this comment.
[Q]
Is there a plan to enhance the check function to also verify the presence of specific properties, like 'op', similar to how the tests currently do it?
Member
Author
There was a problem hiding this comment.
Good point! I believe validation is necessary from the following perspectives, increasing in complexity as you go:
- Does it conform to the structure of a valid JSON Patch(RFC 6902)?
- (If a JSON Schema or valibot schema is defined in
schema.json) Does the resultingschema.jsonafter applying the patch conform to that schema? - Does applying the patch result in any inconsistencies with the current schema state? (e.g., referencing a non-existent key)
This is probably something we’ll implement eventually, but with low technical uncertainty, it’s a bit hard to justify doing it right away.
Member
There was a problem hiding this comment.
Thank you for the details! It seems important to check until we can actually apply!
Understood. Thank you very much.✨
Member
Author
|
The materials created in this PoC might be used later, but I'll submit them in a separate PR. Closing this one for now. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Just PoC. no need to merge.
poc(chat): improve prompt to guide LLM toward JSON Patch format
runChat) for the'build'mode@langchain/langgraphImage
11.mov
🧭 Overview (Flow)
This change transitions from a sequential
mastra.getAgent().generate(...)-based approach to a state-driven LLM processing pipeline using LangGraph.The new process can be represented as the following DAG:
🧩 Role of Each Node
Thanks to this design, the system can retry up to 3 times if a valid JSON Patch is not returned.
🏗️ Integration with Surrounding Code
route.tsThe
/api/chatAPI endpoint callsrunChat(...)ifmode: 'build'.It passes
schemaTextandchatHistoryinto LangGraph for stateful execution.ChatbotDialog.tsxThe frontend explicitly sets
mode: 'build', which activates the LangGraph flow.mastra/agents/databaseSchemaBuildAgent.tsThe
buildPromptused in LangGraph has been enhanced with an action-oriented system prompt to ensure the LLM always returns a JSON Patch wrapped in a ```json code fence.✅ Advantages of the DAG Approach
Clear State Management: With
ChatStateannotations, each step's expected inputs and outputs are type-safe and well-defined.Self-Healing: The
check → remind → checkloop allows automatic retries when output is malformed or incomplete.Extensibility: Adding a separate graph for
askmode or inserting new branches is straightforward and modular.🧠 Summary
This commit represents a shift from "just calling the LLM" to a more robust architecture that controls, verifies, and retries LLM behavior.
By adopting LangGraph, we've built a reliable and extensible LLM integration pipeline grounded in DAG principles.
Issue
Why is this change needed?
What would you like reviewers to focus on?
Testing Verification
What was done
🤖 Generated by PR Agent at deed8e5
runChatusing LangGraph state graphDetailed Changes
3 files
Add LangGraph-based chat workflow for schema build modeSwitch to LangGraph chat pipeline for build modeEnsure chat API uses build mode for LangGraph1 files
Add unit tests for LangGraph chat pipeline1 files
Update system prompt with strict JSON Patch output instructions2 files
Add @langchain/langgraph dependencyUpdate lockfile for LangGraph and related dependencies1 files
Additional Notes