Support json schemas to force the model to generate valid JSON #116
Description
From the Llamacpp documentation:
GBNF (GGML BNF) is a format for defining formal grammars to constrain model outputs in llama.cpp. For example, you can use it to force the model to generate valid JSON, or speak only in emojis.
See: https://github.com/ggml-org/llama.cpp/blob/master/grammars/README.md#json-schemas--gbnf
Usage examples
Extract data
schema:
{
"type": "object",
"properties": {
"name": {
"type": "string"
},
"capital": {
"type": "string"
},
"languages": {
"type": "array",
"items": {
"type": "string"
}
}
},
"required": ["name", "capital", "languages"]
}prompt:
Tell me about Canada
response:
{
"name": "Canada",
"capital": "Ottawa",
"languages": ["English", "French"]
}Intent / Routing
schema:
{
"type": "object",
"properties": {
"action": {
"type": "string",
"description": "The action to take with the selected agent",
"enum": ["talk to", "ignore", "call"]
},
"agent_name": {
"type": "string",
"description": "The name of the agent to route the request to",
"enum": ["Bob", "Sam", "John"]
}
},
"required": ["action", "agent_name"]
}system prompt:
You are an AI assistant helping route user requests to the appropriate agent.
Available agents:
- Bob: Handles sales inquiries
- Sam: Handles technical support
- John: Handles billing questions
Available actions:
- "talk to": Direct conversation with the agent
- "ignore": Skip this request
- "call": Schedule a phone call with the agent
Based on the user's message, determine which action to take and which agent to contact.
Respond with the appropriate action and agent name.
User message:
I need help with my invoice"
response:
{
"action": "talk to",
"agent_name": "John"
}