idea: fine-tuning Gemma 3 270M to offload information extraction

[thinhlpg]

Idea: Fine-tuning Gemma 3 270M to Offload Information Extraction

Goal: use a tiny local model (Gemma 3 270M) to extract memory-worthy facts/preferences/commitments from conversations and emit structured JSON for save_memory without relying on large LLMs.

• Model: Gemma 3 270M (text-only) – runs on CPU; easy to quantize and deploy locally
• Framework: Unsloth fine-tuning (supports Gemma 3; works on consumer GPUs/CPUs)
• Reference: Unsloth Gemma 3 guide link

Dataset and Labels (minimal JSONL)

• Each record contains a conversation window and expected extraction payload
• Focus labels: facts, preferences, promises, entities, projects with spans and confidence

Example (JSONL):

{"input": "User: I prefer dark theme. Also, remind me to ship v1 next Friday.\nAssistant: noted.",
 "output": {
   "preferences": [{"key": "ui_theme", "value": "dark"}],
   "promises": [{"who": "user", "what": "ship v1", "when": "next Friday"}],
   "entities": ["v1"],
   "facts": [],
   "projects": ["v1-release"]
 }}

Prompt/Output Contract

• Prompt template (few-shot) encourages strict JSON only
• Require the model to return a single JSON object with the above keys

Inference prompt (example):

You are an extractor. From the conversation below, extract memory-worthy items.
Return STRICT JSON with keys: preferences, promises, entities, facts, projects.

CONVERSATION:\n{conversation}

Training Notes (Unsloth)

• Use LoRA fine-tuning with small batch sizes
• Recommended inference defaults for Gemma 3: temperature 1.0, top_k 64, top_p 0.95 (per Unsloth guide)
• Export to GGUF for local runtimes (Ollama/llama.cpp) if desired
• See Unsloth Gemma 3 guide for detailed steps and notebooks: link

[Yip-Jia-Qi]

@thinhlpg Correct me if i'm wrong: the plan is to have the search model also call not just search, but an info-extraction API (which is a model itself exposed via API) to summarise.

How would we measure the performance of information extraction?

We should look up some benchmarking papers. This is a well-studied problem.

[thinhlpg]

@Yip-Jia-Qi ah no, this model would be for the internal model of some memory system such as mem0 or graphiti where they need a LLM to extract relationship between entities to build the graph db!
often this is done with OpenAI API or some local model.

I think gemma 3 270M can be finetuned for extract the relationship (i tested this model, it super fast and efficient for local usage and the context length of 32k is suprigingly good, but the quality out of the box is quesitonable)

[Yip-Jia-Qi]

Okay I see. There's definitely more stuff that we can check out before fine tuning our own model. A year ago this kind of knowledge graph things was the hype

Large Language Models Meet Knowledge Graphs for Question Answering: Synthesis and Opportunities
https://arxiv.org/pdf/2505.20099

Retrieval-Augmented Generation with Graphs (GraphRAG)
https://arxiv.org/pdf/2501.00309

Relation Extraction with Fine-Tuned Large Language Models in Retrieval Augmented Generation Frameworks
https://arxiv.org/html/2406.14745v2