Dynamic Context Engineering (DCE) for LLM Tool Usage

This repository contains the implementation of Dynamic Context Engineering (DCE), a structured adaptation of Agentic Context Engineering (ACE) specifically tailored for Large Language Model (LLM) function calling and tool use. This project extends the Berkeley Function-Calling Leaderboard (BFCL) codebase to support playbook construction, dataset splitting, dynamic tool-querying, and model-swappable Reflector/Curator components.

Overview

Tool-calling performance in LLMs can be brittle, with models frequently repeating mistakes across tool interactions. While Agentic Context Engineering (ACE) introduced a powerful Generator -> Reflector -> Curator loop for general reasoning, its monolithic memory becomes unwieldy when dealing with dozens of heterogeneous tools that require highly specific behavioral constraints.

DCE addresses this by treating the learning playbook as a structured memory database. Each tool group receives its own dedicated section, and the Curator issues explicit database-style operations (ADD, MODIFY, DELETE) instead of relying on purely additive updates.

Architecture

The DCE pipeline introduces a specialized, segmented learning and evaluation architecture.

graph TD
    subgraph Learning Phase
        Gen[Generator] -->|Produces Tool Call| Ref[Reflector]
        Ref -->|Evaluates Error & Suggests Improvement| Cur[Curator]
        Cur -->|Issues ADD/MODIFY/DELETE| PB[(Structured Playbook)]
    end

    subgraph Evaluation Phase
        User[User Prompt] --> DQ[Dynamic Query Agent]
        PB -.->|Candidate Sections| DQ
        DQ -->|Selects Relevant Tool Group Sections| GenEval[Generator]
        User --> GenEval
        GenEval -->|Executes with specialized context| Output[Final Tool Call]
    end

Loading

The system operates across three distinct roles:

1. Generator:
   • During Training: Operates without the playbook to produce raw reasoning trajectories and tool calls. This exposes native failure modes against ground-truth for the system to learn from.
   • During Evaluation: Utilizes the curated playbook (either fully or through a dynamic query subset) to improve tool-calling accuracy.
2. Reflector:
   • Analyzes deviations between the Generator's predicted tool calls and the ground-truth execution traces.
   • Extracts concrete lessons, identifies root causes of failures, and proposes corrective usage patterns in natural language.
3. Curator:
   • Acts as an active database agent over the structured playbook.
   • It receives reflections from the Reflector and maps them to specific tool-group sections.
   • Issues targeted ADD, MODIFY, or DELETE operations to actively maintain the quality of the insights, pruning outdated rules and refining existing ones.

Key Contributions

• Tool-Specific Structured Memory: Replaces single monolithic memory buffers with structured, per-tool playbooks. By isolating insights by tool group, DCE prevents cross-tool interference and keeps prompts hyper-specific.
• Enhanced Training Pipeline: Integrates ground-truth tool calls and evaluation execution traces, enabling highly targeted error reflection and fine-grained memory updates.
• Selective Memory Retrieval (Dynamic Query): Introduces a dynamic retrieval mechanism where an agent selects only the relevant tool-group sections of the playbook to inject into the prompt. This prevents unbounded context growth as the number of supported APIs scales, significantly reducing context length overhead while maintaining accuracy.
• Asymmetric Architecture Scaling: Demonstrates that smaller, weaker models (like Claude Haiku) can act as the Reflector and Curator to successfully curate memory for a stronger Generator model (like DeepSeek-V3). This proves that robust memory curation does not strictly require state-of-the-art reasoning capacity for every step.

Dataset and Evaluation

The evaluation is conducted on a subset of the Berkeley Function-Calling Leaderboard (BFCL) benchmark, focusing on categories such as Single-Turn, Multi-Turn, Relevance, and Irrelevance tasks.

• Training Set: 1,866 samples were used to build the playbook.
• Test Set: 934 samples were used to evaluate the model's accuracy.

The Generator model utilized in these baseline experiments is DeepSeek-V3.2-Exp-FC.

Repository Structure

This repository is built on top of a fork of the original Gorilla / BFCL codebase. It extends the core function-calling evaluation engine to support the DCE learning and inference loops.

• berkeley-function-call-leaderboard/: The extended BFCL evaluation framework.
• Final_Report.pdf: The full research paper detailing the methodology, architecture, and empirical findings.
• ACE_Poster.pdf: A visual summary and poster presentation of the DCE framework.

Results

Across the BFCL benchmark categories, DCE provides consistent and robust accuracy gains. The system boosts tool-calling accuracy by approximately 10% over the baseline DeepSeek-V3.2-Exp-FC model. By preserving hyper-specific guidance in a modular format, DCE drastically improves performance on challenging tasks such as Relevance Detection (knowing when not to use a tool) and complex API domains like Math and Twitter APIs. Furthermore, the Dynamic Querying system preserves these gains while maintaining a strict token budget.