Generated via deep research (June 2026). 19 confirmed claims from 24 sources, adversarially verified.
On low-level device configuration (OSPF, BGP, RIP), GPT-4 produces error-free output in only 1 of 3 runs without RAG. With RAG, accuracy approaches 100% except for OSPF. On realistic large-scale queries, AI agents achieve only 13–38% average performance, with worst-case as low as 3%.
| Issue | Detail |
|---|---|
| Low-level config errors | GPT-4 error-free in 1/3 runs on OSPF/RIP/BGP/RIFT |
| Basic mistakes | IP addressing, subnet mask conversion, numeric notation — fluent but wrong |
| Transitive conflicts | Cannot detect indirect contradictions (e.g., "s1→h2 via s2" + "s2 cannot reach h2") |
| Regression on repair | LLMs introduce new errors when fixing misconfigurations |
| Scale degradation | Performance drops sharply as topology grows (20→754 nodes) |
- NetConfEval — Wang et al., CoNEXT 2024 (won 2025 IRTF Applied Networking Research Prize)
- NetArena — Zhou et al., ICLR 2026
- Anwar & Caesar — SIGCOMM CCR 2025, Best of CCR
- Lost in Transmission — 2025
Four-stage pipeline:
- Requirement formalization (NL → formal specs)
- API translation (formal specs → API calls)
- Routing algorithm generation (as code)
- Low-level device configuration (with RAG support)
- Uses FRRouting (vtysh) and Kathara emulator
- Open-source code
- Dataset on HuggingFace
- 40 JSON-defined tasks spanning routing, QoS, and security
- Three evaluation metrics:
- Reasoning score: embedding cosine similarity with reference answer
- Command score: hierarchical
ciscoconfparsediff with F1 - Testcase score: proportion of passed verification checks
- Runs on GNS3 with official vendor images
- Agent-Network Interface:
get-topology,get-running-cfg,update-cfg,execute-cmd - IETF Draft v01
- 9 SOTA LLMs tested on 231 synthesized misconfiguration problems
- Topologies from 20–754 nodes
- Formal verification used as ground truth
- LLMs frequently introduce regressions; iterative feedback helps only slightly
- arXiv:2604.22513
Reliable LLM-powered network automation requires integrating LLMs into iterative, closed-loop workflows guided by formal verification — not one-shot autonomous fixes.
- 37.7% of 11,088 ACLs in a university campus network contain conflicting rule overlaps
- 27% of conflicted ACLs have >20 conflicts
- ~29% of cloud ACLs show overlaps
Source: Mondal et al., HotNets 2025 (UCLA + Microsoft Research)
Multi-agent LLM architectures that decompose natural-language network configuration tasks into subtasks via specialized agents represent a promising approach:
- ConfAgent (IWQoS 2025): dedicated Conflict Detector + Formal Synthesizer agents
- CoTNet (2025): template generation → parameter assignment
- Rozsival et al. (IEEE NCA 2025): explicit multi-agent decomposition
No existing benchmark or framework integrates the MCP (Model Context Protocol) standard for LLM-to-network-device communication. NetConfEval, NetConfBench, Cornetto, and NetArena all use custom Agent-Network Interfaces or direct API calls.
This project's use of Cisco CML's MCP server to bridge LLMs with network devices fills a genuine gap in the research landscape.
- How does Cisco CML's MCP server compare to NetConfBench's custom Agent-Network Interface in terms of latency, reliability, and LLM usability?
- Can iterative formal-verification-guided workflows from Cornetto be integrated with the MCP-based CML interface?
- Do NetConfEval/NetConfBench findings generalize from GNS3/Kathara to physical Cisco hardware via CML?
- What is the minimum viable prompting/RAG strategy for reliable L2/L3 configurations on 10–50 node CML topologies?