netagent-benchmark

netagent-benchmark is a Java CLI for repeatable LLM network-engineering experiments. It connects an agent loop to Cisco Modeling Labs (CML) through Cisco's MCP server, or to a local replay environment during development, and writes one sanitized run.json per run.

The repository is an experiment harness, not a generic agent platform and not a web service. Its long-term research path is:

1. develop repeatable cases with replayed network observations;
2. validate observation and diagnosis against real CML when campus access is available;
3. run isolated, resettable CML configuration and repair experiments with an explicit per-task tool policy.

What works today

• Java 25 Maven CLI with a real OpenAI-compatible LLM adapter (DeepSeek by default).
• A CML MCP stdio adapter built on the official MCP Java SDK.
• A replay network environment and scripted LLM adapter for deterministic local tests and smoke runs.
• A TaskCase format (task input, per-task tool allowlist, replay fixture, expected evidence, scoring rule) with one deterministic OSPF diagnosis case and a rule-based evaluator that fills run.json scores.
• A typed agent loop with tool traces, token counts, duration, and tool audit records.
• Sanitized, benchmark-owned run.json artifacts under experiments/runs/.
• An optional Workbench worker that claims persistent jobs, supervises a child benchmark process, and uploads the unchanged artifact.
• CML diagnostics and tool-discovery commands for use once the internal network is reachable.

Real CML execution is intentionally not claimed as validated until doctor, tools, and a real run have completed in the target environment.

Architecture

CLI
  -> experiment/ExperimentRunner
  -> agent/AgentLoop
  -> ports/llm + ports/network (+ ports/evaluation)
  -> adapters/llm/openai and adapters/network/{replay,cml}
  -> artifact/run.json

The core depends only on small ports:

• LlmClient: asks a model to respond to messages and exposed tools.
• NetworkEnvironment: lists and calls network tools.
• Evaluator: deterministically scores a finished trajectory against its case.

ReplayNetworkEnvironment and CmlMcpEnvironment implement the same network port. This lets local regression tests exercise the same agent loop that later talks to real CML.

Repository layout

src/main/java/com/frankliu/netagent/
├── cli/          # Picocli entry point and CML diagnostics
├── experiment/   # Runtime settings and ExperimentRunner orchestration
├── agent/        # Agent loop, prompts, and per-task tool allowlist
├── cases/        # TaskCase schema, loading, and validation
├── evaluation/   # Rule-based EvidenceEvaluator and score reports
├── ports/        # Stable LLM, network, and evaluation interfaces
├── adapters/     # OpenAI-compatible, replay, and CML MCP implementations
├── artifact/     # run.json schema, redaction, trace, and file writing
└── worker/       # Optional Workbench client and child-process supervisor

src/test/java/    # Mirrors production package boundaries
experiments/cases/# Version-controlled TaskCase definitions (case.json per case)
experiments/runs/ # Generated, sanitized run artifacts; git-ignored
research/         # Literature and benchmark context
docs/             # Public project notes

Quick start

mise.toml pins Java and Maven. Run Maven through mise when Maven is not on your shell PATH:

mise exec -- mvn test
mise exec -- mvn package

The packaged CLI is:

java -jar target/netagent-benchmark-0.1.0-SNAPSHOT.jar --help

Local, deterministic checks

java -jar target/netagent-benchmark-0.1.0-SNAPSHOT.jar artifact-smoke "list all CML labs"
java -jar target/netagent-benchmark-0.1.0-SNAPSHOT.jar loop-smoke "list all CML labs"

loop-smoke uses replay adapters only. It never contacts an LLM or CML.

Benchmark cases

Cases live under experiments/cases/<case-id>/case.json. A case pins the task input, the per-task tool allowlist, the replay fixture, the expected evidence, and the scoring rule.

# Deterministic scripted reference run; must always score 1.0
java -jar target/netagent-benchmark-0.1.0-SNAPSHOT.jar case run replay/ospf-adjacency-down-v1 --scripted

# Real configured LLM against the same replay fixture
java -jar target/netagent-benchmark-0.1.0-SNAPSHOT.jar case run replay/ospf-adjacency-down-v1

Case runs never contact CML. The run artifact gains an evaluation section with per-check results, result.score is filled, and task.case_id records the real case id (schema 1.1).

Workbench worker

After starting agent-eval-workbench, run:

java -jar target/netagent-benchmark-0.1.0-SNAPSHOT.jar worker

The worker polls http://localhost:8080, claims one queued benchmark at a time, and launches case run in a child JVM. Each job uses an isolated output directory. Heartbeats carry cancellation state; timeouts terminate the child; and a generated run.json is uploaded even when it records a failed run.

For a single local claim:

java -jar target/netagent-benchmark-0.1.0-SNAPSHOT.jar worker --once

Configure another service or path with WORKBENCH_URL, NETAGENT_WORKER_ID, NETAGENT_RUNNER_JAR, NETAGENT_CASES_ROOT, and NETAGENT_WORKER_STAGING_ROOT. The worker inherits provider API keys and base URLs from its environment but does not send them to Workbench. It validates and propagates the claimed reasoning mode; LOCAL_MOCK claims use DISABLED.

Real LLM, then real CML

Copy .env.example and let your shell, IDE, or container inject its values. The application deliberately does not parse .env itself.

For DeepSeek, set LLM_MODEL to deepseek-v4-flash or deepseek-v4-pro and set NETAGENT_REASONING_MODE to canonical DISABLED, HIGH, or MAX. DISABLED is the compatibility default. HIGH and MAX enable DeepSeek thinking and send the matching reasoning effort; these modes are rejected for other providers.

These values follow the official DeepSeek model list and thinking mode contract. Thinking tool-call turns retain and resend reasoning_content, as required by that contract for multi-turn tool use.

java -jar target/netagent-benchmark-0.1.0-SNAPSHOT.jar llm-smoke "summarize OSPF in one sentence"
java -jar target/netagent-benchmark-0.1.0-SNAPSHOT.jar doctor
java -jar target/netagent-benchmark-0.1.0-SNAPSHOT.jar tools
java -jar target/netagent-benchmark-0.1.0-SNAPSHOT.jar run "list all CML labs"

Use doctor and tools before run after any CML, campus-network, VPN, or MCP configuration change.

Run artifacts

Every run writes:

experiments/runs/<run-id>-<task-slug>/run.json

run.json is the source of truth for a benchmark run. It contains its schema version, task, agent identity, result, metrics, full tool trace, tool audit, and artifact path. Secret-like fields are removed before it is written. agent.reasoning_mode records the effective mode. This additive metadata keeps schema 1.1 because no existing field or meaning changed.

Case execution remains independent of agent-eval-workbench. The optional worker integrates only at the queue and artifact boundary; it does not control the benchmark runtime model or artifact schema.

Scope boundaries

• Do not add a web server, database, queue, dashboard, or workbench client to this repository without a concrete experiment that requires it.
• Do not add a second LLM adapter unless it is an explicit model-comparison variable in an experiment.
• Do not treat replay results as proof of CML behaviour.
• Before enabling CML write operations, introduce task cases with explicit allowed tools, isolated/resettable labs, and post-change verification.

Research context

• Project topic summary
• Literature review
• NetConfBench IETF Draft