merge: add --watch, --log-level, --validate-on-load CLI flags (#40)

## Summary

Three CLI features from the Future Direction section:

- **--watch** (`-w`): 500ms debounced `fs.watch` with in-place fixture
reload. Keeps previous fixtures on validation failure. Error handler on
FSWatcher surfaces dead-watcher conditions.
- **--log-level**: `silent` / `info` (default) / `debug`. Startup
messages and per-request access logs at info. Match traces at debug.
Warnings/errors always print.
- **--validate-on-load**: Runs `validateFixtures()` at startup, exits 1
on errors. In watch mode, errors prevent reload but don't exit.

### Validation checks

**Errors** (exit 1): unrecognized response type, empty content, empty
tool call name, unparseable tool call arguments, empty error message,
invalid HTTP status, negative latency, chunkSize < 1,
truncateAfterChunks < 1, negative disconnectAfterMs.

**Warnings** (logged): duplicate userMessage shadowing, catch-all not in
last position.

### New files

- `src/logger.ts` — Logger class with silent/info/debug levels
- `src/watcher.ts` — File watcher for fixture hot-reload

### Documentation

- CLI options table updated with new flags
- Future Direction CLI section removed (all items implemented)

## Test plan

- [x] 564/564 tests pass (24 new: 15 validation + 9 CLI integration)
- [x] Prettier + ESLint clean
- [x] 2 rounds of CR (code-reviewer, silent-failure-hunter,
code-simplifier, pr-test-analyzer) — clean on final round

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Author: jpr5

CHANGELOG.md

@@ -1,5 +1,15 @@
 # @copilotkit/llmock
 
+## 1.4.0
+
+### Minor Changes
+
+- `--watch` (`-w`): File-watching with 500ms debounced reload. Keeps previous fixtures on validation failure.
+- `--log-level`: Configurable log verbosity (`silent`, `info`, `debug`). Default `info` for CLI, `silent` for programmatic API.
+- `--validate-on-load`: Fixture schema validation at startup — checks response types, tool call JSON, numeric ranges, shadowing, and catch-all positioning.
+- `validateFixtures()` exported for programmatic use
+- `Logger` class exported for programmatic use
+
 ## 1.3.3
 
 ### Patch Changes

README.md

@@ -590,14 +590,17 @@ The package includes a standalone server binary:
 llmock [options]
 ```
 
-| Option         | Short | Default      | Description                        |
-| -------------- | ----- | ------------ | ---------------------------------- |
-| `--port`       | `-p`  | `4010`       | Port to listen on                  |
-| `--host`       | `-h`  | `127.0.0.1`  | Host to bind to                    |
-| `--fixtures`   | `-f`  | `./fixtures` | Path to fixtures directory or file |
-| `--latency`    | `-l`  | `0`          | Latency between SSE chunks (ms)    |
-| `--chunk-size` | `-c`  | `20`         | Characters per SSE chunk           |
-| `--help`       |       |              | Show help                          |
+| Option               | Short | Default      | Description                               |
+| -------------------- | ----- | ------------ | ----------------------------------------- |
+| `--port`             | `-p`  | `4010`       | Port to listen on                         |
+| `--host`             | `-h`  | `127.0.0.1`  | Host to bind to                           |
+| `--fixtures`         | `-f`  | `./fixtures` | Path to fixtures directory or file        |
+| `--latency`          | `-l`  | `0`          | Latency between SSE chunks (ms)           |
+| `--chunk-size`       | `-c`  | `20`         | Characters per SSE chunk                  |
+| `--watch`            | `-w`  |              | Watch fixture path for changes and reload |
+| `--log-level`        |       | `info`       | Log verbosity: `silent`, `info`, `debug`  |
+| `--validate-on-load` |       |              | Validate fixture schemas at startup       |
+| `--help`             |       |              | Show help                                 |
 
 ```bash
 # Start with bundled example fixtures
@@ -697,12 +700,6 @@ Areas where llmock could grow, and explicit non-goals for the current scope.
 - **Token counts**: Usage fields are always zero across all providers.
 - **Vision/image content**: Image content parts are not handled by any provider.
 
-### CLI
-
-- **`--watch` mode**: No file-watching to auto-reload fixtures on change.
-- **`--log-level`**: No configurable log verbosity.
-- **`--validate-on-load`**: No flag to validate fixture schemas at startup.
-
 ## Real-World Usage
 
 [CopilotKit](https://github.com/CopilotKit/CopilotKit) uses llmock across its test suite to verify AI agent behavior across multiple LLM providers without hitting real APIs. The tests cover streaming text, tool calls, and multi-turn conversations across both v1 and v2 runtimes.

package.json

@@ -1,6 +1,6 @@
 {
   "name": "@copilotkit/llmock",
-  "version": "1.3.3",
+  "version": "1.4.0",
   "description": "Deterministic mock LLM server for testing (OpenAI, Anthropic, Gemini)",
   "license": "MIT",
   "packageManager": "pnpm@10.28.2",

src/__tests__/cli.test.ts

@@ -157,7 +157,200 @@ describe.skipIf(!CLI_AVAILABLE)("CLI: fixture loading", () => {
 
   it("fails with error when --fixtures points to a non-existent path", async () => {
     const { stderr, code } = await runCli(["--fixtures", "/nonexistent/path/to/fixtures"]);
-    expect(stderr).toContain("Fixtures path not found");
+    expect(stderr).toContain("Failed to load fixtures");
     expect(code).toBe(1);
   });
 });
+
+describe.skipIf(!CLI_AVAILABLE)("CLI: --log-level", () => {
+  let tmpDir: string;
+
+  beforeEach(() => {
+    tmpDir = makeTmpDir();
+  });
+
+  afterEach(() => {
+    rmSync(tmpDir, { recursive: true, force: true });
+  });
+
+  it("--log-level silent suppresses startup output", async () => {
+    const fixturePath = writeFixture(tmpDir, "test.json");
+    const child = spawnCli(["--fixtures", fixturePath, "--port", "0", "--log-level", "silent"]);
+
+    // Wait for the server to be ready (listen on port)
+    // With silent, there should be no [llmock] output
+    await new Promise((r) => setTimeout(r, 1500));
+
+    const stdout = child.stdout();
+    expect(stdout).not.toContain("[llmock]");
+
+    child.kill("SIGTERM");
+    await new Promise<void>((resolve) => {
+      child.cp.on("close", () => resolve());
+    });
+  });
+
+  it("--log-level info shows startup messages", async () => {
+    const fixturePath = writeFixture(tmpDir, "test.json");
+    const child = spawnCli(["--fixtures", fixturePath, "--port", "0", "--log-level", "info"]);
+
+    await child.waitForOutput(/listening on/i, 5000);
+    expect(child.stdout()).toContain("[llmock]");
+    expect(child.stdout()).toContain("Loaded 1 fixture(s)");
+
+    child.kill("SIGTERM");
+    await new Promise<void>((resolve) => {
+      child.cp.on("close", () => resolve());
+    });
+  });
+
+  it("--log-level debug starts successfully", async () => {
+    const fixturePath = writeFixture(tmpDir, "test.json");
+    const child = spawnCli(["--fixtures", fixturePath, "--port", "0", "--log-level", "debug"]);
+
+    await child.waitForOutput(/listening on/i, 5000);
+    expect(child.stdout()).toContain("[llmock]");
+
+    child.kill("SIGTERM");
+    await new Promise<void>((resolve) => {
+      child.cp.on("close", () => resolve());
+    });
+  });
+
+  it("rejects invalid --log-level value", async () => {
+    const { stderr, code } = await runCli(["--log-level", "verbose"]);
+    expect(stderr).toContain("Invalid log-level");
+    expect(code).toBe(1);
+  });
+});
+
+describe.skipIf(!CLI_AVAILABLE)("CLI: --validate-on-load", () => {
+  let tmpDir: string;
+
+  beforeEach(() => {
+    tmpDir = makeTmpDir();
+  });
+
+  afterEach(() => {
+    rmSync(tmpDir, { recursive: true, force: true });
+  });
+
+  it("passes validation for valid fixtures", async () => {
+    const fixturePath = writeFixture(tmpDir, "test.json");
+    const child = spawnCli(["--fixtures", fixturePath, "--port", "0", "--validate-on-load"]);
+
+    await child.waitForOutput(/listening on/i, 5000);
+    expect(child.stderr()).not.toContain("Validation failed");
+
+    child.kill("SIGTERM");
+    await new Promise<void>((resolve) => {
+      child.cp.on("close", () => resolve());
+    });
+  });
+
+  it("exits 1 on invalid fixture (empty content)", async () => {
+    const filePath = join(tmpDir, "bad.json");
+    writeFileSync(
+      filePath,
+      JSON.stringify({
+        fixtures: [
+          {
+            match: { userMessage: "hello" },
+            response: { content: "" },
+          },
+        ],
+      }),
+      "utf-8",
+    );
+
+    const { stderr, code } = await runCli(["--fixtures", filePath, "--validate-on-load"]);
+    expect(stderr).toContain("Validation failed");
+    expect(code).toBe(1);
+  });
+
+  it("exits 1 on invalid fixture (unparseable toolCalls arguments)", async () => {
+    const filePath = join(tmpDir, "bad-tool.json");
+    writeFileSync(
+      filePath,
+      JSON.stringify({
+        fixtures: [
+          {
+            match: { userMessage: "weather" },
+            response: {
+              toolCalls: [{ name: "get_weather", arguments: "not json" }],
+            },
+          },
+        ],
+      }),
+      "utf-8",
+    );
+
+    const { stderr, code } = await runCli(["--fixtures", filePath, "--validate-on-load"]);
+    expect(stderr).toContain("Validation failed");
+    expect(code).toBe(1);
+  });
+});
+
+describe.skipIf(!CLI_AVAILABLE)("CLI: --watch", () => {
+  let tmpDir: string;
+
+  beforeEach(() => {
+    tmpDir = makeTmpDir();
+  });
+
+  afterEach(() => {
+    rmSync(tmpDir, { recursive: true, force: true });
+  });
+
+  it("survives invalid JSON during reload", async () => {
+    const fixturePath = writeFixture(tmpDir, "test.json");
+    const child = spawnCli(["--fixtures", fixturePath, "--port", "0", "--watch"]);
+
+    await child.waitForOutput(/listening on/i, 5000);
+
+    // Write invalid JSON
+    writeFileSync(fixturePath, "{ not valid json", "utf-8");
+
+    // Wait for the reload attempt — server should stay up
+    await new Promise((r) => setTimeout(r, 1500));
+
+    // Server should still be running (not crashed)
+    expect(child.cp.exitCode).toBeNull();
+
+    child.kill("SIGTERM");
+    await new Promise<void>((resolve) => {
+      child.cp.on("close", () => resolve());
+    });
+  });
+
+  it("reloads fixtures when file changes", async () => {
+    const fixturePath = writeFixture(tmpDir, "test.json");
+    const child = spawnCli(["--fixtures", fixturePath, "--port", "0", "--watch"]);
+
+    await child.waitForOutput(/listening on/i, 5000);
+    expect(child.stdout()).toContain("Watching");
+
+    // Modify the fixture file
+    writeFileSync(
+      fixturePath,
+      JSON.stringify({
+        fixtures: [
+          {
+            match: { userMessage: "goodbye" },
+            response: { content: "Bye!" },
+          },
+        ],
+      }),
+      "utf-8",
+    );
+
+    // Wait for reload
+    await child.waitForOutput(/Reloaded/i, 5000);
+    expect(child.stdout()).toContain("Reloaded 1 fixture(s)");
+
+    child.kill("SIGTERM");
+    await new Promise<void>((resolve) => {
+      child.cp.on("close", () => resolve());
+    });
+  });
+});