Fix Gemini CLI evaluation issues and update notebook

- Fix LICENSE: Remove 'pytest-' prefix from project name
- Fix Gemini CLI hanging: Patch metacoder to add --yolo flag
- Fix Gemini CLI auth prompts: Copy OAuth credentials to workdirs
- Update run_gemini_eval.sh: Add OPENAI_API_KEY and ANTHROPIC_API_KEY
- Update experiment_1_run_evaluations.ipynb: Make cells executable
- Document fixes in notes/GEMINI_CLI_YOLO_FIX.md

The Gemini CLI was hanging indefinitely because it waited for user
approval for tool calls even in non-interactive mode. Fixed by adding
--yolo flag to auto-approve all actions.

Additionally, each test workdir triggered OAuth re-authentication.
Fixed by copying google_accounts.json and oauth_creds.json from
~/.gemini/ to each workdir.

Both fixes are temporary patches to the installed metacoder package
and should be contributed upstream.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

Author: justaddcoffee

LICENSE

@@ -11,7 +11,7 @@ modification, are permitted provided that the following conditions are met:
   this list of conditions and the following disclaimer in the documentation
   and/or other materials provided with the distribution.
 
-* Neither the name of pytest-mcp_literature_eval nor the names of its
+* Neither the name of mcp_literature_eval nor the names of its
   contributors may be used to endorse or promote products derived from
   this software without specific prior written permission.
 

notebook/experiment_1_run_evaluations.ipynb

@@ -55,14 +55,7 @@
    "execution_count": null,
    "metadata": {},
    "outputs": [],
-   "source": [
-    "# Run Claude evaluation\n",
-    "# Uncomment to run:\n",
-    "# !./run_claude_eval.sh\n",
-    "\n",
-    "print(\"Claude evaluation script: ./run_claude_eval.sh\")\n",
-    "print(\"Status: Already completed (results/compare_agents/claude_20251031.yaml)\")"
-   ]
+   "source": "# Check if Claude evaluation already exists\nclaude_result = f\"results/compare_agents/claude_{datetime.now().strftime('%Y%m%d')}.yaml\"\nif os.path.exists(claude_result):\n    print(f\"✓ Claude evaluation already completed: {claude_result}\")\nelse:\n    print(\"Running Claude evaluation (this takes 2-3 hours)...\")\n    result = subprocess.run([\"./run_claude_eval.sh\"], capture_output=True, text=True)\n    if result.returncode == 0:\n        print(\"✓ Claude evaluation complete!\")\n    else:\n        print(f\"✗ Error: {result.stderr}\")"
   },
   {
    "cell_type": "markdown",
@@ -82,14 +75,7 @@
    "execution_count": null,
    "metadata": {},
    "outputs": [],
-   "source": [
-    "# Run Goose evaluation\n",
-    "# Uncomment to run:\n",
-    "# !./run_goose_eval.sh\n",
-    "\n",
-    "print(\"Goose evaluation script: ./run_goose_eval.sh\")\n",
-    "print(\"Status: Needs re-run with fixed MCP configs\")"
-   ]
+   "source": "# Check if Goose evaluation already exists\ngoose_result = f\"results/compare_agents/goose_{datetime.now().strftime('%Y%m%d')}.yaml\"\nif os.path.exists(goose_result):\n    print(f\"✓ Goose evaluation already completed: {goose_result}\")\nelse:\n    print(\"Running Goose evaluation (this takes 2-3 hours)...\")\n    result = subprocess.run([\"./run_goose_eval.sh\"], capture_output=True, text=True)\n    if result.returncode == 0:\n        print(\"✓ Goose evaluation complete!\")\n    else:\n        print(f\"✗ Error: {result.stderr}\")"
   },
   {
    "cell_type": "markdown",
@@ -109,14 +95,7 @@
    "execution_count": null,
    "metadata": {},
    "outputs": [],
-   "source": [
-    "# Run Gemini evaluation\n",
-    "# Uncomment to run:\n",
-    "# !./run_gemini_eval.sh\n",
-    "\n",
-    "print(\"Gemini evaluation script: ./run_gemini_eval.sh\")\n",
-    "print(\"Status: Currently running in background\")"
-   ]
+   "source": "# Check if Gemini evaluation already exists\ngemini_result = f\"results/compare_agents/gemini_{datetime.now().strftime('%Y%m%d')}.yaml\"\nif os.path.exists(gemini_result):\n    print(f\"✓ Gemini evaluation already completed: {gemini_result}\")\nelse:\n    print(\"Running Gemini evaluation (this takes 2-3 hours)...\")\n    result = subprocess.run([\"./run_gemini_eval.sh\"], capture_output=True, text=True)\n    if result.returncode == 0:\n        print(\"✓ Gemini evaluation complete!\")\n    else:\n        print(f\"✗ Error: {result.stderr}\")"
   },
   {
    "cell_type": "markdown",
@@ -175,14 +154,7 @@
    "execution_count": null,
    "metadata": {},
    "outputs": [],
-   "source": [
-    "# Run cross-agent analysis notebook\n",
-    "# Uncomment when all evaluations are complete:\n",
-    "# !jupyter nbconvert --execute --to notebook --inplace experiment_1_cross_agent_analysis.ipynb\n",
-    "\n",
-    "print(\"Analysis notebook: experiment_1_cross_agent_analysis.ipynb\")\n",
-    "print(\"Run after all three evaluations are complete\")"
-   ]
+   "source": "# Check if all evaluations are complete, then run analysis\nresult_files = glob('results/compare_agents/*.yaml')\nagents_complete = {'claude': False, 'goose': False, 'gemini': False}\n\nfor f in result_files:\n    agent = Path(f).stem.split('_')[0]\n    if agent in agents_complete:\n        agents_complete[agent] = True\n\nif all(agents_complete.values()):\n    print(\"All evaluations complete! Running cross-agent analysis...\")\n    result = subprocess.run([\n        \"jupyter\", \"nbconvert\", \"--execute\", \"--to\", \"notebook\", \n        \"--inplace\", \"experiment_1_cross_agent_analysis.ipynb\"\n    ], capture_output=True, text=True)\n    if result.returncode == 0:\n        print(\"✓ Analysis complete!\")\n    else:\n        print(f\"✗ Error: {result.stderr}\")\nelse:\n    missing = [k for k, v in agents_complete.items() if not v]\n    print(f\"⏳ Waiting for evaluations to complete: {', '.join(missing)}\")\n    print(\"Run this cell again after all evaluations finish.\")"
   },
   {
    "cell_type": "markdown",
@@ -250,4 +222,4 @@
  },
  "nbformat": 4,
  "nbformat_minor": 4
-}
+}

notes/GEMINI_CLI_YOLO_FIX.md

@@ -0,0 +1,100 @@
+# Gemini CLI --yolo Flag Fix
+
+## Issue
+
+When running Gemini CLI evaluations through metacoder, the process was hanging indefinitely after sending the prompt. Investigation revealed:
+
+1. Gemini CLI was invoked with `-p` (non-interactive prompt mode)
+2. The session log showed only the user message, no response
+3. Process remained stuck for 17+ minutes with no progress
+
+## Root Cause
+
+Gemini CLI's non-interactive mode (`-p/--prompt`) **still requires user approval for tool calls** by default. When the MCP tools (like ARTL) needed to be invoked, Gemini was waiting for user confirmation, causing the process to hang indefinitely.
+
+From `gemini --help`:
+```
+--yolo      Automatically accept all actions (aka YOLO mode)?  [boolean] [default: false]
+--approval-mode  Set the approval mode: default (prompt for approval), auto_edit (auto-approve edit tools), yolo (auto-approve all tools)  [string] [choices: "default", "auto_edit", "yolo"]
+```
+
+## Solution
+
+Modified metacoder's `GeminiCoder` class with two fixes:
+
+### Fix 1: Add --yolo flag for non-interactive mode
+
+**File**: `.venv/lib/python3.10/site-packages/metacoder/coders/gemini.py`
+
+**Line 154-156** (changed):
+```python
+# Build the command using non-interactive mode (-p flag)
+# Add --yolo flag to auto-approve all tool actions in non-interactive mode
+command = ["gemini", "-p", text, "--yolo"]
+```
+
+Previously:
+```python
+# Build the command using non-interactive mode (-p flag)
+command = ["gemini", "-p", text]
+```
+
+### Fix 2: Copy OAuth credentials to avoid re-authentication
+
+Each test workdir gets a fresh `.gemini/` directory, causing Gemini CLI to re-authenticate (opening browser) for every test.
+
+**File**: `.venv/lib/python3.10/site-packages/metacoder/coders/gemini.py`
+
+**Line 116-133** (added after line 114):
+```python
+# Copy OAuth credentials from global config to avoid re-authentication
+# This copies google_accounts.json and oauth_creds.json
+global_gemini_dir = Path.home() / ".gemini"
+for auth_file in ["google_accounts.json", "oauth_creds.json"]:
+    auth_path = global_gemini_dir / auth_file
+    if auth_path.exists():
+        try:
+            with open(auth_path) as f:
+                auth_content = json.load(f)
+            config_objects.append(
+                CoderConfigObject(
+                    file_type=FileType.JSON,
+                    relative_path=f".gemini/{auth_file}",
+                    content=auth_content,
+                )
+            )
+        except Exception as e:
+            logger.warning(f"Could not load {auth_file}: {e}")
+```
+
+## Testing
+
+Verified the fix with a single test case:
+
+```bash
+# Before fix: Hung indefinitely (>17 minutes)
+# After fix: Completed in ~19-21 seconds ✓
+
+$ timeout 120 uv run metacoder eval /tmp/gemini_test_one_case.yaml -o /tmp/gemini_test_result.yaml
+💎 Running command: gemini with prompt
+💎 Command took 19.11 seconds
+✓ Evaluation completed 🎉!
+```
+
+## Impact
+
+- **Before**: Gemini evaluations impossible due to infinite hang
+- **After**: Gemini evaluations complete successfully in reasonable time (~20-30 seconds per test)
+- **Note**: This is a temporary patch to the installed metacoder package. The fix should be contributed upstream to the metacoder project.
+
+## Related Issues
+
+- Gemini evaluations also require `OPENAI_API_KEY` and `ANTHROPIC_API_KEY` environment variables for the evaluation metrics (CorrectnessMetric uses these models for scoring)
+- Updated `run_gemini_eval.sh` to export these keys
+
+## Upstream Fix Needed
+
+This fix should be submitted as a pull request to the metacoder repository:
+- Repository: https://github.com/ai4curation/metacoder
+- Suggested change: Add `--yolo` flag to the gemini command in `metacoder/coders/gemini.py`
+- Alternative: Make approval mode configurable via the coder config YAML

run_gemini_eval.sh

@@ -1,5 +1,7 @@
 #!/bin/bash
 cd /Users/jtr4v/PythonProject/mcp_literature_eval
+export OPENAI_API_KEY=$(cat ~/openai.key)
+export ANTHROPIC_API_KEY=$(cat ~/anthropic.key)
 export PUBMED_EMAIL=justinreese@lbl.gov
 export PUBMED_API_KEY=01eec0a16472164c6d69163bd28368311808
 rm -f results/compare_agents/gemini_$(date +%Y%m%d).yaml