Replace Gemini with Codex evaluation and update analysis#24
Merged
Conversation
- Increase y-axis label font size to 15pt with custom line breaks - Increase x-axis label font size to 15pt - Increase legend font size (labels: 13pt, title: 14pt) - Adjust layout to prevent legend overlap with plots 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Update fig1_mcp_performance_by_agent.png and fig3_model_comparison.png to match current notebook state after Figure 2 modifications. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
- Add codex CLI evaluation config - Update experiment 1 notebook to use codex instead of gemini - Regenerate Figures 1 and 2 with codex evaluation results - Cross-agent comparison now includes Claude Code, Codex CLI, and Goose CLI 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Contributor
|
- Remove gemini config file - Update config file symlinks (typechange from symlink to file) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
justaddcoffee
changed the title
🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
- claude_20251126.yaml - codex_20251205.yaml - goose_20251125.yaml 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
- Create project/templates/codex_template.yaml for codex agent - Generate project/generated/literature_mcp_eval_config_codex.yaml - Update generate_configs.py to include codex in template mapping - Fix codex symlink to point to correct generated file 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
- Run codex with gpt-5.1-codex-max: 63% pass rate (63/100) - Run goose with gpt-5.1: 17% pass rate (17/100) - Update cross-agent analysis notebook to use new codex results - Regenerate figures with GPT-5 data - Add failure analysis notebook diagnosing goose hallucination - Diagnose goose hallucination issue with empty MCP responses Key finding: Goose doesn't handle MCP tool failures gracefully, fabricating plausible but incorrect answers when tools return empty results, while codex reports inability to access. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
- Codex: 64% pass rate (sandbox_permissions=[] blocked filesystem cheating) - Claude: 48% pass rate (verified no answer file access) - Goose: ~17% pass rate (honest baseline) Renamed result files to simpler agent_date format. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Update experiment_1_cross_agent_analysis.ipynb to use verified honest run results: - claude_20251207.yaml (isolated /tmp run) - codex_20251208.yaml (sandbox_permissions=[]) - goose_20251206.yaml (honest by default) - Regenerate fig1 and fig2 with updated data (Codex 64%, Claude 48%, Goose 17%) - Clean up experiment_1_run_evaluations.ipynb (remove duplicate analysis cells) - Move shell scripts to notes/ directory 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Switch from Goose agent to Codex agent for model comparison: - gpt-5 (baseline, symlinked from compare_agents) - gpt-5-mini (pending evaluation) - gpt-5-nano (pending evaluation) Changes: - Add config files for Codex with gpt-5, gpt-5-mini, gpt-5-nano - Update experiment_2_run_evaluations.ipynb for Codex workflow - Update experiment_2_cross_model_analysis.ipynb for Codex results - Create results/compare_models/ with symlink to GPT-5 baseline 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Use correct Codex-specific model names available with ChatGPT account: - gpt-5.1-codex-max (baseline) - gpt-5.1-codex-mini (smaller model) - gpt-5.1-codex (standard model) Rename config files: - codex_gpt5_mini -> codex_gpt51_codex_mini - codex_gpt5_nano -> codex_gpt51_codex Add sandbox_permissions=[] to all configs to disable filesystem access. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add truncated YAML result files for gpt-5.1-codex and gpt-5.1-codex-mini (original files too large for GitHub: 398MB and 8.1GB respectively) - Update notebook to reference truncated files instead of originals - Add version manifest documenting experiment configuration - Add/update Experiment 2 visualization figures Results summary: - gpt-5.1-codex-max: 64% pass rate - gpt-5.1-codex: 58% pass rate - gpt-5.1-codex-mini: 53% pass rate 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Changes
Evaluation Results
Codex CLI achieved ~84% pass rate on the MCP literature evaluation, comparable to Claude Code and Goose CLI.
🤖 Generated with Claude Code