feat: VLLM provider, BrowseComp benchmark, and OpenCode fixes#8
Open
noahpro99 wants to merge 125 commits into
Open
feat: VLLM provider, BrowseComp benchmark, and OpenCode fixes#8noahpro99 wants to merge 125 commits into
noahpro99 wants to merge 125 commits into
Conversation
…cy flag to set concurrency.
- Add opencode CLI integration for running agents via `opencode run --format json` - Per-agent SDK detection (Claude SDK vs opencode) based on options type - SEAL-QA evaluation script with positional train/val split - SEAL-QA scorer using Gemini 2.5 Flash as LLM-as-judge grader - SLURM batch script for running on ARC cluster - Robust git checkout with stash fallback and force-checkout - Error handling in loop iterations to prevent silent exits - Fall back to raw result text when structured output is missing Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…skill export - Add opencode CLI agent path in base.py for non-Claude SDK models - Add run_sealqa_opencode.py with --no-skills, --model, --provider flags - Add skill hiding (baseline) and install-from-branch support for evals - Export best frontier skills back to working branch after loop completes - Add sbatch scripts for SEAL-QA eval and loop jobs - Add opencode.json with Gemini, ARC, and Together AI providers - Update sealqa_agent to support both Claude and opencode SDK paths Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…skill export - Add opencode CLI agent path in base.py for non-Claude SDK models - Add run_sealqa_opencode.py with --no-skills, --model, --provider flags - Add skill hiding (baseline) and install-from-branch support for evals - Export best frontier skills back to working branch after loop completes - Add sbatch scripts for SEAL-QA eval and loop jobs - Add opencode.json with Gemini, ARC, and Together AI providers - Update sealqa_agent to support both Claude and opencode SDK paths Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Replace fragile hide_skills() with prepare_run_dir() that creates isolated .evoskill-runs/<session>/ dirs with their own .claude/skills/ - Baseline runs get empty skills dir, evolved runs get skills copied in - Both can now run in parallel safely on shared filesystems - Add run_dir support in opencode agent subprocess (base.py cwd param) - Add --session CLI arg to name run dirs - Add .evoskill-runs/ to .gitignore Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
opencode tool for web search and fetch
- Supports seal-0, officeqa, dabstep, livecodebench via dataset_path - Uses stratified split with configurable train/val ratios - Exports best skills to starting branch after loop completes - Update sbatch_loop to use comb script Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…sions Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…seline Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
All load_* functions now use split_held_out() instead of calling stratified_split directly. This prevents forgetting to split (like load_livecode was doing). LiveCodeBench now properly splits by difficulty into train/val/test. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Without this, the loop uses the full dataset for train/val while eval uses a sliced subset, causing train/test overlap. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…() from scorer inputs Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Sonnet and other thorough models need more time to research and test. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This reverts commit 61ea923.
- Add load_browsecomp to load_dataset.py with stratified split - Wire browsecomp.csv into run_eval.py and run_loop.py - Uses sealqa_agent (WebSearch + WebFetch) and sealqa_scorer (LLM judge) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Proposer and generator agents now respect --model and --provider CLI args instead of always using Claude SDK default. Each agent profile has a make_*_options(model, provider) factory that returns either ClaudeAgentOptions or opencode dict based on the SDK setting. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR introduces several enhancements to EvoSkill:
New Features
Improvements
Technical Details
The changes maintain backward compatibility while extending the framework's capabilities for more diverse agent workflows.