feat(evals): Evals for SDK install skills - #181
Closed
dingsdax wants to merge 9 commits into
Closed
Conversation
Add vitest-evals based eval system for testing SDK install skills against realistic fixture projects with real Claude API calls. Includes sandboxed runner with file tools (no shell/network), LLM judge for soft assertions, auto-discovered scenarios, and CI workflow gated on eval/skill path changes or run-evals label. Initial scenario: sentry-ruby-sdk rails-basic (Rails 8, Puma, SQLite). Assertions validated against real telemetry using a file-writing transport. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Tests the non-Rails code path: sentry-ruby only, Rack middleware in config.ru, no Rails initializer. Validated with file-writing transport (event + transaction envelopes confirmed). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds a scenario that validates the agent detects Sidekiq in a Rails 8 project and recommends sentry-sidekiq alongside sentry-rails. Verified with a custom file transport that all three telemetry types (event, transaction, log) flow correctly. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Prefix config keys with `config.` to ensure they appear as actual config lines rather than in comments. Keep soft assertions to one per scenario for semantic detection only. Relax sinatra file placement to allow extracted initializer files. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add "Writing good assertions" section covering general principles and SDK-install-specific patterns. Update existing examples to use config-prefixed keys consistently. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Tests that the agent detects an existing Rollbar setup, removes it, and replaces it with Sentry including migrating Rollbar API calls in application code to Sentry equivalents. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Tests that the agent detects an existing OpenTelemetry setup and chooses the OTLP integration path — config.otlp.enabled instead of config.traces_sample_rate, plus sentry-opentelemetry gem. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add separate try/catch for JSON.parse in the judge so unparsable LLM output gets a clear error message instead of a generic throw. Add .default([]) to JudgeResponseSchema arrays so scenarios with no negative_assertions don't silently score 0 when the LLM omits the field. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
dingsdax
changed the title
Author
|
closing:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Evals for SDK install skills
Built on vitest-evals; runs SDK install skills against fixture projects with (for now) only Claude API calls and asserts on resulting files.
How it works
Ruby SDK install skill PoC
rails-basic,sinatra-basicrails-sidekiqrails-rollbar-replacerails-otel