feat: add FeedbackBuilder utility for structured reflective datasets#294
Open
rohitsandadi wants to merge 7 commits into
Open
feat: add FeedbackBuilder utility for structured reflective datasets#294rohitsandadi wants to merge 7 commits into
rohitsandadi wants to merge 7 commits into
Conversation
…le, and constraints
Changed Files
|
|
Hi! I was looking into #288/#264 and noticed this PR already covers the default Score injection path and part of the FeedbackBuilder direction. Since this PR has been idle for a while, would maintainers prefer to keep this PR as the main path, or would a smaller follow-up PR focused only on #288's default Score injection be useful? I’m happy to help with a narrowly scoped PR or tests/docs if that would make this easier to review, and also happy to defer to this PR if it is still the preferred direction. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements
FeedbackBuilder(#264). Addresses #261 (anti-overfit), #262 (task drift), #288 (score visibility in reflective dataset).Summary
FeedbackBuilderis a configurable utility ingepa.utilsthat replacesmake_reflective_dataset()boilerplate. It replicates the existing side_info extraction logic, then layers on optional enrichments:include_score_diff— injects score + gap-from-perfect into each record'sFeedbackfield, so the reflection LM always knows how examples scored (feat: auto-inject score into side_info when not already present #288)rationale_field— extracts a named human-rationale field from side_info into a dedicatedRationalerecord fieldglobal_constraints— appends constraint text to every record, preventing the reflection LM from copying training data (feat: built-in anti-overfit guard to prevent prompt copying training data #261) or drifting task definitions (feat: task definition preservation — prevent optimizer from changing task parameters #262)When no
FeedbackBuilderis configured, the adapter falls back to existing behavior.Changed files
src/gepa/utils/feedback_builder.py— new:FeedbackBuilderdataclass withbuild()methodsrc/gepa/utils/__init__.py— exportFeedbackBuildersrc/gepa/adapters/optimize_anything_adapter/optimize_anything_adapter.py— accept + delegate toFeedbackBuildersrc/gepa/optimize_anything.py— addfeedback_buildertoReflectionConfig, wire to adaptertests/test_feedback_builder.py— 13 unit testsTest plan
ruff check+pyrightclean