-
Notifications
You must be signed in to change notification settings - Fork 26
Implement pre/post processing framework for multi-target workflows #86
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
+1,403
−309
Merged
Changes from all commits
Commits
Show all changes
18 commits
Select commit
Hold shift + click to select a range
2f3603e
Implement pre/post processing framework for multi-target workflows
obie b359ba1
Simplify pre_processing data structure by removing 'output' intermediary
obie fe540a1
Refactor pre/post processing test to respect black box testing princi…
obie db27d23
Refactor all PR tests to be black box and behavioral
obie 26862b3
final touches
obie d20b2e6
update readme
obie 89ff225
rename setup_test_environment
obie a901090
final touches
obie 0ae7197
final cleanup
obie aeb54c0
added some prompts
obie b77c62c
Merge branch 'main' into feature/pre-post-processing
obie dbe519d
add rg to github CI
obie bf8edd8
Merge branch 'main' into feature/pre-post-processing
obie d4e6eb9
Fix targetless workflow test for pre/post processing framework
obie e33e2b6
Improve verbose mode and error reporting for better user experience
obie b7cfb94
Release v0.2.2
obie 90de452
Merge branch 'main' into feature/pre-post-processing
obie 4a75c13
Fix step_executor_coordinator_test to use proper Mocha stubbing
obie File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,111 @@ | ||
# Pre/Post Processing Example: Test Suite Optimization | ||
|
||
This example demonstrates how to use Roast's pre/post processing framework to optimize an entire test suite across multiple files. | ||
|
||
## Overview | ||
|
||
The workflow processes multiple test files, but performs setup and aggregation tasks only once: | ||
|
||
- **Pre-processing**: Runs once before any test files are processed | ||
- Gathers baseline metrics for comparison | ||
- Sets up the test environment | ||
|
||
- **Main workflow**: Runs for each test file matching the target pattern | ||
- Analyzes test quality and coverage | ||
- Improves test coverage | ||
- Optimizes test performance | ||
- Validates changes | ||
|
||
- **Post-processing**: Runs once after all test files have been processed | ||
- Aggregates metrics from all files | ||
- Generates a comprehensive report | ||
- Cleans up the environment | ||
|
||
## Workflow Structure | ||
|
||
```yaml | ||
name: test_optimization | ||
model: gpt-4o | ||
target: "test/**/*_test.rb" | ||
|
||
pre_processing: | ||
- gather_baseline_metrics | ||
- setup_test_environment | ||
|
||
steps: | ||
- analyze_test_file | ||
- improve_test_coverage | ||
- optimize_test_performance | ||
- validate_changes | ||
|
||
post_processing: | ||
- aggregate_metrics | ||
- generate_summary_report | ||
- cleanup_environment | ||
``` | ||
|
||
## Directory Structure | ||
|
||
``` | ||
pre_post_processing/ | ||
├── workflow.yml | ||
├── pre_processing/ | ||
│ ├── gather_baseline_metrics/ | ||
│ │ └── prompt.md | ||
│ └── setup_test_environment/ | ||
│ └── prompt.md | ||
├── analyze_test_file/ | ||
│ └── prompt.md | ||
├── improve_test_coverage/ | ||
│ └── prompt.md | ||
├── optimize_test_performance/ | ||
│ └── prompt.md | ||
├── validate_changes/ | ||
│ └── prompt.md | ||
└── post_processing/ | ||
├── aggregate_metrics/ | ||
│ └── prompt.md | ||
├── generate_summary_report/ | ||
│ └── prompt.md | ||
└── cleanup_environment/ | ||
└── prompt.md | ||
``` | ||
|
||
## Key Features Demonstrated | ||
|
||
1. **Shared State**: Pre-processing results are available to all subsequent steps | ||
2. **Result Aggregation**: Post-processing has access to results from all workflow executions | ||
3. **One-time Operations**: Setup and cleanup happen only once, regardless of target count | ||
4. **Metrics Collection**: Each file's results are stored and aggregated for reporting | ||
|
||
## Running the Example | ||
|
||
```bash | ||
cd examples/pre_post_processing | ||
roast workflow.yml | ||
``` | ||
|
||
This will: | ||
1. Run pre-processing steps once | ||
2. Process each test file matching `test/**/*_test.rb` | ||
3. Run post-processing steps once with access to all results | ||
4. Generate a comprehensive optimization report | ||
|
||
## Use Cases | ||
|
||
This pattern is ideal for: | ||
- **Code migrations**: Setup migration tools, process files, generate migration report | ||
- **Performance audits**: Baseline metrics, analyze files, aggregate improvements | ||
- **Documentation generation**: Analyze codebase, generate docs per file, create index | ||
- **Dependency updates**: Check current versions, update files, verify compatibility | ||
- **Security scanning**: Setup scanners, check each file, generate security report | ||
|
||
## Customization | ||
|
||
To adapt this example for your use case: | ||
|
||
1. Update the `target` pattern to match your files | ||
2. Modify pre-processing steps for your setup needs | ||
3. Adjust main workflow steps for your processing logic | ||
4. Customize post-processing for your reporting requirements | ||
5. Use appropriate AI models for each step type |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,23 @@ | ||
# Analyze Test File | ||
|
||
Current test file: {{file}} | ||
|
||
Please analyze this test file and identify: | ||
|
||
1. **Test Structure**: Number of test cases, test suites, and overall organization | ||
2. **Coverage Gaps**: Areas of the code that aren't adequately tested | ||
3. **Test Quality Issues**: | ||
- Tests that are too brittle or implementation-dependent | ||
- Missing edge cases | ||
- Unclear test descriptions | ||
- Excessive mocking that reduces test value | ||
4. **Performance Issues**: | ||
- Slow setup/teardown methods | ||
- Inefficient test data generation | ||
- Unnecessary database operations | ||
5. **Opportunities for Improvement**: | ||
- Tests that could be parameterized | ||
- Common patterns that could be extracted to helpers | ||
- Better use of test fixtures or factories | ||
|
||
Provide specific, actionable recommendations for each issue found. |
17 changes: 17 additions & 0 deletions
17
examples/pre_post_processing/improve_test_coverage/prompt.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,17 @@ | ||
# Improve Test Coverage | ||
|
||
Based on the analysis of {{file}}, implement the following improvements: | ||
|
||
1. **Add Missing Test Cases**: Write tests for uncovered code paths, edge cases, and error conditions | ||
2. **Improve Test Descriptions**: Make test names more descriptive and follow consistent naming conventions | ||
3. **Enhance Assertions**: Add more specific assertions to catch regressions | ||
4. **Test Data**: Use more realistic test data that better represents production scenarios | ||
5. **Remove Redundancy**: Eliminate duplicate tests or merge similar ones | ||
|
||
Generate the improved test code and explain the rationale for each change. | ||
|
||
Remember to: | ||
- Maintain backward compatibility with existing test interfaces | ||
- Follow the project's testing conventions and style guide | ||
- Ensure new tests are fast and deterministic | ||
- Add appropriate comments for complex test scenarios |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In a future PR, we probably want to adopt the
dev
way here and have a key users can use to verify a requirement has been met and not need to run this command every time.