Bolt bug: openmaps Part 1 (fixes small bugs, improves prompts, culls noise) #17
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ncy-enforcing state machine
Domiii
changed the title
openmaps Part 1
Domiii
commented
Jan 20, 2025
| text = """ | ||
| You have concluded the analysis. | ||
|
|
||
| IMPORTANT: NOW review, then implement the hypothesized changes using tools. The code is available in the workspace. Start by answering these questions: |
There was a problem hiding this comment.
NOTE: This self-consistency check helps tremendously. I have another TODO that'll fortify this aspect of our state machine.
Domiii
commented
Jan 20, 2025
| @@ -173,6 +173,10 @@ | |||
| parameters={ | |||
| 'type': 'object', | |||
| 'properties': { | |||
| 'problem': { | |||
There was a problem hiding this comment.
NOTE: Accuracy-improving self-consistency.
Domiii
commented
Jan 20, 2025
| @@ -312,11 +312,13 @@ async def _handle_observation(self, observation: Observation) -> None: | |||
| elif isinstance(observation, ReplayPhaseUpdateObservation): | |||
| new_phase = observation.new_phase | |||
| if self.state.replay_phase == new_phase: | |||
| raise ValueError( | |||
| f'Unexpected ReplayPhaseUpdateAction. Already in phase: {new_phase}' | |||
| self.log( | |||
There was a problem hiding this comment.
NOTE: Since we ask it to verify its own hypothesis, it sometimes tries to re-submit it, which takes us here. No reason to throw in that case. (Although it is weird because it technically has no longer access to the tool. Might want to debug why it can even this sometime.)
Domiii
commented
Jan 20, 2025
| asyncio.run( | ||
| run_controller( | ||
| config=config, | ||
| initial_user_action=initial_user_action, | ||
| sid=sid, | ||
| exit_on_message=True, | ||
| fake_user_response_fn=fake_user_response, |
There was a problem hiding this comment.
NOTE: Sometimes it enters the AWAITING_USER_INPUT state. This fixes it. resolve_issue.py does the same.
Domiii
commented
Jan 20, 2025
| suffix = f'* This bug had a timetravel debugger recording.\n* Use below `Initial Analysis` and the timetravel debugger `inspect-*` tools to find the bug.\n* Once found, `submit-hypothesis`, so your analysis can be used to implement the solution.\n\n## Initial Analysis\n{json.dumps(data, indent=2)}' | ||
| suffix = """ | ||
| # Instructions | ||
| 0. Take a look at below `Initial Analysis`, based on a recorded trace of the bug. Pay special attention to `IMPORTANT_NOTES`. |
There was a problem hiding this comment.
NOTEs: This works!
- It indeed pays very careful attention to
IMPORTANT_NOTES, which we can now add into analysis data. - It does state the main problem and propose a plan (again: for self-consistency).
Domiii
commented
Jan 20, 2025
| @@ -145,7 +145,9 @@ def setup_initial_env(self) -> None: | |||
| { | |||
| 'REPLAY_API_KEY': self.config.replay.api_key, | |||
| 'REPLAY_DEV_MODE': os.environ.get('REPLAY_DEV_MODE', ''), | |||
| 'REPLAY_ENABLE_TOOL_CACHE': os.environ.get('REPLAY_DEV_MODE', ''), | |||
| 'REPLAY_ENABLE_TOOL_CACHE': os.environ.get( | |||
| 'REPLAY_ENABLE_TOOL_CACHE', '' | |||
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes small bugs, improves prompts, culls noise.