Feature/best of (dev)#453
Open
kkzxak47 wants to merge 32 commits into
Open
Conversation
Reviewer's Guide by SourceryThis pull request introduces a 'best_of' setting to the Model class, enabling the generation of multiple completions and selection of the best one based on accumulated log probabilities. It also implements batched inference using a new BatchedLLM class to support parallel generation of multiple sequences. Additionally, it adds a setting in the settings window to control the 'best_of' parameter and includes comprehensive tests for the new functionality. No diagrams generated as the changes look simple and do not need a visual representation. File-Level Changes
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
There was a problem hiding this comment.
Hey @kkzxak47 - I've reviewed your changes - here's some feedback:
Overall Comments:
- Consider adding a brief explanation of the
best_ofparameter in the class docstring. - The logic for choosing the best response could be extracted into a separate helper function for better readability.
Here's what I looked at during the review
- 🟢 General issues: all looks good
- 🟢 Security: all looks good
- 🟡 Testing: 1 issue found
- 🟡 Complexity: 2 issues found
- 🟢 Documentation: all looks good
Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.
| top_p: float = 0.95, | ||
| repeat_penalty: float = 1.1, | ||
| # you can override the best_of setting here | ||
| best_of: int | None = None, |
There was a problem hiding this comment.
issue (complexity): Consider extracting the best-of evaluation logic and logging into separate helper functions to reduce nesting and improve readability of the generate_best_of_response function, such as creating a function to compute the logprob sum and content from the response dicts, and using it in the main function to simplify the logic there .
Consider extracting separate helper functions to encapsulate the best‐of evaluation logic and logging. For example, you can split out the token logprob summing and sorting logic into a dedicated function, which will reduce nesting in generate_best_of_response.
Suggested changes:
- Extract a helper to compute the logprob sum and content:
def _extract_result(response: dict) -> tuple:
"""Extract the sum of token logprobs and the generated content."""
token_logprobs = response["choices"][0]["logprobs"]["token_logprobs"]
content = response["choices"][0]["message"]["content"]
return (sum(token_logprobs), content)- Refactor
generate_best_of_responseto use the helper:
def generate_best_of_response(
self,
prompt: str,
max_tokens: int = 50,
temperature: float = 0.1,
top_p: float = 0.95,
repeat_penalty: float = 1.1,
best_of: int | None = None,
) -> str:
try:
messages = [{"role": "user", "content": prompt}]
_best_of = best_of or self.config.get("best_of") or 1
logprob_args = {}
if _best_of > 1:
logprob_args = {"logprobs": True, "top_logprobs": 1}
responses = [
self.model.create_chat_completion(
messages,
max_tokens=max_tokens,
temperature=temperature,
top_p=top_p,
repeat_penalty=repeat_penalty,
**logprob_args
)
for _ in range(_best_of)
]
logger.debug(f"Responses: {responses}")
if _best_of > 1:
results = [_extract_result(response) for response in responses]
logger.debug(f"Results: {results}")
results.sort(key=lambda x: x[0])
else:
results = [(0, responses[0]["choices"][0]["message"]["content"])]
best_response = results[-1]
self.model.reset()
logger.debug(f"Selected Response: {best_response}")
return best_response[1]
except Exception as e:
print(f"GPU inference error ({e.__class__.__name__}): {str(e)}")
return f"({e.__class__.__name__}): {str(e)}"This refactor maintains functionality while reducing nesting and increasing readability.
|
With recent work do we still want to merge this @kkzxak47 ? |
Collaborator
Author
The current implementation is relatively slow, so I'm exploring an optimization. We don't need to merge it right away. It can be merged later. |
There was a problem hiding this comment.
Hey @kkzxak47 - I've reviewed your changes - here's some feedback:
Overall Comments:
- Consider adding a brief explanation of the batched inference approach in the description.
- The new
BatchedLLMclass introduces a lot of complexity; ensure it's thoroughly tested, especially edge cases and error handling.
Here's what I looked at during the review
- 🟡 General issues: 1 issue found
- 🟢 Security: all looks good
- 🟢 Testing: all looks good
- 🟡 Complexity: 1 issue found
- 🟢 Documentation: all looks good
Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.
There was a problem hiding this comment.
Hey @kkzxak47 - I've reviewed your changes - here's some feedback:
Overall Comments:
- Consider adding a brief description of the new
BatchedLLMclass to the docstring. - The
_select_best_resultmethod could be a standalone function instead of a static method.
Here's what I looked at during the review
- 🟡 General issues: 3 issues found
- 🟢 Security: all looks good
- 🟢 Testing: all looks good
- 🟡 Complexity: 1 issue found
- 🟢 Documentation: all looks good
Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.
Comment on lines
+262
to
+263
| if 0 <= index < 5: | ||
| return index + 1 |
There was a problem hiding this comment.
issue (code-quality): We've found these issues:
- Lift code into else after jump in control flow (
reintroduce-else) - Replace if statement with if expression (
assign-if-exp)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Added a
best_ofsetting, when it is larger than 1, # of completions will be generated and the best one will be returned (the one with the largest mean logprobs).Implemented batched inference
Summary by Sourcery
Implements a 'best_of' setting to generate multiple completions and return the best one based on accumulated log probabilities. This enhances the quality of generated responses by allowing the model to explore multiple possibilities and select the most likely one.
New Features:
Tests:
Summary by Sourcery
Implement a 'best_of' feature for generating multiple text completions and selecting the most probable one
New Features:
Enhancements:
Tests:
Chores: