Add repetition test #1339
Add repetition test #1339
Conversation
| ngram_len_threshold = 10 | ||
| repeat_threshold = 5 |
There was a problem hiding this comment.
Are these values standard or should they be adjustable? If possible, a comment explaining why these values were chosen.
There was a problem hiding this comment.
This is for internal test, so I just treat these numbers to start with.
| @@ -154,5 +173,6 @@ def main(args): | |||
| parser.add_argument('-c', '--chat_template', type=str, default='', help='Chat template to use for the prompt. User input will be injected into {input}') | |||
| parser.add_argument('-s', '--system_prompt', type=str, default='You are a helpful assistant.', help='System prompt to use for the prompt.') | |||
| parser.add_argument('-r', '--rewind', action='store_true', default=False, help='Rewind to the system prompt after each generation. Defaults to false') | |||
| parser.add_argument('-ret', '--repetition_test', action='store_true', default=False, help='Perform repetition test. Defaults to false') | |||
There was a problem hiding this comment.
Is it a test or more like 'repetition_metrics'? At first I thought this PR was adding a unit test.
There was a problem hiding this comment.
This test here is for debugging purposes. We want to test whether the repetition happens in output, for the ongoing investigation of some of models generated repetitions after some point.
There was a problem hiding this comment.
Should this be in our public examples, if it's for testing perhaps this should be in test\python?
There was a problem hiding this comment.
do we expect public users to use model-chat.py? If so, I am fine to use test/python, which file may be good to add these utility?
There was a problem hiding this comment.
Many users are using the posted examples to run with ONNX Runtime GenAI and these examples are linked in many public tutorials as well. We can add this debugging test in the end-to-end unit tests here instead.
| @@ -87,6 +87,8 @@ def main(args): | |||
| system_tokens = tokenizer.encode(system_prompt) | |||
| generator.append_tokens(system_tokens) | |||
| system_prompt_length = len(system_tokens) | |||
| ngram_len_threshold = 10 | |||
There was a problem hiding this comment.
| ngram_len_threshold = 10 | |
| ngram_threshold = 10 |
No description provided.