Skip to content

Add add_end_token to the Qwen Models #2481

Open
@pbontrager

Description

@pbontrager

Llama3 Tokenizer was updated to support add_end_token in tokenize_messages to support correct generation #1494. These changes need to be made to Qwen2 and 2.5.

Changes:

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingcommunity help wantedWe would love the community's help completing this issuegood first issueGood for newcomers

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions