Skip to content

代码是这么写的?预先定义好句号的token id是什么意思,只能测试GLM呗????? #14

@kkwhale7

Description

@kkwhale7

period_tokens = [30930]

搞了半天测试,原来needle一直在context开头,无语了
image
搞不懂,既然你们只想要30930作为句号的tokenizer,你们开源repo的意义是什么????????????????????????????????????????????????????

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions