tokenizerTest-huggingface

输入测试数据集，修改vocab.txt，输出分词结果

Name		Name	Last commit message	Last commit date
Latest commit History 2 Commits
.gradle		.gradle
.idea		.idea
build		build
gradle/wrapper		gradle/wrapper
src/main		src/main
.gitignore		.gitignore
Desktop.ini		Desktop.ini
README.md		README.md
build.gradle		build.gradle
gradlew		gradlew
gradlew.bat		gradlew.bat
guava-23.6-jre.jar		guava-23.6-jre.jar
settings.gradle		settings.gradle