Skip to content
This repository was archived by the owner on Sep 18, 2024. It is now read-only.
This repository was archived by the owner on Sep 18, 2024. It is now read-only.

char-level Tokenizer.sequences_to_texts() insert additional SPACEs #346

@XiYuan68

Description

@XiYuan68
  • Check that you are up-to-date with the master branch of keras-preprocessing. You can update with:
    pip install git+git://github.com/keras-team/keras-preprocessing.git --upgrade --no-deps

  • Provide a link to a GitHub Gist of a Python script that can reproduce your issue (or just copy the script here if it is short).

Describe the problem.

from tensorflow.keras.preprocessing.text import Tokenizer

text = ['abc def']
tokenizer = Tokenizer(char_level=True, split='')
tokenizer.fit_on_texts(text)
sequence = tokenizer.texts_to_sequences(text)
text_after = tokenizer.sequences_to_texts(sequence)

print(text_after)
>>> ['a b c   d e f']

notice that text_after and text are different, additional SPACEs are inserted

Describe the expected behavior.

text_after should be same as text

I believe this line is where the problem is, replacing:

vect = ' '.join(vect)

with

vect = self.split.join(vect)

will fix the bug in my mini case

Metadata

Metadata

Assignees

No one assigned

    Labels

    textRelated to text

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions