[Bug]: Stopword removal is ineffective in `SemanticDoubleMergingSplitterNodeParser`

[rautaditya2606]

Bug Description

Bug Description

In the SemanticDoubleMergingSplitterNodeParser text splitter, the internal _clean_text_advanced method attempts to clean strings and remove stopwords before calculating sentence similarity (when using the SpaCy backend).

However, stopword removal is currently completely ineffective due to a tokenization bug:

1. Punctuation is stripped from the text first:

   text = text.translate(str.maketrans("", "", string.punctuation))

2. The method then tokenizes the text using a sentence tokenizer:

   tokens = globals_helper.punkt_tokenizer.tokenize(text)

3. Since all punctuation has already been stripped, the sentence tokenizer treats the entire text block as a single sentence. Thus, tokens contains only one element—the entire string.
4. Consequently, the check w not in self.language_config.stopwords is performed against the entire text block as a single string (rather than word-by-word), so individual stopwords are never filtered out.

Version

0.14.23

Steps to Reproduce

Run the following script to observe that stopwords (like this, is, a, some, the, and) are not removed at all by _clean_text_advanced:

from llama_index.core.node_parser.text.semantic_double_merging_splitter import SemanticDoubleMergingSplitterNodeParser
from llama_index.core.utils import globals_helper

# Instantiate the splitter
splitter = SemanticDoubleMergingSplitterNodeParser()

# Manually load the default English stopwords (simulating the behavior of load_model())
splitter.language_config.stopwords = set(globals_helper.stopwords)

input_text = "this is a test text containing some stopwords like the and a"
cleaned_text = splitter._clean_text_advanced(input_text)

print("Original text:", input_text)
print("Cleaned text :", cleaned_text)

# Expected: "test text containing stopwords like" (stopwords removed)
# Actual  : "this is a test text containing some stopwords like the and a" (no change)
assert cleaned_text != input_text, "Stopwords were not removed!"

Relevant Logs/Tracebacks

[dosubot]

Hi @rautaditya2606! I'm Dosu and I'm helping the LlamaIndex team.

I can confirm this bug. Looking at the _clean_text_advanced method [1], the issue is exactly as you described — globals_helper.punkt_tokenizer is a sentence tokenizer (NLTK Punkt), not a word tokenizer. After punctuation is stripped on line 393, the Punkt tokenizer has no sentence-boundary cues left, so it returns the entire text as a single "sentence." The stopword check then compares that whole string against individual stopwords, which never matches.

A fix would be to replace the Punkt sentence tokenizer with simple word-level tokenization, e.g.:

tokens = text.split()
filtered_words = [w for w in tokens if w not in self.language_config.stopwords]

Since punctuation has already been removed and the text is lowercased, a plain str.split() is sufficient for word tokenization here. There don't appear to be any existing PRs addressing this yet.

To reply, just mention @dosu.

---

Docs are dead. Just use Dosu.