"man" and "men" are in stop_words, but "woman" and "women" are not

[grasshoppermouse]

This got me in trouble recently 🙂

Just a heads up for users of this package, and perhaps a reason to remove "man" and "men" from stop_words.

[juliasilge]

Yep! @EmilHvitfeldt and I wrote about this in detail here:
https://smltar.com/stopwords

We note that the stop word lists are typically generated by selecting the most frequent words across a large corpus of text and such a corpus, it turns out, has more representation for text about men than women.

The stop word lists that we make available via tidytext are specific stop word lexicons so it would be inappropriate for us to make changes to them, but I definitely encourage you to customize them for your use case!

[EmilHvitfeldt]

💯 what Julia just said!

You are already doing great by looking at the stop word list! None of the stop word lists will work for every task. Looking at the list is the first step

[grasshoppermouse]

Thanks for the pointer to your book, which is very useful. It lead me to a possible solution for this problem, from the textstem package (which I now see hasn't been updated in 8 years):

library(textstem)
> lemmatize_words("dried")
[1] "dry"
> lemmatize_words("dry")
[1] "spin-dry"

Your book recommended spaCy for lemmatizing. I'll give that a shot. Many thanks again for your rapid responses.