Support for Slovene language

[filips123]

Is it possible to add official support for Slovene stemming algorithm to Snowball?

Martin Porter started working on Slovene stemmer in 2005, but never finished it because it had some problems. That stemmer could probably be used as a starting point.

I found some papers about Slovene stemming that might be useful:

• Stemming of Slovenian library science texts: https://www.researchgate.net/publication/50392133_Stemming_of_Slovenian_library_science_texts
• The effectiveness of stemming for natural-language access to Slovene textual data: https://asistdl.onlinelibrary.wiley.com/doi/abs/10.1002/%28SICI%291097-4571%28199206%2943%3A5%3C384%3A%3AAID-ASI6%3E3.0.CO%3B2-L
• Processing of documents and queries in a Slovene language free text retrieval system: https://academic.oup.com/dsh/article-abstract/5/2/182/943275 (this one is actually referenced in the Snowball introduction)

I'm not familiar with now Snowball algorithms work, but here are my suggestions for some of the questions for the original algorithm:

Would not sloven (or slov), be a more desirable stem in this case?

I think "sloven" would be the most appropriate.

Another point. I notice a common -ah suffix, which you have not removed, as for example here [...] besedah besedah [...] Could this be added to the list of suffixes?

I don't know which other things that will affect, but -ah suffix should be removed in cases like "besedah".

[ojwb]

More recently in https://lists.tartarus.org/pipermail/snowball-discuss/2012-May/001387.html Martin said:

We never took the contributed stemmer on board as we were unable to resolve certain questions, such as how it differs from the published Popovic/Willett stemmer. I'd always hoped someone would send us the Popovic/Willett stemmer in snowball, but this has not happened!

Reviewing the code in those old emails, it unfortunately really does not look suitable for merging as-is, and I'd agree with the conclusion that an implementation of the Popovič/Willett stemmer seems like it would do a much better job.

I found some papers about Slovene stemming that might be useful: [...]

Two of these are by Popovič/Willett and seem to be discussing the same algorithm. The 1992 paper is more about evaluation, while the 1990 paper has a description of the algorithm but unfortunately this refers to a "suffix list" which doesn't seem to be in either paper. Earlier it says "The resulting longest-match, context-sensitive algorithm is based on the use of 2,086 endings [...]", so it looks like we're missing a list of 2,086 suffixes. There are a few example suffixes in the paper, but it's not going to be easy to rediscover around 2000 suffixes that aren't listed, along with their minimum stem lengths, etc.

The 1990 paper looks to have useful details of how to rewrite stems (tables 5, 6, 7 and the text which refers to them), which would probably be useful for developing an algorithm to stem Slovene, but that's much more of an undertaking than implementing a stemmer from a complete description and/or an existing implementation in another language.

I don't know if an existing implementation of the Popovič/Willett stemmer is available - 1990 is before it was common to publish code for a paper, but e.g. we have a C implementation from the authors of the Kraaij-Pohlmann Dutch stemmer and the papers for that were 1994 and 1995. I failed to find one with a few minutes web searching.

The other paper you mention is:

Stemming of Slovenian library science texts: https://www.researchgate.net/publication/50392133_Stemming_of_Slovenian_library_science_texts

The main text of this is in Slovene, but from reading the English abstract I'm wondering if their stemmer is a machine learning system trained to stem Slovene, which would be hard to implement a Snowball stemmer from. However they compare their stemmer against two others, one of which is "The Popovič stemmer", so it seems they must have had access to an implementation of that, or enough information to implement it themselves. Does the Slovene text of this paper give more information about that? And is their stemmer based on machine learning?

[filips123]

And is their stemmer based on machine learning?

It doesn't seem to be a proper machine learning, more like some "dynamic" algorithm that thanges its stem lists. It seems it's supposed to be customized for each domain of texts (for example, with domain-specific terminology) before the first use, and while using the algorithm, the "system administrator" should review the three stem lists and reorder found stems into the correct lists:

Algoritem sodi v družino algoritmov, ki odstranjujejo pone, natančneje besedne
končnice. Deluje po načelu najdaljšega ujemanja, a tako, da poskuša vedno
ohraniti najdaljši možni niz znakov, ki nato zastopa oblikoslovne različice se-
mantično povezanih besed. Pri najkrajši možni različici namreč prehitro lahko
pride do premočnega krnjenja. Dodatna ideja avtorjev je bila, da je potrebno
krnjenje bolj prilagoditi strokovnim podjezikom. Iz tega drugega razloga je za
delovanje algoritma vedno v začetku potrebna t. i. učna faza, v kateri pridobi-
mo nabor krnov, ki izvirajo iz terminologije nekega področja. Tako zgradimo
sestavne dele algoritma, ki jih moramo v nadaljnjih postopkih seveda redno
dopolnjevati in vzdrževati.

Najpomembnejše v celotnem postopku je vzdrževanje seznama veljavnih kr-
nov, ki vsebuje nabor besed oz. krnov podjezika, in seznama imenskih krnov.
Seznama naraščata z vsakim obdelanim dokumentom in tako se algoritem “uči”
podjezika. Dela s seznami je z naraščanjem zbirke dokumentov vedno manj,
kajti vse več izrazov se že nahaja v enem izmed njiju, zato jih je vse manj v
seznamu novih krnov. Administrator sistema določa najustreznejšo obliko
krnov. Ni nujno, da se pri pregledovanju seznama novih krnov odloči za ob-
liko krna, ki jo za določeno besedo predlaga algoritem. S pomočjo spiska vseh
krnov, ki so nastali v zaporednih stopnjah krnjenja, izbere tistega, ki je po
njegovem najustreznejši. Če ima občutek za jezik, obstaja velika verjetnost, da
bo tak krn res optimalen.

So yeah, it probably isn't applicable to Snowball, unless some generic static lists could be created and function good enough.

Does the Slovene text of this paper give more information about that?

Unfortunately, not. It just cites two Popovič papers, and a few more:

1. Adamič, Š. (1995). Temelji biostatistike. Ljubljana: Medicinska fakulteta.
2. Dimec, J., Todorovski, L., Hristovski, D., & Džeroski, S. (1999). The personalized search engine for Slovenian and English medical documents. V Managing multimedia collections. 23rd Library systems seminar, Bled, 21-23 April 1999 (str. 56-63). Ljubljana: National and University Library..
3. Dimec, J., Todorovski, L., Hristovski, D., & Džeroski, S. (2000). Three new stemmers for Slovenian language. Pridobljeno 29.2.2000 s spletne strani: http://www.mf.uni-lj.si/ds/new-stemmers.html
4. Popovič, M. (1991). Implementation of a Slovene language free-text retrieval system: a study submitted in fulfilment of the requirements for the degree of Doctor of Philosophy at the University of Sheffield. Sheffield: Department of Information Studies. Vilar, P.; Maver, J. Krnjenje slovenskih besedil s področja bibliotekarstva Knjižnica 46(2002)1-2, 111-136 136
5. Popovič, M., & Willett, P. (1992). The effectiveness of stemming for natural language access to Slovene textual data. Journal of the American Society for Information Science, 43 (5), 384-390.
6. Vilar, P., & Dimec, J. (2000). Krnjenje kot osnova nekaterih nekonvencionalnih metod poizvedovanja. Knjižnica, 44 (4), 7-31.
7. Vilar, P. (2001). Krnjenje slovenskih besedil s področja bibliotekarstva. Magistrsko delo. Ljubljana: Oddelek za bibliotekarstvo, informacijsko znanost in knjigarstvo.

[ojwb]

OK, thanks.

I'll try emailing Martin Porter (he's retired from Snowball development now) - he found the C sources to the Kraaij-Pohlmann stemmer in an old backup so perhaps he has this one too.

I've also found an address for Peter Willett who has also now retired but still seems to be publishing the occasional paper so is still somewhat active - I can try asking him too.

If that doesn't get us anywhere, we could ask the authors of the more recent paper as it seems they must have had an implementation.

[ojwb]

Martin doesn't have code or the suffix list, but spotted the title of Popovič's PhD thesis is in the 1992 paper's references and with the title I found that online:

https://www.dlib.si/details/URN:NBN:SI:DOC-5WB6EMGL?&language=eng

That says the suffix list is on an attached floppy disk. I'll contact Peter Willett.

[ojwb]

Peter Willett found the files:

popovic-suffix-list.txt
popovic-stopword-list.txt

Both these lists are pure ASCII, with diacritics encoded using ^, [ and @.

https://en.wikipedia.org/wiki/Slovene_alphabet#Computer_encoding says:

In TeX notation, č, š and ž become \v c, \v s, \v z, \v{c}, \v{s}, \v{z} or in their macro versions, "c, "s and "z, or in other representations as \~, \{, \' for lowercase and \^, \[, \@ for uppercase.

It seems "č, š and ž become [...] \^, \[, \@ for uppercase" (ignoring the backslashes) describes the encoding used here.

Before I spotted this in Wikipedia I independently came up with the same mapping, which gives me additional confidence it is the right one. My reasoning was: these punctuation can appear at the start or end of a suffix or stopword so it seems they don't mean an accent on the letter before or after. There are three common accented letters (č, š and ž) so I guessed a simple one-to-one mapping. Looking at stopword entries, DALE^ seems to be daleč (far), BOLJ[I seems to be boljši (better) and KRI@EMKRA@EM seems to be križemkražem (the game of Scrabble - presumably the stopword list was prepared from documents which mention it a lot). For each of these words, substituting either of the other accented letters does not seem to make a common word (I don't read Slovene so I just checked in online dictionaries).

Peter also shared that sadly Mirko died in 1992, so not long after his work on this: https://sl.wikipedia.org/wiki/Mirko_Popovi%C4%8D

@filips123 If you want to have a go at implementing this, please do and I can review. If not I can code it up at some point then get you to check that it's working sensibly. I should prioritise dealing with merging the stemmers already in the process of being reviewed though.

[filips123]

Unfortunately, I don't have experience with Snowball and time to write the algorithm. It's fine if it waits until you (or someone else) has the time to implement it. I can then help test it.