German: Investigate removing -ge- and -zu-

[ojwb]

Splitting out from comment in #139:

I think we can however probably handle the related case of removing infix -ge- (and also -zu-). It seems there are a fixed and fairly small set of possible separable verb prefixes and if we check for one of these followed by ge or zu as a new step after removing suffixes (and skip this new step if we removed a suffix which means the word can't be a verb or derived from a verb) then the results look plausible. This gives ~260 merges for ge and ~220 for zu. I haven't checked more than the first few so it might be we need to exclude some prefixes or adjust the conditions to avoid affecting some cases we aren't actually an infix after a separable verb prefix but look like one.

[ojwb]

I've found a problem case for -ge- removal, which is verbs which start which ge- after the separable prefix, e.g. ausgehen and wiedergeben, which are also cases which make safely removing a ge- prefix difficult - it seems those two problems are closely connected. There is at least one difference though - in the infix case we know that -ge- could only have been added if the word looks like a past participle. I'm not sure that actually helps much though since the possible endings for a past participle (-t, -et, -en) also occur as endings for other verb forms. Perhaps Martin's conclusion that removing ge- and -ge- are both "almost intractable problems" stands.

Martin does however note:

Infixed zu can be removed algorithmically, but this minor feature is not shown here.

Investigating this, I found a case where it would be incorrect to remove -zu- even if we check for a separable verb prefix before it: zusammenzucken. This is the only example I've spotted, but I haven't looked much. The key question is whether there's an example where removal would cause unwanted conflation or prevent desirable existing conflation (for this case, zusammenck is not linguistically correct but is OK as the stem for our purposes since it doesn't collide with the stem of anything unrelated, and other forms which are currently conflated with zusammenzucken will also get -zu- removed and so will still get conflated).

I'm going to leave this for now.

[ojwb]

Investigating this, I found a case where it would be incorrect to remove -zu- even if we check for a separable verb prefix before it: zusammenzucken. This is the only example I've spotted, but I haven't looked much. The key question is whether there's an example where removal would cause unwanted conflation or prevent desirable existing conflation (for this case, zusammenck is not linguistically correct but is OK as the stem for our purposes since it doesn't collide with the stem of anything unrelated, and other forms which are currently conflated with zusammenzucken will also get -zu- removed and so will still get conflated).

Maybe this could be addressed by checking the word after the -zu- - e.g. if we require a vowel then we don't mangle the stem of zusammenzucken (note we've already removed -en at this point):

( ['zu'] goto v delete )

Comparing the results on our test vocab with and without this tweak reveals more cases where the simpler rule removes -zu- when it's not an infix, and some of these were resulting in undesirable collisions and splits:

• hinzu conflated with hin
• anzuge, anzüge, etc conflated with Anger (surname)
• Auszug split from Auszüge (similarly for vor-, zu-)

Current patch:

diff --git a/algorithms/german.sbl b/algorithms/german.sbl
index ded450c1..5bea2c92 100644
--- a/algorithms/german.sbl
+++ b/algorithms/german.sbl
@@ -161,5 +161,19 @@ define stem as (
     do mark_regions
     backwards
         do standard_suffix
+    do (
+        substring among (
+            // separable:
+            'ab' 'an' 'auf' 'aus' 'bei' 'ein' 'los' 'mit' 'nach' 'her' 'hin'
+            'vor' 'weg' 'zu' 'zur{u"}ck'
+            'fort'
+            'zusammen'
+            // can be separable or inseparable:
+            'durch' 'hinter' '{u"}ber' 'um' 'unter' 
+            'wieder'
+            'weiter'
+                 ( ['zu'] goto v delete )
+        )
+    )
     do postlude
 )

This needs more thorough checking, but we may now have a workable rule for -zu- removal. This results in:

• A total of 353 words changed stem
• 125 words changed stem but aren't interesting
• 222 merges of groups of stems:

The main downside is that this among has to be tested for every word. I notice every prefix has a vowel, as does -zu- and we require one afterwards, so there must be at least 3 vowels, so adding $(p2 < limit) before the new substring will exclude some cases which can't possibly match (14027 out of 35033 in our sample vocabulary, though probably a higher proportion once adjusted for frequency as more frequent words are usually shorter).

This is not a complete list of separable prefixes, but we don't even have examples for all the above in our sample vocabulary currently (e.g. no hinter-zu-). We should look at a larger wordlist to determine which actually occur in practice with -zu-.