Irregular results

[stell]

Everything is up to date and freshly indexed. I have two pages containing the word "geokoordinaten" which is German and stands for Geocoordinates.

When searching for "geokoordinaten" i get 2 results, which is correct.

"geokoordina" outputs 0 results.
"geokoo" outputs 2 results again.

On the other hand if searching for "geoko" i get 2 results again, but different pages. Should this not output 4 pages?

I don't know if this is an issue or something is badly setup, but i don't see whats the problem here.

You can try it out here:
https://www.jfewo.de/docs/de/suche?q=geokoordinaten

This is the config:

enabled: true
search_route: /suche
query_route: /s
built_in_css: true
built_in_js: true
built_in_search_page: true
enable_admin_page_events: true
search_type: auto
fuzzy: false
phrases: true
stemmer: german
display_route: true
display_hits: true
display_time: true
live_uri_update: true
limit: "20"
min: "4"
snippet: "300"
index_page_by_default: true
scheduled_index:
  enabled: false
  at: "* 2 * * 1-7"
  logs: logs/tntsearch-index.out
filter:
  items:
    - root@.descendants
powered_by: true
search_object_type: Grav

[nqb]

Hello,

I noticed same kind of issue on YunoHost's documentation.
If I search for "revers", I got 5 pages with "reverse" or "Reverse" results but if I add a "e" (for "reverse"), I got only one result.

[ViliusS]

Everything is up to date and freshly indexed. I have two pages containing the word "geokoordinaten" which is German and stands for Geocoordinates.

When searching for "geokoordinaten" i get 2 results, which is correct.

"geokoordina" outputs 0 results.
"geokoo" outputs 2 results again.

On the other hand if searching for "geoko" i get 2 results again, but different pages. Should this not output 4 pages?

I don't know if this is an issue or something is badly setup, but i don't see whats the problem here.

You can try it out here:
https://www.jfewo.de/docs/de/suche?q=geokoordinaten

This is the config:

enabled: true
search_route: /suche
query_route: /s
built_in_css: true
built_in_js: true
built_in_search_page: true
enable_admin_page_events: true
search_type: auto
fuzzy: false
phrases: true
stemmer: german
display_route: true
display_hits: true
display_time: true
live_uri_update: true
limit: "20"
min: "4"
snippet: "300"
index_page_by_default: true
scheduled_index:
  enabled: false
  at: "* 2 * * 1-7"
  logs: logs/tntsearch-index.out
filter:
  items:
    - root@.descendants
powered_by: true
search_object_type: Grav

Try disabling the stemmer and rebuilding the index after that. See if that helps.

[stell]

Already tried that. Same irregular results.

[ViliusS]

Just tried your link ant it shows 2 same results for "geokoordina" and "geokoo".

[stell]

Cannot confirm.
"geokoordina" outputs 0 results
"geokoo" outputs 2 results
"geoko" outputs 2 (different) results

In fact all of this should output more that 2 results.

[ViliusS]

I don't know enough about German language but the difference between "geokoordina" and "geokoordinat" looks like really a stemmer issue. Setting stemmer to 'no' should return the same results for both queries.

[stell]

Yeah, I thought the same.

[ViliusS]

Without stemmer:

[]# php -r " include('Stemmer.php'); require_once ('NoStemmer.php'); echo TeamTNT\TNTSearch\Stemmer\NoStemmer::stem('geokoordina').PHP_EOL;"
geokoordina

[]# php -r " include('Stemmer.php'); require_once ('NoStemmer.php'); echo TeamTNT\TNTSearch\Stemmer\NoStemmer::stem('geokoordinat').PHP_EOL;"
geokoordinat

[]# php -r " include('Stemmer.php'); require_once ('NoStemmer.php'); echo TeamTNT\TNTSearch\Stemmer\NoStemmer::stem('geokoordinaten').PHP_EOL;"
geokoordinaten

With stemmer:
[]# php -r " include('Stemmer.php'); require_once ('GermanStemmer.php'); echo TeamTNT\TNTSearch\Stemmer\GermanStemmer::stem('geokoordina').PHP_EOL;"
geokoordina

[]# php -r " include('Stemmer.php'); require_once ('GermanStemmer.php'); echo TeamTNT\TNTSearch\Stemmer\GermanStemmer::stem('geokoordinat').PHP_EOL;"
geokoordinat

[]# php -r " include('Stemmer.php'); require_once ('GermanStemmer.php'); echo TeamTNT\TNTSearch\Stemmer\GermanStemmer::stem('geokoordinaten').PHP_EOL;"
geokoordinat

Check if you really disabled the stemmer, i.e. set it to 'no', because at least the last difference is a stemmer issue. Stemming is a complex beast, so to properly debug issue I would disable it for now.

[stell]

Like I said. I tested it without stemmer at the beginning and two more times after. Results are the same.
I tested setting stemmer: no and stemmer: "no" which is saved from admin backend.

[mbirth]

Same here:

But if I add the final "e", I get no results:

My config:

enabled: true
search_route: /search
query_route: /s
built_in_css: true
built_in_js: true
built_in_search_page: true
enable_admin_page_events: true
search_type: auto
fuzzy: true
phrases: true
stemmer: "no"
display_route: true
display_hits: true
display_time: true
live_uri_update: true
limit: '20'
min: '3'
snippet: '300'
index_page_by_default: true
scheduled_index:
  enabled: false
  at: '30 3 * * *'
  logs: logs/tntsearch-index.out
filter:
  items:
    - root@.descendants
  published: true
powered_by: false
search_object_type: Grav

PHP:

PHP 7.4.21 (cli) (built: Jun 29 2021 15:17:15) ( NTS )
Copyright (c) The PHP Group
Zend Engine v3.4.0, Copyright (c) Zend Technologies
    with Zend OPcache v7.4.21, Copyright (c), by Zend Technologies

Grav and TNTSearch are up to date.

[ViliusS]

Did you rebuild index after disabling the stemmer? Delete old index file fully.

Again this looks like a German stemmer issue:
[]# php -r " include('Stemmer.php'); require_once ('GermanStemmer.php'); echo TeamTNT\TNTSearch\Stemmer\GermanStemmer::stem('spectr').PHP_EOL;"
spectr

[]# php -r " include('Stemmer.php'); require_once ('GermanStemmer.php'); echo TeamTNT\TNTSearch\Stemmer\GermanStemmer::stem('spectre').PHP_EOL;"
spectr

[]# php -r " include('Stemmer.php'); require_once ('NoStemmer.php'); echo TeamTNT\TNTSearch\Stemmer\NoStemmer::stem('spectr').PHP_EOL;"
spectr

[]# php -r " include('Stemmer.php'); require_once ('NoStemmer.php'); echo TeamTNT\TNTSearch\Stemmer\NoStemmer::stem('spectre').PHP_EOL;"
spectre

[mbirth]

Of course, I've rebuilt the index several times. And the German stemmer was never activated at any time. I've even deleted the index files before reindexing to make sure they are built from scratch.

...
Added   335 /de/software/autohotkey/u3helper/tutorial
Added   336 /de/software/autohotkey/u3helper/u3helper-vs-packagefactory
Total rows 336


Indexed in 6.8s
mbirth@server /...> bin/plugin tntsearch query Spectre
{
    "number_of_hits": 0,
    "execution_time": "1.0729 ms"
}
mbirth@server /...>

EDIT: It looks like the indexing process does something weird, as I can't find the word "Spectre" in the index:

sqlite> select * from wordlist where term="spectre";
sqlite> select * from wordlist where term="spectr";
id|term|num_hits|num_docs
5201|spectr|9|4

[stell]

Same problem is still present here.

[mbirth]

Okay, I don't know why, but it seems the Indexer still used the PorterStemmer even though I had "no" in my config. Now after changing the value via the Grav Admin interface and then setting it back to "no" via text editor (the same thing I did yesterday), it seems to work correctly and the word "Spectre" is indexed fine.

On a sidenote: Selecting "Disable" from the Grav Admin thingy turns the Yaml into stemmer: no which translates into stemmer: false or stemmer: 0 and makes the indexer trying to load a class 0Stemmer which fails. The correct entry has to be stemmer: 'no' for it to work.

[ViliusS]

Interesting indeed. Could be related to #116 which is still waiting for merge, unfortunately. Also check https://github.com/teamtnt/tntsearch/pull/243/files . Not sure which Grav version you are using and how up-to-date TNTSearch library it includes.