After upgrade to 0.8 search produces inconsistent results (0.7 works fine)

[z88kat]

Hi,

When I upgrade flexsearch from 0.7 to 0.8 my numeric search now produces too many results. From my experiments this is what I have found out.

Search Term: "9941"

Indexed Data:

0 "Call Sammy on 9944"
1 "Call Sammy on 9944343432"
2 "Call Jimmy on 9941343432"
3 "Call Jimmy on 9941"

Results:

0,1,2,3

All 4 result are returned. This makes no sense and is NOT the behaviour for version 0.7 which only returns entries 2 & 3

The problem appears to concern numeric search whereby it seems the last digital is ignored on the search term. For example:

Search Term --> What appears to be queried
9941 --> 994
99412 --> 9941
994123 --> 99413
X-9941-X --> 994 (yep, here too)

Just to mention again. This DOES NOT happen with 0.7.

The result is we now get 50 contacts returned instead of 3 which we get with 0.7 when searching with numbers.

This might be solved with a configuration change, but I tried everything and cannot get the old behaviour back.

Here is my configuration:

const options = {
    tokenize: "full",
    threshold: 1,
    optimize: true,
    resolution: 9,
    depth: 3,
    language: "en", 
    stemmer: true,
    document: {
        worker: true,
        id: "id", 
        index: ["document_id", "title", "content"],
        store: ["document_id", "title", "contact_id"]
    }
};

document.search({
            query: "9941",
            enrich: true,
        });

Any help would be greatly appreciated.

[hlovdal]

This is a perfect opportunity for using git bisect:

mv /your/project/node_modules/flexsearch/dist /your/project/node_modules/flexsearch/dist.orig
cd /some/appropriate/directory
git clone https://github.com/nextapps-de/flexsearch
cd flexsearch
mkdir tmp  # Builds fails without this directory present for some reason
git bisect start
git bisect good 0.7.0
git bisect bad 0.8.2

Then build and test each bisect commit:

npm ci
npm run build:all
rm -f /your/project/node_modules/flexsearch/dist
mv dist /your/project/node_modules/flexsearch/dist
pushd /your/project
echo ...Build and test your project...
popd
git bisect good # If testing was successful
git bisect bad  # If testing was unsuccessful

This will let you find out which commit between 0.7.0 and 0.8.2 that introduces the change that breaks your build.

[ts-thomas]

Hello,
a full working example when using v0.8+

const data = [
    { "document_id": 0, "title": "Call Sammy on 9944", "content": "", "contact_id": "" },
    { "document_id": 1, "title": "Call Sammy on 9944343432", "content": "", "contact_id": "" },
    { "document_id": 2, "title": "Call Jimmy on 9941343432", "content": "", "contact_id": "" },
    { "document_id": 3, "title": "Call Jimmy on 9941", "content": "", "contact_id": "" }
];

const index = new Document({
    document: {
        id: "document_id",
        store: ["document_id", "title", "contact_id"],
        index: [{
            field: "title",
            tokenize: "forward"
        }]
    }
});

data.forEach(item => index.add(item));

const result = index.search({
    query: "9941",
    pluck: "title",
    enrich: true
});

console.log(result);

When you really need the "full" tokenizer, then you should use this configuration instead:

const data = [
    { "document_id": 0, "title": "Call Sammy on 9944", "content": "", "contact_id": "" },
    { "document_id": 1, "title": "Call Sammy on 9944343432", "content": "", "contact_id": "" },
    { "document_id": 2, "title": "Call Jimmy on 9941343432", "content": "", "contact_id": "" },
    { "document_id": 3, "title": "Call Jimmy on 9941", "content": "", "contact_id": "" }
];

const encoder = new Encoder({ numeric: false }); // disable numeric tokenizer
const index = new Document({
    document: {
        id: "document_id",
        store: ["document_id", "title", "contact_id"],
        index: [{
            field: "title",
            tokenize: "full",
            encoder
        }]
    }
});

data.forEach(item => index.add(item));

const result = index.search({
    query: "432", // this query does not return any results with the configuration from first example
    pluck: "title",
    enrich: true
});

console.log(result);