Elasticsearch 7.* and 8.* integration. OpenSearch integration.

[ivanmrsulja]

What does this pull request do?

Updates current ES 6.x integration to 8.x.

What's new?

Changes in ResponseParser and ES documentation on the first draft.

Example:

• Changed src/main/java/edu/cornell/mannlib/vitro/webapp/searchengine/elasticsearch/ResponseParser.java to be in line with current ES API
• Updated src/main/java/edu/cornell/mannlib/vitro/webapp/searchengine/elasticsearch/Elasticsearch_notes_on_the_first_draft.md with new mapping
• Updated example.applicationSetup.n3 to show ES setup example

How should this be tested?

Initial setup

• Install elasticsearch/opensearch somewhere.
• Create a search index with the appropriate mapping (see below).
• Check out VIVO and this branch of Vitro (see below), and do the usual installation procedure.
• Modify {vitro_home}/config/applicationSetup.n3 to use this driver (see below).
• Modify the vitro.local.searchengine.url configuration property to contain ES index base URL (due to backward compatibility, Solr can also be configured using vitro.local.solr.url. This will however result in a warning that is shown in logs, advising the client to switch to a new configuration parameter)
• Modify the vitro.local.searchengine.username configuration property to contain ES/OS basic auth username
• Modify the vitro.local.searchengine.password configuration property to contain to contain ES/OS basic auth password
• Start elasticsearch/opensearch
• Start VIVO

A mapping for the search index

curl -X PUT "localhost:9200/vivo?pretty" -H 'Content-Type: application/json' -d'
{
  "settings":{
    "index":{
      "analysis":{
        "tokenizer":{
          "keyword_tokenizer":{
            "type":"keyword"
          },
          "whitespace_tokenizer":{
            "type":"whitespace"
          }
        },
        "filter":{
          "lowercase_filter":{
            "type":"lowercase"
          },
          "edgengram_filter":{
            "type":"edge_ngram",
            "min_gram":2,
            "max_gram":25
          },
          "word_delimiter_filter":{
            "type":"word_delimiter",
            "generate_word_parts":true,
            "generate_number_parts":true,
            "catenate_words":false,
            "catenate_numbers":false,
            "catenate_all":false,
            "split_on_case_change":true
          },
          "porter_stem_filter":{
            "type":"snowball",
            "language":"English"
          }
        },
        "analyzer":{
          "default":{
            "type":"english"
          },
          "edgengram_untokenized":{
            "type":"custom",
            "tokenizer":"keyword_tokenizer",
            "filter":[
              "lowercase_filter",
              "edgengram_filter"
            ]
          },
          "edgengram_untokenized_query":{
            "type":"custom",
            "tokenizer":"keyword_tokenizer",
            "filter":[
              "lowercase_filter"
            ]
          },
          "edgengram_stemmed":{
            "type":"custom",
            "tokenizer":"whitespace_tokenizer",
            "filter":[
              "word_delimiter_filter",
              "lowercase_filter",
              "porter_stem_filter",
              "edgengram_filter"
            ]
          },
          "edgengram_stemmed_query":{
            "type":"custom",
            "tokenizer":"whitespace_tokenizer",
            "filter":[
              "word_delimiter_filter",
              "lowercase_filter",
              "porter_stem_filter"
            ]
          },
          "sort_field_analyzer":{
            "type":"custom",
            "tokenizer":"keyword",
            "filter":[
              "lowercase"
            ]
          }
        }
      }
    }
  },
  "mappings":{
    "dynamic_templates":[
      {
        "field_sort_template":{
          "match":"*_label_sort",
          "mapping":{
            "type":"text",
            "fields":{
              "keyword":{
                "type":"keyword"
              }
            },
            "fielddata":true,
            "analyzer":"sort_field_analyzer"
          }
        }
      },
      {
        "field_ss_template":{
          "match":"*_ss",
          "mapping":{
            "type":"text",
            "fields":{
              "keyword":{
                "type":"keyword",
                "ignore_above":256
              }
            },
            "fielddata":true
          }
        }
      },
      {
        "date_range_template":{
          "match":"*_drsim",
          "mapping":{
            "type":"date_range",
            "format":"strict_date_optional_time||epoch_millis"
          }
        }
      }
    ],
    "properties":{
      "ALLTEXT":{
        "type":"text",
        "analyzer":"english",
        "fields":{
          "keyword":{
            "type":"keyword",
            "ignore_above":256
          }
        }
      },
      "ALLTEXTUNSTEMMED":{
        "type":"text",
        "analyzer":"standard"
      },
      "DocId":{
        "type":"keyword"
      },
      "classgroup":{
        "type":"keyword"
      },
      "type":{
        "type":"keyword"
      },
      "mostSpecificTypeURIs":{
        "type":"keyword"
      },
      "indexedTime":{
        "type":"long"
      },
      "nameRaw":{
        "type":"keyword"
      },
      "URI":{
        "type":"keyword"
      },
      "THUMBNAIL":{
        "type":"integer"
      },
      "THUMBNAIL_URL":{
        "type":"keyword"
      },
      "nameLowercaseSingleValued":{
        "type":"text",
        "analyzer":"standard",
        "fielddata":true
      },
      "BETA":{
        "type":"float"
      },
      "acNameUntokenized":{
        "type":"text",
        "analyzer":"edgengram_untokenized",
        "search_analyzer":"edgengram_untokenized_query"
      },
      "acNameStemmed":{
        "type":"text",
        "analyzer":"edgengram_stemmed",
        "search_analyzer":"edgengram_stemmed_query"
      }
    }
  }
}
'

Modify applicationSetup.n3

• Change this (it is already changed in this PR):

# ----------------------------
#
# Search engine module: 
#    The Solr-based implementation is the only standard option, but it can be
#    wrapped in an "instrumented" wrapper, which provides additional logging 
#    and more rigorous life-cycle checking.
#

:instrumentedSearchEngineWrapper 
    a   <java:edu.cornell.mannlib.vitro.webapp.searchengine.InstrumentedSearchEngineWrapper> , 
        <java:edu.cornell.mannlib.vitro.webapp.modules.searchEngine.SearchEngine> ;
    :wraps :solrSearchEngine .

• To this:

# ----------------------------
#
# Search engine module: 
#    The Solr-based implementation is the only standard option, but it can be
#    wrapped in an "instrumented" wrapper, which provides additional logging 
#    and more rigorous life-cycle checking.
#

:instrumentedSearchEngineWrapper 
    a   <java:edu.cornell.mannlib.vitro.webapp.searchengine.InstrumentedSearchEngineWrapper> , 
        <java:edu.cornell.mannlib.vitro.webapp.modules.searchEngine.SearchEngine> ;
    :wraps :elasticSearchEngine .

:elasticSearchEngine
    a   <java:edu.cornell.mannlib.vitro.webapp.searchengine.elasticsearch.ElasticSearchEngine> ,
        <java:edu.cornell.mannlib.vitro.webapp.modules.searchEngine.SearchEngine> .

Your setup should be completed now 😃 ! After this, you should perform common manual tests that are done for every new release.

Interested parties

@chenejac

Reviewers' expertise

Candidates for reviewing this PR should have some of the following expertises:

1. Java
2. Elasticsearch 7.* or 8.*

[chenejac]

The following features should be tested:

• search form - searching, filtering and sorting (changing localization)
• list of research, persons, org units - alphabetical index (changing localization)
• lookup/autocompletion at some forms - for instance adding author to a publication (changing localization)
• visualizations - map of science, collaboration network

[chenejac]

@ivanmrsulja please create a VIVO PR with updated example.runtime.properties. Also, please move JSON configuration into vivo-es project. Add in the vivo-es project a Docker file, and update README file to explain how ES should be run.

[chenejac]

@ivanmrsulja basic VIVO search functionalities works for me. I didn't review the code. Instructions from the PR description about setup of the elasticsearch index might be replaced with a pointer to the vivo-es Readme file.

[litvinovg]

It seems the mapping in the PR description is not the same as the mapping in https://github.com/ivanmrsulja/vivo-es/blob/main/index-config.json
Also if search string is empty then there are no search result on search page, also no filters are visible, that is different from current behavior with Solr based search indexes.

[litvinovg]

It seems facets aren't displayed on search page anymore.

[ivanmrsulja]

It seems facets aren't displayed on search page anymore.

Facets had a small problem but they should work fine now. For me, they displayed either way, just the filters did not work sometimes. How do you mean they are not even displayed?

[litvinovg]

I don't see filters if I open search page.

[ivanmrsulja]

I don't see filters if I open search page.

I fixed the empty search field edgecase 🙂

[litvinovg]

The edge case is fixed. Why should it only be applied to the /search page?
Is it possible to add tests and clean up the code?

[ivanmrsulja]

The edge case is fixed. Why should it only be applied to the /search page? Is it possible to add tests and clean up the code?

I updated the code as you suggested. Also, I cleaned up stale, commented-out code and dependencies and added tests for utility classes.

[litvinovg]

I think we should avoid conditionals based on servlet name.

[ivanmrsulja]

Unfortunately this was the only way I could differentiate from which actual page is this particular servlet called, furthermore, only search page requests require plain text query interpretation. Do you have another suggestion on how I can implement this?

[litvinovg]

It seems that different clients are used when using http or https. In case of using http provided username and password have no effect. In case someone would try to use combination of http and user/password authentication it might not work.

[ivanmrsulja]

You are right. I encapsulated everything into one shared factory. Great suggestion!

[litvinovg]

It seems ssl connection doesn't work.
error.log

[litvinovg]

Thank you for the fix.
What I also noticed is that date time filters work with OpenSearch, but do not work with ElasticSearch.
@ivanmrsulja could you check it?

[litvinovg]

Thank you for the fix. What I also noticed is that date time filters work with OpenSearch, but do not work with ElasticSearch. @ivanmrsulja could you check it?

It works with Elastic Search 8.19.6
It doesn't work with Elastic Search 9.2.1.
@chenejac do we need to support Elastic Search 9?

[chenejac]

Thank you for the fix. What I also noticed is that date time filters work with OpenSearch, but do not work with ElasticSearch. @ivanmrsulja could you check it?

It works with Elastic Search 8.19.6 It doesn't work with Elastic Search 9.2.1. @chenejac do we need to support Elastic Search 9?

I think we can go with support for ES 7 and 8 (as in the title of the PR).

[chenejac]

@ivanmrsulja Works for me, tested with OpenSearch.