Research replacing Postgress full-text search with ElasticSearch

[neilmb]

Feature/what we're after

Fast, accurate search is a key feature for the catalog, so let's explore replacing our current Postgres full-text search approach with a separate ElasticSearch cluster.

Anticipated/hypothesized benefits

• Faster responses for searches with many results (like "data")
• Search algorithms that give us "better" results for the data people are looking for

Measurements/metrics

• Search API response times for queries like "data" or "state"

References/background

• Cloud.gov brokered ElasticSearch https://docs.cloud.gov/platform/services/aws-elasticsearch/

[neilmb]

Ingesting and synchronizing data between Postgres and OpenSearch is the key. Rather than starting with manual syncing in the harvesting code (called "double writes" for writing changes to the SQL database and the search database), it would be nice to have an "automatic" syncing process. OpenSearch has a source/sink-based synchronizer that can read from our Amazon RDS Postgres https://docs.opensearch.org/latest/data-prepper/pipelines/configuration/sources/rds/ and write to our Amazon OpenSearch service https://docs.opensearch.org/latest/data-prepper/pipelines/configuration/sinks/opensearch/#usage using the credentials that Cloud.gov provides (probably need to save them into the environment).

We should be able to run the sync code in a Docker-deployed Cloud.gov app https://docs.opensearch.org/latest/data-prepper/getting-started/ and we should be able to build the configuration file from the environment credentials in our .profile script. So there will be a new app running in our spaces called something like datagov-catalog-opensearch-sync that will handle syncing our Postgres data into our OpenSearch cluster.

That's the plan. We'll start prototyping it now using a Docker Compose setup for the local development version of this.