Migrate from MySQL to OpenSearch for proteins #76
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
tibvdm
approved these changes
Apr 23, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR changes the backend used for keeping track of all UniProt accessions from MySQL (MariaDB) to OpenSearch.
OpenSearch is a document-based search engine that's optimized for retrieving the documents that match with a set of text-based query-parameters, as fast as possible. I've configured OpenSearch in such a way that the UniProt accession number of proteins is used as the ID (which means that entries can still be retrieved instantly by ID), while simultaneously the other fields (name, taxon ID, accession number) are indexed in a smart way such that they can be found relatively fast.
This change from MySQL to OpenSearch was introduced because the
FULLTEXTindex in MySQL that we introduced earlier (for thenameanduniprot_accession_numbercolumns of proteins) is still not sufficiently performant for the use case we want.I've added a new guide on the Unipept Wiki that explains how OpenSearch can be installed and deployed on our API-servers.
For reference, the whole index takes op 110GiB of disk space and works sufficiently well with 20GiB of RAM.