Relevancy Delta Review

Work in Progress.

About

When development or operations work changes the Searchworking Solr index, there may be changes to UX aspects of the service like new relevancy rankings, or changes in results returned for known searches.

To balance the need to keep Solr and index resources up to date & able to take on new features with the need to keep a consistent and reviewed Searchworks search results user experience, use the following "relevancy delta review". This relevancy delta review enables feedback from identified Searchworks super users to be taken into consideration by DLSS teams before deploying changes to production. It is a process to be used at the discretion of the a team lead in consultation with identified key stakeholders.

Please adapt these steps as needed for each work cycles' particular needs. This should be just a starting point for unblocking index resource changes while keeping SW Index consistency.

What is the Relevancy Delta

The following only measures deltas in relevancy across Solr indices (normally, SW Production Index and a proposed dev index), not the quality of relevancy in a given index. Deltas can be good or needed improvements, but still may need review before shipping.

Relevancy Delta Review

1. When the changed resources (codebases, configs, solr indices, or other) are ready for review, changes should be deployed to the Searchworks Relevancy Delta review environments (currently, sw-webapp-sandbox-e.stanford.edu which points to sw-relevancy index on sw-solr, and sw-webapp-sandbox-f.stanford.edu which points to searchworks-dev index on sul-solr).
2. If not already done as part of the dev process, run the sw_index_tests suite on the new index to identify any changes to identified issues.
3. If not already done as part of the dev process, run the last month's top 1000 (or other subset of) queries through SW Relevancy Delta Dashboard & identify any divergences that need to be accounted for (they may be improvements & should just be highlighted).
4. The identified Searchworks Super Users should be emailed asking for a time-boxed (~2 days currently) review of the Searchworks Relevancy Delta review environments. If appropriate, the email should contain possible identified gotchas from the sw-solr-tests and the SW Relevancy Delta Dashboard that the Super User Group should take into account or respond to.
5. After gathering feedback, the project team should review for any possible blockers and plan how to address.