Merge pull request #65 from metabrainz/upd-docs

yvanzo · web-flow · commit c01db67b9b8f · 2025-05-15T11:32:04.000+02:00
Update the documentation after moving to SolrCloud 9
diff --git a/HACKING.md b/HACKING.md
@@ -54,8 +54,8 @@ Solr logs are useful to debug making a search query with MBS and/or indexing doc
 
 In a terminal, follow Solr logs using:
 
-```sh
-docker-compose logs -f --tail 1 search
+```bash
+docker compose logs -f --tail 1 search
 ```
 
 For an example, browse your local MusicBrainz Server instance at
@@ -73,7 +73,7 @@ Solr Admin
 To debug how other components (MBS, SIR) interact with the search server,
 the Solr Admin web interface (browsable from <http://localhost:8983>) is your friend.
 
-See “[Getting Started / Solr Admin UI](https://solr.apache.org/guide/solr/latest/getting-started/solr-admin-ui.html)"
+See “[Getting Started / Solr Admin UI](https://solr.apache.org/guide/solr/9_7/getting-started/solr-admin-ui.html)"
 in the Apache Solr Reference Guide 9.7 for more information.
 
 ### Query screen
@@ -124,11 +124,11 @@ Some parameters are automatically set on all queries based on the core configura
 
 Some parameters are set based on the request handler.
 These are identified as "invariants", which means that the parameters will always be set to this value even if you specify a different value in the admin interface.
-For example the [`basic` request handler](https://github.com/metabrainz/mbsssss/blob/v-2021-05-14/common/requestHandler-basic.xml) erases the `pf` field for good.
+For example the [`basic` request handler](https://github.com/metabrainz/mbsssss/blob/v-2025-05-13/common/requestHandler-basic.xml) erases the `pf` field for good.
 
 Some parameters for ranking are set with defaults and can be overridden.
 These can be changed by copying them into the relevant fields in the admin query panel to see the resulting change in ranking order.
-For example the [release’s request parameters](https://github.com/metabrainz/mbsssss/blob/v-2021-05-14/release/conf/request-params.xml)
+For example the [release’s request parameters](https://github.com/metabrainz/mbsssss/blob/v-2025-05-13/release/conf/request-params.xml)
 has a default value for the `fl` field which can be overriden if needed
 (while the `pf` value will still be erased if used from the `/basic` endpoint).
 
diff --git a/LICENSE b/LICENSE
@@ -12,6 +12,8 @@ which is included in this file below the License text:
 
 * start-local-solrcloud is derived from start-local-solr
 * start-musicbrainz-solrcloud is derived from start-create
+* mb-solr/src/main/java/org/musicbrainz/search/analysis/*.java
+  are adapted almost entirely from the Lucene Core libraries
 
 License
 -------
diff --git a/README.md b/README.md
@@ -1,63 +1,111 @@
-# MusicBrainz Solr [![Build Status](https://travis-ci.org/metabrainz/mb-solr.svg?branch=master)](https://travis-ci.org/metabrainz/mb-solr)
+# MusicBrainz Solr
 
 This package includes a
-[QueryResponseWriter](https://cwiki.apache.org/confluence/display/solr/Response+Writers)
+[QueryResponseWriter](https://solr.apache.org/guide/solr/9_7/query-guide/response-writers.html)
 for Apache Solr that will generate
 [mmd-schema](https://github.com/metabrainz/mmd-schema) compliant responses
-for Solr cores running on an [mbsssss](https://github.com/mineo/mbsssss) schema.
+for Solr cores running on an [mbsssss](https://github.com/metabrainz/mbsssss) schema.
 
-## Licensing
+## Prerequisites
 
-Note - Part of the code at org.musicbrainz.search.analysis is adapted almost entirely from Lucene core libs.
-As such those files are licensed under Apache 2.0 license which is compatible with the existing BSD license of MB-Solr.
+In May 2025, Solr data for indexing the whole MusicBrainz database takes ~75 GB of disk space.
+Since indexing takes hours, you might want to load the Zstandard-compressed backup archives.
+Those archives take ~60 GB of disk space and their temporarily extracted files take ~75 GB more.
+Hence, it is currently recommended to provision at least 250 GB of disk space for Solr.
+
+Recommendations:
+* RAM: 4 GB
+* CPU: 8 threads, x86-64 architecture
+* Disk Space: 250 GB (or 100 GB if you don't load backups.)
+
+Required software:
+* Git
+* Docker 2
+  (or Java SE 17, Maven 3, and Solr 9.7.0)
 
 ## Installation
 
-### Installing brainz-mmd2-jaxb
+The generally recommended installation method is using [Docker](https://docs.docker.com).
+A Docker image is available through the eponymous [Docker Hub repository](https://hub.docker.com/r/metabrainz/mb-solr) for each release of MusicBrainz Solr.
+These images can be used through the [MusicBrainz Docker Compose project](https://github.com/metabrainz/musicbrainz-docker) which comes with instructions.
+
+For [development](#development) or other purposes, these images can also be run alone, see the below subsection “[Installation without MusicBrainz Server](#installation-without-musicbrainz-server)”.
+
+To set up a cluster of Solr nodes, it can be preferred to install to the host system directly, see the below subsection “[Installation without Docker](#installation-without-docker)” and the [Apache Solr Reference Guide 9.7](https://solr.apache.org/guide/solr/9_7/).
+
+### Installation without MusicBrainz Server
+
+Clone the repository with Git:
+
+```bash
+git clone --recursive https://github.com/metabrainz/mb-solr.git
+```
+
+Run it alone on port 8983 with Docker:
+
+```bash
+docker compose up
+```
+
+### Installation without Docker
+
+:warning: This section is outdated as it predates upgrading to SolrCloud 9.
+
+:newspaper: The MusicBrainz Solr cluster powering musicbrainz.org is using [Ansible](https://docs.ansible.com/) for deployment.
+
+#### Installing brainz-mmd2-jaxb
 
 Clone the repository with Git:
 
-    git clone https://github.com/metabrainz/mmd-schema.git
+```bash
+git clone https://github.com/metabrainz/mmd-schema.git
+```
 
 And install the package:
 
-    cd mmd-schema/brainz-mmd2-jaxb
-    mvn install
+```bash
+cd mmd-schema/brainz-mmd2-jaxb
+mvn install
+```
 
-### Installing the query writer
+#### Installing the query writer
 
 Clone the repository with Git:
 
-    git clone --recursive https://github.com/metabrainz/mb-solr.git
+```bash
+git clone --recursive https://github.com/metabrainz/mb-solr.git
+```
 
 Navigate to the **mb-solr** folder in a terminal and build a JAR
 file:
 
-    mvn package
+```bash
+mvn package
+```
 
 This will create a file called
 **solrwriter-0.0.1-SNAPSHOT-jar-with-dependencies.jar** in the **target** folder.
 
 Now you need to make this JAR file available to all Solr cores that need it.
 The easiest option is to configure a **sharedLib** in your
-[solr.xml](https://cwiki.apache.org/confluence/display/solr/Format+of+solr.xml)
+[solr.xml](https://solr.apache.org/guide/solr/9_7/configuration-guide/configuring-solr-xml.html)
 and put the JAR file into that.
 
 All that's left to do now is enabling the Query Response Writers in your cores
-[solrconfig.xml](https://cwiki.apache.org/confluence/display/solr/Configuring+solrconfig.xml).
+[solrconfig.xml](https://solr.apache.org/guide/solr/9_7/configuration-guide/configuring-solrconfig-xml.html).
 To do that, add the following lines as children of the **config** element:
 
 ```xml
-    <queryResponseWriter name="mbxml" class="org.musicbrainz.search.solrwriter.MBXMLWriter">
-        <str name="entitytype">$entitytype</str>
-    </queryResponseWriter>
-    <queryResponseWriter name="mbjson" class="org.musicbrainz.search.solrwriter.MBJSONWriter">
-        <str name="entitytype">$entitytype</str>
-    </queryResponseWriter>
+<queryResponseWriter name="mbxml" class="org.musicbrainz.search.solrwriter.MBXMLWriter">
+    <str name="entitytype">$entitytype</str>
+</queryResponseWriter>
+<queryResponseWriter name="mbjson" class="org.musicbrainz.search.solrwriter.MBJSONWriter">
+    <str name="entitytype">$entitytype</str>
+</queryResponseWriter>
 ```
 
 The solrconfig.xml of the cores defined by
-[mbsssss](https://github.com/mineo/mbsssss) already includes this snippet, as
+[mbsssss](https://github.com/metabrainz/mbsssss) already includes this snippet, as
 well as the **sharedLib** configuration in the solr.xml file.
 
 **$entitytype** needs to be replaced by the entity type of the documents in the store.
@@ -69,7 +117,6 @@ Valid values are:
 - cdstub
 - editor
 - event
-- freedb
 - instrument
 - label
 - place
@@ -83,37 +130,70 @@ Valid values are:
 
 Now the core needs to be reloaded.
 After that, two new values for the
-[wt paramter](https://cwiki.apache.org/confluence/display/solr/Response+Writers)
+[wt paramter](https://solr.apache.org/guide/solr/9_7/query-guide/common-query-parameters.html#wt-parameter)
 are available:
 
 - **mbxml**, which returns mmd-compliant XML documents
-- **mbjson**, which returns JSON document as described by
-  [this page](https://beta.musicbrainz.org/doc/Development/JSON_Web_Service)
+- **mbjson**, which returns JSON document as described by the
+  [MusicBrainz API documentation page](https://musicbrainz.org/doc/MusicBrainz_API)
 
 At the moment, the
-[field list](https://cwiki.apache.org/confluence/display/solr/Common+Query+Parameters#CommonQueryParameters-Thefl)
+[field list](https://solr.apache.org/guide/solr/9_7/query-guide/common-query-parameters.html#fl-field-list-parameter)
 parameter of each query needs to include the **score** field for the code to
 work correctly.
 
 A branch of the MusicBrainz server that can query a Solr server with this
 QueryResponseWriter is available on
 [GitHub](https://github.com/mineo/musicbrainz-server/tree/solr-search).
 
-### Known Vulnerabilities
+## Development
 
-- [CVE-2021-44228](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-44228):
-  It can be mitigated by setting the environement variable `LOG4J_FORMAT_MSG_NO_LOOKUPS=true`.
+If you followed “[Installation without MusicBrainz Server](#installation-without-musicbrainz-server)”,
+just stop the last command and run it again after having made some changes.
 
-## Docker Installation
+Otherwise, if you need it to be tested with either the indexer or MusicBrainz Server:
 
-Clone the repository with Git:
+1. Make some changes in your local Git clone (of MusicBrainz Solr)
+   (The two next steps are optional but recommended to better identify images.)
+2. Commit these changes to your local Git branch.
+3. Tag this commit with a meaningful version:
+
+   ```bash
+   git tag adastrawberry-1969
+   ```
+
+4. Build a docker image:
 
-    git clone --recursive https://github.com/metabrainz/mb-solr.git
+   ```bash
+   ./build.sh
+   ```
 
-Either run it alone on port 8983:
+5. Use this tag to [set `MB_SOLR_VERSION` in MusicBrainz Docker Compose project](https://github.com/metabrainz/musicbrainz-docker?tab=readme-ov-file#local-development-of-musicbrainz-solr).
 
-    docker-compose up
+See also [The Debugger’s Guide to MusicBrainz Solr](HACKING.md).
 
-Or build a tagged image to run with [MusicBrainz Docker](https://github.com/metabrainz/musicbrainz-docker):
+## Helper commands
+
+SolrCloud collections backups from the MusicBrainz Solr cluster
+powering musicbrainz.org are made available twice a week to save
+mirror owners the initial cost of indexing the whole MusicBrainz data
+from scratch which can take hours and significant resources.
+Those backups are provided as Zstandard-compressed tar archives.
+Helper commands to handle these archives are provided in Docker images:
+
+```bash
+fetch-backup-archives
+load-backup-archives
+remove-backup-archives
+```
+
+In May 2025, the measured time for fetching 60 GB was 12 min (depending on your bandwidth,)
+and the measured time for loading was 12 min (depending on your CPU/RAM/Disk speed.)
+
+A fourth helper command allows to delete all the Solr data if needed:
+
+```bash
+delete-indexed-documents
+```
 
-    ./build.sh
+Each of these commands is self-documented through the option `--help`.
diff --git a/docker-compose.yml b/docker-compose.yml
@@ -8,6 +8,7 @@ services:
     build:
       context: .
       dockerfile: Dockerfile
+    mem_swappinness: 1
     ports:
       - "127.0.0.1:8983:8983"
     volumes:
diff --git a/mb-solr/src/main/resources/oxml.xml b/mb-solr/src/main/resources/oxml.xml
@@ -18,7 +18,6 @@
                 <xml-element java-attribute="recordingList" xml-path="."/>
                 <xml-element java-attribute="urlList" xml-path="."/>
                 <xml-element java-attribute="cdstubList" xml-path="."/>
-                <xml-element java-attribute="freedbDiscList" xml-path="."/>
                 <xml-element java-attribute="releaseList" xml-path="."/>
                 <xml-element java-attribute="tagList" xml-path="."/>
                 <xml-element java-attribute="workList" xml-path="."/>
@@ -235,16 +234,6 @@
                 <xml-element java-attribute="trackList" xml-path="."/>
             </java-attributes>
         </java-type>
-        <java-type name="FreedbDiscList">
-            <java-attributes>
-                <xml-element java-attribute="freedbDisc" name="freedb-discs"/>
-            </java-attributes>
-        </java-type>
-        <java-type name="FreedbDisc">
-            <java-attributes>
-                <xml-element java-attribute="trackList" xml-path="."/>
-            </java-attributes>
-        </java-type>
         <java-type name="LabelInfoList">
             <java-attributes>
                 <xml-element java-attribute="labelInfo" name="label-info"/>
