Skip to content

local-loading-notes.adoc

Latest commit

 

History

History
103 lines (86 loc) · 2.42 KB

local-loading-notes.adoc

File metadata and controls

103 lines (86 loc) · 2.42 KB

Local SOLR Loading Notes

Notes and instructions for running and loading a Solr instance from your local machine.

GUS_HOME Setup

If you don’t have a GUS_HOME set up already, a minimal setup is described here:

  1. Create a directory for your GUS_HOME.

    mkdir -p ~/gus_home/lib/python/SiteSearchData/Model

  2. Copy the lib file from Model/lib/python into the GUS_HOME path we just created.

    cp -t ~/gus_home/lib/python/SiteSearchData/Model Model/lib/python/BatchReportUtils.py

You should now be able to run the Solr loading scripts.

SOLR Setup

  1. make build

  2. make run

  3. docker exec -it <container-name> bash

    mkdir -p ~/site_search/conf
cp -rt ~/site_search/conf/ /opt/solr/server/solr/configsets/site-search/conf/*

  4. From http://localhost:8983/ go to Core Admin

  5. Use the following configuration options:

    name

    site_search

    instanceDir

    /home/solr/site_search/

    dataDir

    /home/solr

    config

    /home/solr/site_search/conf/solrconfig.xml

    schema

    /home/solr/site_search/conf/schema.xml

  6. Press "Add Core"

Loading

Download the target files from yew a build and project directory under the root /eupath/data/EuPathDB/siteSearchDataDumps/ into a local directory. For the following examples we will use build-65 and ToxoDB

Using SFTP

  1. Create a local directory to contain the batches for the target project, then cd into that directory.

    mkdir ~/ToxoDB
cd ~/ToxoDB

  2. Open an SFTP connection to yew.

    sftp <connection info for yew>

  3. Run the following SFTP commands:

    cd /eupath/data/EuPathDB/siteSearchDataDumps/bld65/ToxoDB
get -R .
exit

At this point you should now have a mirror of the ToxoDB batches in your local ToxoDB directory. From here you can run the target Model/bin loading script(s) to populate the site_search Solr core in your local instance.

For this example we will use the multi-batch loading script for our downloaded files:

# Go to the directory with the bin scripts
cd /path/to/SiteSearchData/Model/bin

# Export the necessary env vars
export GUS_HOME=/path/to/local/gus_home
export PATH=$PATH:$PWD

# Load the batches into Solr
./ssLoadMultipleBatches https://localhost:8983/solr/site_search ~/ToxoDB

# Commit the typeahead index
./ssCommitSuggesterIndex