Added support for pip install (#47)

soldni · web-flow · commit e7011615b96f · 2019-06-18T22:32:45.000-04:00
Fixes #18.
diff --git a/MANIFEST.in b/MANIFEST.in
@@ -0,0 +1,2 @@
+include requirements.txt
+include README.md
diff --git a/README.md b/README.md
@@ -1,39 +1,33 @@
-**We recommend to download the latest tested version from the [releases section](https://github.com/Georgetown-IR-Lab/QuickUMLS/releases)**.
-
-**NEW: v.1.2 now includes client/server support!** Start a QuickUMLS server once, avoid loading QuickUMLS each time your experiments run! See <a href="#client_server">below</a> for more info.
+[**NEW: v.1.3 is pip-ready!**](https://giphy.com/embed/BlVnrxJgTGsUw) You can now install QuickUMLS through a simple `pip install quickumls`.
 
 # QuickUMLS
 
 QuickUMLS (Soldaini and Goharian, 2016) is a tool for fast, unsupervised  biomedical concept extraction from medical text.
 It takes advantage of [Simstring](http://www.chokkan.org/software/simstring/) (Okazaki and Tsujii, 2010) for approximate string matching.
 For more details on how QuickUMLS works, we remand to our paper.
 
-This project should be compatible with both Python 2 and 3 and run on any UNIX system (support for Windows is experimental, please report bugs!). **If you find any bugs, please file an issue on GitHub or email the author at luca@ir.cs.georgetown.edu**.
+This project should be compatible with Python 3 (Python 2 is [no longer supported](https://pythonclock.org/)) and run on any UNIX system (support for Windows is experimental, please report bugs!). **If you find any bugs, please file an issue on GitHub or email the author at luca@ir.cs.georgetown.edu**.
 
 ## Installation
 
-#### Before Starting
-
-1. Make sure that your Python installation include C headers (e.g., on Ubuntu, make sure `python3-dev` or `python-dev` are installed).
-2. This software requires all packages listed in the `requirements.txt` file. You can install all of them by running `pip install -r requirements.txt`.
-3. Note that, in order to use `spacy`, you are required to download its corpus. You can do that by running `python -m spacy download en`.
-4. This system requires you to have a valid UMLS installation on disk. To install UMLS, you must first obtain a [license](https://uts.nlm.nih.gov/license.html) from the National Library of Medicine; then you should download all UMLS files from [this page](https://www.nlm.nih.gov/research/umls/licensedcontent/umlsknowledgesources.html); finally, you can install UMLS using the [MetamorphoSys](https://www.nlm.nih.gov/pubs/factsheets/umlsmetamorph.html) tool as [explained in this guide](https://www.nlm.nih.gov/research/umls/implementation_resources/metamorphosys/help.html).  The installation can be removed once the system has been initialized.
+1. **Obtain a UMLS installation** This tool requires you to have a valid UMLS installation on disk. To install UMLS, you must first obtain a [license](https://uts.nlm.nih.gov/license.html) from the National Library of Medicine; then you should download all UMLS files from [this page](https://www.nlm.nih.gov/research/umls/licensedcontent/umlsknowledgesources.html); finally, you can install UMLS using the [MetamorphoSys](https://www.nlm.nih.gov/pubs/factsheets/umlsmetamorph.html) tool as [explained in this guide](https://www.nlm.nih.gov/research/umls/implementation_resources/metamorphosys/help.html).  The installation can be removed once the system has been initialized.
+2. **Install QuickUMLS**: You can do so by either running `pip install quickumls` or `python setup.py install`. On macOS, using anaconda is **strongly recommended**<sup>†</sup>. 
+3. **Obrain a SpaCy corpus**: After you install QuickUMLS and its dependencies, you should be able to do so by running `python -m spacy download en`.
+3. **Create a QuickUMLS installation** Initialize the system by running `python -m quickumls.install <umls_installation_path> <destination_path>`, where `<umls_installation_path>` is where the installation files are (in particular, we need `MRCONSO.RRF` and `MRSTY.RRF`) and `<destination_path>` is the directory where the QuickUmls data files should be installed. This process will take between 5 and 30 minutes depending how fast the CPU and the drive where UMLS and QuickUMLS files are stored are (on a system with a Intel i7 6700K CPU and a 7200 RPM hard drive, initialization takes 8.5 minutes). `python -m quickumls.install` supports the following optional arguments:
+    - `-L` / `--lowercase`: if used, all concept terms are folded to lowercase before being processed. This option typically increases recall, but it might reduce precision;
+    - `-U` / `--normalize-unicode`: if used, expressions with non-ASCII characters are converted to the closest combination of ASCII characters.
+    - `-E` / `--language`: Specify the language to consider for UMLS concepts; by default, English is used. For a complete list of languages, please see [this table provided by NLM](https://www.nlm.nih.gov/research/umls/knowledge_sources/metathesaurus/release/abbreviations.html#LAT).
 
-#### How To get the System Initialized
 
-1. Download and compile Simstring by running `bash setup_simstring.sh <python_version>`, where `<python_version>` is either "`2`" or "`3`".
-2. Initialize the system by running `python install.py <umls_installation_path> <destination_path>`, where `<umls_installation_path>` is where the installation files are (in particular, we need `MRCONSO.RRF` and `MRSTY.RRF`) and `<destination_path>` is the directory where the QuickUmls data files should be installed. This process will take between 5 and 30 minutes depending how fast the CPU and the drive where UMLS and QuickUMLS files are stored are (on a system with a Intel i7 6700K CPU and a 7200RPM hard drive, initialization takes 8.5 minutes).
-
-`install.py` supports the following optional arguments:
-- `-L` / `--lowercase`: if used, all concept terms are folded to lowercase before being processed. This option typically increases recall, but it might reduce precision;
-- `-U` / `--normalize-unicode`: if used, expressions with non-ASCII characters are converted to the closest combination of ASCII characters.
-- `-E` / `--language`: Specify the language to consider for UMLS concepts; by default, English is used. For a complete list of languages, please see [this table provided by NLM](https://www.nlm.nih.gov/research/umls/knowledge_sources/metathesaurus/release/abbreviations.html#LAT).
+**†**: If the installation fails on macOS when using Anaconda, install `leveldb` first by running `conda install -c conda-forge python-leveldb`.
 
 ## APIs
 
 A QuickUMLS object can be instantiated as follows:
 
 ```python
+from quickumls import QuickUMLS
+
 matcher = QuickUMLS(quickumls_fp, overlapping_criteria, threshold,
                     similarity_name, window, accepted_semtypes)
 ```
@@ -57,22 +51,22 @@ matcher.match(text, best_match=True, ignore_syntax=False)
 Set `best_match` to `False` if you want to return overlapping candidates, `ignore_syntax` to `True` to disable all heuristics introduced in (Soldaini and Goharian, 2016).
 
 
-<h2 id="client_server">[NEW] Server / Client Support</h2>
+## Server / Client Support
 
 Starting with v.1.2, QuickUMLS includes a support for being used in a client-server configuration. That is, you can start one QuickUMLS server, and query it from multiple scripts using a client.
 
-To start the server, run `server.py`:
+To start the server, run `python -m quickumls.server`:
 
 ```bash
-python server.py /path/to/quickumls/files {-P QuickUMLS port} {-H QuickUMLS host} {QuickUMLS options}
+python -m quickumls.server /path/to/quickumls/files {-P QuickUMLS port} {-H QuickUMLS host} {QuickUMLS options}
 ```
 
-Host and port are optional; by default, QuickUMLS runs on `localhost:4645`. You can also pass any QuickUMLS option mentioned above to the server. To obtain a list of options for the server, run `python server.py -h`.
+Host and port are optional; by default, QuickUMLS runs on `localhost:4645`. You can also pass any QuickUMLS option mentioned above to the server. To obtain a list of options for the server, run `python -m quickumls.server -h`.
 
-To load the client, import `get_quickumls_client` from `client.py`:
+To load the client, import `get_quickumls_client` from `quickumls`:
 
 ```bash
-from client import get_quickumls_client
+from quickumls import get_quickumls_client
 matcher = get_quickumls_client()
 text = "The ulna has dislocated posteriorly from the trochlea of the humerus."
 matcher.match(text, best_match=True, ignore_syntax=False)
@@ -84,7 +78,7 @@ The API of the client is the same of a QuickUMLS object.
 In case you wish to run the server in the background, you can do so as follows:
 
 ```bash
-nohup python server.py /path/to/QuickUMLS {server options} > /dev/null 2>&1 & echo $! > nohup.pid
+nohup python -m quickumls.server /path/to/QuickUMLS {server options} > /dev/null 2>&1 & echo $! > nohup.pid
 
 ```
 
diff --git a/__init__.py b/__init__.py
diff --git a/quickumls/__init__.py b/quickumls/__init__.py
@@ -0,0 +1,3 @@
+from .core import QuickUMLS
+from .client import get_quickumls_client
+from .about import *
diff --git a/quickumls/about.py b/quickumls/about.py
@@ -0,0 +1,12 @@
+# inspired from:
+# https://python-packaging-user-guide.readthedocs.org/en/latest/single_source_version/
+# https://github.com/pypa/warehouse/blob/master/warehouse/__about__.py
+# https://github.com/explosion/spaCy/blob/master/spacy/about.py
+
+__title__ = 'quickumls'
+__version__ = '1.3.0r4'
+__author__ = 'Luca Soldaini'
+__email__ = 'luca@ir.cs.georgetown.edu'
+__license__ = 'MIT'
+__uri__ = "https://github.com/Georgetown-IR-Lab/QuickUMLS"
+__copyright__ = '2014-2019, Georgetown University Information Retrieval Lab'
diff --git a/quickumls/client.py b/quickumls/client.py
@@ -1,9 +1,5 @@
-try:
-    from network import MinimalClient
-    from quickumls import QuickUMLS
-except ImportError:
-    from .network import MinimalClient
-    from .quickumls import QuickUMLS
+from .network import MinimalClient
+from .core import QuickUMLS
 
 
 def get_quickumls_client(host='localhost', port=4645):
diff --git a/quickumls/constants.py b/quickumls/constants.py
diff --git a/quickumls/core.py b/quickumls/core.py
@@ -14,12 +14,8 @@
 from unidecode import unidecode
 
 # project modules
-try:
-    import toolbox
-    import constants
-except ImportError:
-    from . import toolbox
-    from . import constants
+from . import toolbox
+from . import constants
 
 
 class QuickUMLS(object):
@@ -32,25 +28,25 @@ def __init__(
             accepted_semtypes=constants.ACCEPTED_SEMTYPES,
             verbose=False):
         """Instantiate QuickUMLS object
-        
+
             This is the main interface through which text can be processed.
 
         Args:
             quickumls_fp (str): Path to which QuickUMLS was installed
-            overlapping_criteria (str, optional): 
-                    One of "score" or "length". Choose how results are ranked. 
+            overlapping_criteria (str, optional):
+                    One of "score" or "length". Choose how results are ranked.
                     Choose "score" for best matching score first or "length" for longest match first.. Defaults to 'score'.
             threshold (float, optional): Minimum similarity between strings. Defaults to 0.7.
             window (int, optional): Maximum amount of tokens to consider for matching. Defaults to 5.
-            similarity_name (str, optional): One of "dice", "jaccard", "cosine", or "overlap". 
+            similarity_name (str, optional): One of "dice", "jaccard", "cosine", or "overlap".
                     Similarity measure to be used. Defaults to 'jaccard'.
             min_match_length (int, optional): TODO: ??. Defaults to 3.
             accepted_semtypes (List[str], optional): Set of UMLS semantic types concepts should belong to.
                 Semantic types are identified by the letter "T" followed by three numbers
-                (e.g., "T131", which identifies the type "Hazardous or Poisonous Substance"). 
+                (e.g., "T131", which identifies the type "Hazardous or Poisonous Substance").
                 Defaults to constants.ACCEPTED_SEMTYPES.
             verbose (bool, optional): TODO:??. Defaults to False.
-        
+
         Raises:
             ValueError: Raises a ValueError if QuickUMLS was installed for a language that is not currently supported TODO: verify this?
             OSError: Raises an OSError if the required Spacy model was not installed.
@@ -123,10 +119,6 @@ def __init__(
 
         self.accepted_semtypes = accepted_semtypes
 
-        self.ss_db = toolbox.SimstringDBReader(
-            simstring_fp, similarity_name, threshold
-        )
-        self.cuisem_db = toolbox.CuiSemTypesDB(cuisem_fp)
         try:
             self.nlp = spacy.load(spacy_lang)
         except OSError:
@@ -139,10 +131,15 @@ def __init__(
                 constants.SPACY_LANGUAGE_MAP.get(self.language_flag, 'xx')
             )
             raise OSError(msg)
+        
+        self.ss_db = toolbox.SimstringDBReader(
+            simstring_fp, similarity_name, threshold
+        )
+        self.cuisem_db = toolbox.CuiSemTypesDB(cuisem_fp)
 
     def get_info(self):
         """Computes a summary of the matcher options.
-        
+
         Returns:
             Dict: Dictionary containing information on the QuicUMLS instance.
         """
@@ -372,10 +369,10 @@ def _make_token_sequences(self, parsed):
             for j in xrange(
                     i + 1, min(i + self.window, len(parsed)) + 1):
                 span = parsed[i:j]
-                
+
                 if not self._is_longer_than_min(span):
                     continue
-                
+
                 yield (span.start_char, span.end_char, span.text)
 
     def _print_verbose_status(self, parsed, matches):
@@ -396,18 +393,18 @@ def match(self, text, best_match=True, ignore_syntax=False):
         """Perform UMLS concept resolution for the given string.
 
         [extended_summary]
-        
+
         Args:
             text (str): Text on which to run the algorithm
 
             best_match (bool, optional): Whether to return only the top match or all overlapping candidates. Defaults to True.
             ignore_syntax (bool, optional): Wether to use the heuristcs introduced in the paper (Soldaini and Goharian, 2016). TODO: clarify,. Defaults to False.
-        
+
         Returns:
             List: List of all matches in the text
             TODO: Describe format
         """
-        
+
         parsed = self.nlp(u'{}'.format(text))
 
         if ignore_syntax:
diff --git a/quickumls/install.py b/quickumls/install.py
@@ -10,8 +10,8 @@
 from six.moves import input
 
 # project modules
-from toolbox import countlines, CuiSemTypesDB, SimstringDBWriter, mkdir
-from constants import HEADERS_MRCONSO, HEADERS_MRSTY, LANGUAGES
+from .toolbox import countlines, CuiSemTypesDB, SimstringDBWriter, mkdir
+from .constants import HEADERS_MRCONSO, HEADERS_MRSTY, LANGUAGES
 
 try:
     from unidecode import unidecode
@@ -116,7 +116,36 @@ def parse_and_encode_ngrams(extracted_it, simstring_dir, cuisty_dir):
         cuisty_db.insert(term, cui, stys, preferred)
 
 
-def driver(opts):
+def parse_args():
+    ap = argparse.ArgumentParser()
+    ap.add_argument(
+        'umls_installation_path',
+        help=('Location of UMLS installation files (`MRCONSO.RRF` and '
+              '`MRSTY.RRF` files)')
+    )
+    ap.add_argument(
+        'destination_path',
+        help='Location where the necessary QuickUMLS files are installed'
+    )
+    ap.add_argument(
+        '-L', '--lowercase', action='store_true',
+        help='Consider only lowercase version of tokens'
+    )
+    ap.add_argument(
+        '-U', '--normalize-unicode', action='store_true',
+        help='Normalize unicode strings to their closest ASCII representation'
+    )
+    ap.add_argument(
+        '-E', '--language', default='ENG', choices=LANGUAGES,
+        help='Extract concepts of the specified language'
+    )
+    opts = ap.parse_args()
+    return opts
+
+
+def main():
+    opts = parse_args()
+
     if not os.path.exists(opts.destination_path):
         msg = ('Directory "{}" does not exists; should I create it? [y/N] '
                ''.format(opts.destination_path))
@@ -172,28 +201,4 @@ def driver(opts):
 
 
 if __name__ == '__main__':
-    ap = argparse.ArgumentParser()
-    ap.add_argument(
-        'umls_installation_path',
-        help=('Location of UMLS installation files (`MRCONSO.RRF` and '
-              '`MRSTY.RRF` files)')
-    )
-    ap.add_argument(
-        'destination_path',
-        help='Location where the necessary QuickUMLS files are installed'
-    )
-    ap.add_argument(
-        '-L', '--lowercase', action='store_true',
-        help='Consider only lowercase version of tokens'
-    )
-    ap.add_argument(
-        '-U', '--normalize-unicode', action='store_true',
-        help='Normalize unicode strings to their closest ASCII representation'
-    )
-    ap.add_argument(
-        '-E', '--language', default='ENG', choices=LANGUAGES,
-        help='Extract concepts of the specified language'
-    )
-    opts = ap.parse_args()
-
-driver(opts)
+    main()
diff --git a/quickumls/network.py b/quickumls/network.py
diff --git a/quickumls/server.py b/quickumls/server.py
@@ -1,12 +1,8 @@
-try:
-    from quickumls import QuickUMLS
-    from network import run_server
-except ImportError:
-    from .quickumls import QuickUMLS
-    from .MinimalServer import run_server
-
 from argparse import ArgumentParser
 
+from .core import QuickUMLS
+from .network import run_server
+
 
 def run_quickumls_server(opts):
     matcher = QuickUMLS(
@@ -22,7 +18,7 @@ def run_quickumls_server(opts):
     run_server(matcher, host=opts.host, port=opts.port, buffersize=4096)
 
 
-if __name__ == '__main__':
+def parse_args():
     ap = ArgumentParser(
         prog='QuickUMLS server',
         description=(
@@ -76,5 +72,13 @@ def run_quickumls_server(opts):
         help='return verbose information while running'
     )
 
-    opts = ap.parse_args()
+    return ap.parse_args()
+
+
+def main():
+    opts = parse_args()
     run_quickumls_server(opts)
+
+
+if __name__ == '__main__':
+    main()
diff --git a/quickumls/toolbox.py b/quickumls/toolbox.py
@@ -14,10 +14,8 @@
 import leveldb
 
 # project imports
-try:
-    from simstring import simstring
-except ImportError:
-    from .simstring import simstring
+from quickumls_simstring import simstring
+
 
 # Python version specific imports
 if six.PY2:
diff --git a/requirements.txt b/requirements.txt
@@ -3,3 +3,4 @@ numpy>=1.8.2
 spacy>=1.6.0
 unidecode>=0.4.19
 nltk>=3.3
+quickumls_simstring>=1.1.5r1
diff --git a/setup.py b/setup.py
diff --git a/setup_simstring.sh b/setup_simstring.sh

Original file line number	Diff line number	Diff line change
`@@ -0,0 +1,2 @@`
	`1`	`+include requirements.txt`
	`2`	`+include README.md`
Original file line number	Diff line number	Diff line change
`@@ -0,0 +1,3 @@`
	`1`	`+from .core import QuickUMLS`
	`2`	`+from .client import get_quickumls_client`
	`3`	`+from .about import *`