Generate positive pairs (#87)

## Description

This PR creates a function that generates positive pairs of given LOINC
data and writes them to a file for use in model training.

## Related Issues

Closes #61 

## Additional Notes:

While I couldn't test it on the LOINC files that we haven't made yet, I
did some testing on some dummy files and directories on my local (since
it's a function that produces file output and will be run one-off I
didn't see much point in unit tests beyond making sure it works).

That means this PR is actually **not blocked** by other dependencies,
we'll just need to wire up the file paths once that's done.

Author: bamader

data_curation/generation.py

@@ -0,0 +1,69 @@
+import random
+from io import TextIOWrapper
+
+BASE_FILE_PATH = ""
+OUT_FILE_PATH = ""
+
+
+def generate_positive_pairs(file_handle: str, num_examples: int, out_file: str):
+    """
+    Given the location of one or more files of LOINC codes and some corresponding
+    augmented examples for those codes, this function compiles a list of
+    positive pairs that can be read for model training. A positive pair is a
+    tuple of the form (original_loinc_code, augmented_example_of_code).
+
+    :param file_handle: Either the path to a specific file of LOINC codes and
+      examples, or the prefix path for multiple data files across name variants.
+    :param num_examples: The number of positive pairs to generate. If -1, one
+      positive pair will be created for every element in the pool spanned by
+      the files accessible via the handle parameter.
+    :param out_file: The destination at which to write the positive pair file.
+    :returns: None
+    """
+
+    handle_parts = file_handle.split(".")
+    data_pool = []
+    pairs = []
+
+    # Given handle is a prefix to multiple files, so we'll use naming
+    # conventions to open the three appropriate ones
+    if len(handle_parts) == 0 or handle_parts[-1] != "txt":
+        for variant in ["lcn.txt", "sn.txt", "dn.txt"]:
+            with open(file_handle + "_" + variant, "r") as fp:
+                _append_to_data_pool(fp, data_pool)
+
+    # Handle is actually a file, can just open that
+    else:
+        with open(file_handle, "r") as fp:
+            _append_to_data_pool(fp, data_pool)
+
+    # Pre-specified number of examples to generate
+    # If num_examples is -1, that's "generate all" mode, where we
+    # produce one positive pair per code in the data pool, but we
+    # can achieve that just by not truncating the pool list
+    if num_examples != -1:
+        random.shuffle(data_pool)
+        data_pool = data_pool[:num_examples]
+
+    for element in data_pool:
+        pool_parts = element.split(":")
+        base_code = pool_parts[0].strip()
+        augmented_examples = pool_parts[1].strip().split("|")
+
+        # Randomly choose one of the augmented examples to pair
+        chosen_ex = random.choice(augmented_examples)
+        pairs.append((base_code, chosen_ex.strip()))
+
+    # Now we just write the created examples to the output file
+    with open(out_file, "w") as fp:
+        for pair in pairs:
+            fp.write(pair[0] + "|" + pair[1] + "\n")
+
+
+def _append_to_data_pool(fp: TextIOWrapper, data_pool):
+    """
+    Simple helper method to append non-blank lines to a list.
+    """
+    for line in fp:
+        if line.strip() != "":
+            data_pool.append(line.strip())