Software identity resolution — Label Studio annotation

This repo builds side-by-side HTML panels from two software metadata documents (e.g. BioConda, bio.tools, Galaxy) so human annotators can decide, in Label Studio, whether the two records refer to the same software.

Annotation happens in rounds: a pilot (to teach the task), then round 1, then further rounds that re-annotate subsets based on prior results (e.g. all pairs marked Unclear or where annotators disagreed).

Pipeline

The flow is deliberately two-staged so the database is hit only once and everything downstream is reproducible and offline. Run both scripts from the repo root.

1. build_enriched_pairs.py (online — needs MongoDB) — reads a conflict-blocks file, fetches full metadata from MongoDB, expands each conflict into disambiguation pairs (pairing.build_pairs), fixes Galaxy links, and writes a shared pairs file. This is the only step that touches the database.

   python build_enriched_pairs.py \
       --conflicts data/conflicts.20260421T111602Z-8d84134d-manual_group_correction.json \
       --output data/enriched_pairs.json

2. generate_tasks.py (offline) — loads an enriched-pairs file, renders each pair to HTML with the Jinja2 template pair_panels.html.j2, assigns each pair to exactly one annotator (load-balanced, random.seed(42) for reproducibility), and writes per-annotator task chunks plus a manifest under rounds/<round>/.

   python generate_tasks.py --pairs data/enriched_pairs.14052026.json --round round_1
   # optional: --sample-size N   --chunk-size 30

   Outputs:

   • rounds/<round>/tasks/tasks_<annotator>_part_NN.json — Label Studio import files
   • rounds/<round>/manifest.json — full pair→annotator assignment record (with itemA/itemB)

   The annotator list is hardcoded in the __main__ block of generate_tasks.py; edit it there.

3. Label Studio — set the project labeling interface from labelstudio_config.html (asks Same / Different / Unclear + Confidence High/Medium/Low + a notes box), then import the task files. Each task's data.html is displayed via a HyperText tag.

4. analysis/ — scripts that parse a Label Studio export into tidy tables/plots (parse_annotations.py, plots.py, agreement.py). They currently target the pilot export; see "Analysis" below.

Repository layout

build_enriched_pairs.py  generate_tasks.py     # the pipeline (run from root)
pairing.py utils.py metadata.py database_entries.py
pair_panels.html.j2  labelstudio_config.html
software_instance/   db/                        # shared model + MongoDB packages
analysis/                                       # annotation analysis scripts
data/                                           # shared, cross-round source data
  conflict_blocks_*.json  conflicts.*.json  enriched_pairs.*.json
rounds/                                         # per-round artifacts
  pilot/   {tasks, annotations, analysis}
  round_1/ {tasks, annotations, analysis} + manifest.json
  round_N/ ...                                  # future rounds
archive/                                        # superseded one-off scripts & previews

Setup

pip install -r requirements.txt

build_enriched_pairs.py connects to MongoDB via db/mongo/mongo_adapter.py, which reads credentials from an external .env (path hardcoded in that file) and expects an SSH tunnel to the Mongo host on 127.0.0.1:27018. Task generation and analysis need no database.

Analysis

The scripts in analysis/ read a single Label Studio export and reproduce the pilot's CSVs/plots into rounds/pilot/analysis/. Round 1 exported one file per annotator (rounds/round_1/annotations/), so running these against a full round requires generalizing them to glob a round's annotations/ directory — tracked as a follow-up, not yet done.

Task JSON shape

{
  "id": "f6a24b9e-1234-4e7c-a89b-9db8fa04aabb",
  "data": {
    "pair_uid": "block-42__pair_1",
    "conflict_id": "block-42",
    "annotator": "EVA MARTIN DEL PICO",
    "html": "<div>…rendered HTML for pair A/B…</div>"
  }
}