GutBrainIE 2026 — Our System Summary

Team 6.0 — Task T611 (NER) + Task T612 (NERD)

1. Overview

GutBrainIE@CLEF26 Task 6: Biomedical IE on gut-brain axis PubMed abstracts. We participate in two subtasks:

Subtask	Name	Description	Match tuple
T611 (NER)	Named Entity Recognition	Extract + classify entities (13 types)	(start_idx, end_idx, location, text_span, label)
T612 (NERD)	NER + Disambiguation	NER + link to ontology URI	(start_idx, end_idx, location, text_span, label, uri)

T611: Input → BERT+CRF → BIO-tagged entities → Cross-model voting → NER predictions
T612: NER predictions → Entity Linking (lookup + semantic matching) → NERD predictions

Input: PubMed abstracts with title + abstract
T611 Output: Entities with (start_idx, end_idx, location, text_span, label)
T612 Output: Entities with (start_idx, end_idx, location, text_span, label, uri)

2. Repository Structure

gutbrainie26/
├── CLAUDE.md                          # Project conventions
├── experiments.md                     # Full experiment log with results
├── configs/                           # TOML experiment configs
│   ├── exp1.4_*.toml                  # BiomedBERT gold+silver (best base)
│   ├── exp1.6_*.toml                  # BiomedBERT + original bronze
│   ├── exp1.7_*.toml                  # BiomedBERT + relabeled bronze
│   ├── exp1.8_*.toml                  # BioLinkBERT + relabeled bronze
│   └── exp2.2.1_*.toml               # BioLinkBERT gold+silver
├── src/
│   ├── T611/
│   │   ├── model_with_crf.py         # BertCrfForTokenClassification
│   │   ├── train.py                  # Training loop
│   │   └── infer.py                  # Standard inference
│   ├── T612/
│   │   ├── linker.py                 # Multi-layer entity linker (Dict+SapBERT+TypePrior)
│   │   ├── build_kb.py               # Build KB from ontologies + training data
│   │   ├── build_index.py            # SapBERT encoding + FAISS index
│   │   ├── eval_nel.py               # Oracle linking + full NERD evaluation
│   │   └── infer_nerd.py             # Test-time NERD inference
│   └── utils/
│       ├── config_parsing.py         # TOML config parser (Pydantic)
│       ├── chunk_articles.py         # Sentence-boundary chunking
│       └── load_data.py              # Dataset loading
├── tools/
│   ├── tokenize_and_align.py         # Tokenize + BIO label alignment
│   ├── infer_long.py                 # Chunked inference (splits >512 abstracts)
│   ├── infer_ensemble.py             # Multi-seed CRF ensemble inference
│   ├── ensemble_vote.py              # Cross-model majority voting
│   ├── eval_ner_detailed.py          # Per-category evaluation
│   └── relabel_bronze.py             # Re-annotate bronze with trained model
├── data/
│   ├── label_mappings.json           # 27 BIO labels
│   ├── 26/gutbrainie2026/            # Raw 2026 annotations (gold/silver/bronze/dev)
│   ├── 25/                           # 2025 silver annotations
│   ├── processed/                    # Tokenized datasets
│   ├── augmentations/                # Relabeled bronze data
│   └── nel/                          # Entity linking artifacts (KB, FAISS index, dictionary)
├── experiments/                      # Model outputs per experiment
│   ├── biomedbert_25s_26gs_*/        # exp1.4 models
│   ├── biomedbert_25s_26gsb_*/       # exp1.6 models
│   ├── biomedbert_25s_26gs_relab_*/  # exp1.7 models
│   ├── biolinkbert_25s_26gs_*/       # exp2.2.1 models
│   ├── biolinkbert_25s_26gs_relab_*/ # exp1.8 models
│   └── cross_model_*/               # Voting ensemble outputs
├── models/                           # Local model weights
│   └── BiomedNLP-BiomedBERT-base-uncased-abstract-fulltext/
└── paper/                            # LaTeX build artifacts

3. Model Architecture

BertCrfForTokenClassification (src/T611/model_with_crf.py):

Input → BERT(768) → Dropout(0.1) → Linear(768→27) → CRF(27) → Viterbi decode

BIO tagging: 27 labels (O + B/I × 13 entity types)
CRF enforces valid BIO transitions
Loss: negative log-likelihood from CRF

4. Training Configuration

Parameter	Value
Base models	BiomedBERT-base (110M), BioLinkBERT-base (110M)
LR (BERT / classifier / CRF)	2e-5 / 2e-4 / 2e-3
Batch size	32
Max epochs	40
Early stopping	patience=5 on seqeval F1
Warmup	10%
Max seq len	512
Seeds	7, 11, 17

5. Data

Dataset	Articles	Chunks	Usage
2025 silver	773	—	NER training
2026 gold (train)	80	—	NER training
2026 silver	80	—	NER training
gold+silver total	933	4146	exp1.4, exp2.2.1
+ relabeled bronze	+2643	+6222	exp1.7, exp1.8
gold+silver+relab total	3576	10368	exp1.7, exp1.8
2026 dev	80	—	Evaluation

6. Pipeline

Training

# 1. Tokenize & align BIO labels
python -m tools.tokenize_and_align --config configs/exp1.4_*.toml

# 2. Train (3 seeds)
python -m src.T611.train --config configs/exp1.4_*_s11.toml
python -m src.T611.train --config configs/exp1.4_*_s7.toml
python -m src.T611.train --config configs/exp1.4_*_s17.toml

Inference & Evaluation

# 3. Single-model chunked inference
python -m tools.infer_long --config <config> --model_dir <model> --dev_json <dev> --output <pred>

# 4. Within-model ensemble (average CRF emissions + transitions)
python -m tools.infer_ensemble --model_dirs <s7> <s11> <s17> --dev_json <dev> --output <pred> --chunked

# 5. Cross-model majority voting
python -m tools.ensemble_vote --predictions <pred1> <pred2> <pred3> --output <voted> --min_votes 2

# 6. Evaluate
python -m tools.eval_ner_detailed --predictions <pred> --ground_truth <dev> --output <eval>

7. T611 Current Best Results (2026 Dev)

System	Macro-F1	Micro-F1
Best single ensemble (exp1.7, BiomedBERT+relab bronze)	0.8034	0.8522
3-way vote ≥2/3 (exp1.4+1.6+2.2.1), chunked	0.8056	0.8552
5-way vote ≥3/5 (all 5 ensembles), chunked	0.8059	0.8543

8. T612 — Entity Linking (NERD)

8.1 Task Definition

Given NER predictions from T611, assign each entity a URI from biomedical ontologies. Evaluation matches on the full 6-tuple: (start_idx, end_idx, location, text_span, label, uri).

8.2 Target Ontologies (per entity type)

Entity Type	Priority Order
Anatomical Location	NCIT > UMLS > GBIE
Animal	NCIT > NCBITaxon > UMLS > GBIE
Bacteria	NCIT > NCBITaxon > MESH > OMIT > UMLS > GBIE
Biomedical Technique	NCIT > OMIT > NCBITaxon > UMLS > GBIE
Chemical	NCIT > CHEBI > OMIT > UMLS > GBIE
Dietary Supplement	NCIT > CHEBI > NCBITaxon > OMIT > MESH > UMLS > GBIE
DDF	NCIT > OMIT > NCBITaxon > UMLS > GBIE
Drug	NCIT > CHEBI > OMIT > NCBITaxon > UMLS > GBIE
Food	NCIT > UMLS > GBIE
Gene	NCIT > OMIT > CHEBI > UMLS > GBIE
Human	NCIT > MESH > UMLS > GBIE
Microbiome	NCIT > NCBITaxon > OHMI > UMLS > GBIE
Statistical Technique	NCIT > STATO > SWO > UMLS > GBIE

8.3 Available Resources

Training data URIs: All gold/silver/bronze annotations include URIs (18512 unique (text, label) → uri mappings in gold+silver+silver25)
Ontology definitions: 3699 URIs with 52263 aliases (names, synonyms, definitions) from GutBrainIE_2026_Baseline/Train/NEL/definitions/
Baseline approach (from organizers): Exact match lookup → PubMedBERT semantic similarity via txtai
Evaluation: GutBrainIE_2026_Baseline/Eval/evaluate.py → eval_submission_NERD()

8.4 Approach: 3-Layer Entity Linker

NER predictions → Layer 1: Dictionary → Layer 2: SapBERT + FAISS → Layer 3: Type Prior → URI

Layer 1 — Dictionary Lookup (handles 68.2% of dev entities, 89.8% accuracy)

Gold+Silver+Silver25 (text_span, label) → majority_vote_uri
18512 mappings from data/nel/dictionary_gs.json

Layer 2 — SapBERT Similarity (handles 31.8% of dev entities, 95.6% accuracy)

Model: cambridgeltl/SapBERT-from-PubMedBERT-fulltext
FAISS index over 52263 alias embeddings from KB (data/nel/faiss.index)
KB built from: ontology names/definitions + training corpus surface forms
Top-50 retrieval, re-ranked with type prior

Layer 3 — Type-Compatibility Prior (alpha=0.05)

score = sapbert_sim + 0.05 * P(label|uri)
Prior from training data: data/nel/uri_label_prior.json
Marginal improvement (+0.2%), but stable

Fallback — Generic concept URI per entity type (never triggered on dev)

8.5 Code & Data Artifacts

Artifact	Path
KB builder	`src/T612/build_kb.py`
FAISS index builder	`src/T612/build_index.py`
Entity linker	`src/T612/linker.py`
Evaluation script	`src/T612/eval_nel.py`
Knowledge base	`data/nel/kb.json` (3699 URIs, 52263 aliases)
FAISS index	`data/nel/faiss.index` (52263 × 768 vectors)
Dictionary	`data/nel/dictionary_gs.json` (18512 mappings)
URI-label prior	`data/nel/uri_label_prior.json`

8.6 How to Run

Step 1: Build Knowledge Base & FAISS Index (one-time setup)

# Build KB from ontology definitions + training data surface forms
python -m src.T612.build_kb

# Encode all KB aliases with SapBERT and build FAISS index
python -m src.T612.build_index

This creates: data/nel/kb.json, data/nel/faiss.index, data/nel/index_meta.json, data/nel/dictionary_gs.json, data/nel/uri_label_prior.json.

Step 2: Evaluate Linking Quality (oracle NER)

# Oracle linking accuracy (uses gold NER spans, isolates linking quality)
python -m src.T612.eval_nel --nel_dir data/nel/ --alpha 0.05 --top_k 20

Step 3: Full NERD Evaluation (NER + NEL)

# Evaluate full NERD F1 using predicted NER spans
python -m src.T612.eval_nel --nel_dir data/nel/ --alpha 0.05 --top_k 20 \
    --ner_pred experiments/cross_model_ensemble/predictions_vote3_chunked.json \
    --output experiments/nerd/eval_nerd_final.json

Step 4: Test-Time Inference (produce submission file)

# Link NER predictions to URIs — produces NERD output JSON
python -m src.T612.infer_nerd \
    --ner_pred <NER_PREDICTIONS.json> \
    --nel_dir data/nel/ \
    --output <OUTPUT_NERD.json> \
    --alpha 0.05 --top_k 20

Input format (NER_PREDICTIONS.json): output of T611 NER inference — list of articles, each with entities containing start_idx, end_idx, location, text_span, label.

Output format (OUTPUT_NERD.json): same structure with uri field added to each entity.

Full End-to-End (T611 NER → T612 NERD)

# 1. Run NER inference (T611)
python -m tools.ensemble_vote \
    --predictions <exp1.4_pred> <exp1.6_pred> <exp2.2.1_pred> \
    --output experiments/cross_model_ensemble/predictions_vote3_chunked.json \
    --min_votes 2

# 2. Link entities to URIs (T612)
python -m src.T612.infer_nerd \
    --ner_pred experiments/cross_model_ensemble/predictions_vote3_chunked.json \
    --nel_dir data/nel/ \
    --output experiments/nerd/submission_nerd.json \
    --alpha 0.05 --top_k 20

8.7 Results (2026 Dev)

Oracle NER (linking accuracy in isolation): 91.63% (2310/2521)

Full NERD (with best NER ensemble):

System	Macro-P	Macro-R	Macro-F1	Micro-P	Micro-R	Micro-F1
Organizer baseline	0.3820	0.4045	0.3916	0.4281	0.4522	0.4398
Ours (3-way vote NER + SapBERT linker)	0.7854	0.6967	0.7313	0.8179	0.7588	0.7872

Per-label NERD F1 (full 6-tuple matching including URI):

Entity Type	P	R	F1	TP	FP	FN
human	0.882	0.854	0.868	164	22	28
drug	0.877	0.853	0.865	64	9	11
animal	0.888	0.836	0.861	127	16	25
DDF	0.872	0.840	0.856	666	98	127
microbiome	0.798	0.818	0.808	189	48	42
anatomical location	0.800	0.781	0.790	132	33	37
bacteria	0.788	0.710	0.747	130	35	53
dietary supplement	0.714	0.703	0.709	45	18	19
chemical	0.735	0.683	0.708	250	90	116
statistical technique	0.657	0.657	0.657	23	12	12
biomedical technique	0.731	0.547	0.626	76	28	63
food	0.813	0.441	0.571	26	6	33
gene	0.656	0.333	0.442	21	11	42

8.8 Error Analysis

Error buckets (211 oracle linking errors, alpha=0.05):

Bucket	Count	%	Description
dict_ambiguity	176	83.4%	Dictionary picked wrong majority URI
sapbert_wrong_rank	34	16.1%	GT in top-50 but not top-1 (all in top-5)
sapbert_not_in_top50	1	0.5%	GT URI not in FAISS top-50

SapBERT top-k recall: Top-1=95.6%, Top-3=99.9%, Top-5=100%

Dictionary ambiguity breakdown:

89/176: GT URI never seen in training for that (text,label) — unfixable annotation noise
87/176: GT is minority URI — theoretically fixable with context-aware disambiguation

Cross-encoder re-ranker (attempted, abandoned):

Trained PubMedBERT cross-encoder on 34.9K (mention, candidate_name) pairs
3-seed ensemble (seeds 7, 11, 17), val accuracy ~79%
Best result: beta=0.01 → 91.67% (+1 entity) — negligible improvement
Root cause: cross-encoder and SapBERT solve the same text-pair matching problem, but SapBERT compares against all 52K aliases while cross-encoder only sees preferred names

Substring/normalization matching (attempted, abandoned):

Normalized match (order-invariant, stopword-removed): 16 hits, 69% accuracy
Substring match (longest dictionary substring): 216 hits at 60% accuracy
Both strictly worse than SapBERT's 95.6% — adding them BEFORE SapBERT reduces overall accuracy

Key findings:

Annotator inconsistency dominates errors: "gut microbiota" has 3-4 valid URIs (OMIT, OHMI, NCIT, MESH) — same terminology, different meaning in context
Gene/interleukin confusion: IL-6, TNF-α, IFN-γ annotated with NCIT in some vs CHEBI in others
SapBERT is near-optimal: 95.6% top-1 accuracy; no heuristic or learned re-ranker improves on it
Practical ceiling: ~91.6% oracle linking accuracy is near the limit given annotation noise

See TASK6.1.2.md and experiments_612.md for full details.

9. Key Design Decisions

BIO > BIOES: 27 labels outperforms 53 labels at this data scale
Single [CLS] prefix: No semantic prefix tokens ([TITLE]/[ABSTRACT])
Sentence-boundary chunking: Both train and inference, handles >512 token abstracts
Separate LRs: BERT 2e-5, classifier 2e-4, CRF 2e-3
Relabeled bronze: Re-annotate bronze with best model before use as training data
Two-level ensemble: CRF parameter averaging (within-model) + majority voting (cross-model)
BiomedBERT > BioLinkBERT > BioBERT > SciBERT: Domain-specific PubMed pretraining wins
Base > Large: 4K training samples insufficient for 340M models

10. Environment

Python: D:/Develop/programming/ai/envs/gutbrain26/python.exe
PyTorch: 2.6.0+cu124
GPU: RTX 4090 Laptop

Navigation Menu

Uh oh!

GutBrainIE 2026 — Our System Summary

1. Overview

2. Repository Structure

3. Model Architecture

4. Training Configuration

5. Data

6. Pipeline

Training

Inference & Evaluation

7. T611 Current Best Results (2026 Dev)

8. T612 — Entity Linking (NERD)

8.1 Task Definition

8.2 Target Ontologies (per entity type)

8.3 Available Resources

8.4 Approach: 3-Layer Entity Linker

8.5 Code & Data Artifacts

8.6 How to Run

Step 1: Build Knowledge Base & FAISS Index (one-time setup)

Step 2: Evaluate Linking Quality (oracle NER)

Step 3: Full NERD Evaluation (NER + NEL)

Step 4: Test-Time Inference (produce submission file)

Full End-to-End (T611 NER → T612 NERD)

8.7 Results (2026 Dev)

8.8 Error Analysis

9. Key Design Decisions

10. Environment

Uh oh!

FilesExpand file tree

OFFICIAL.md

Latest commit

History

OFFICIAL.md

File metadata and controls

GutBrainIE 2026 — Our System Summary

1. Overview

2. Repository Structure

3. Model Architecture

4. Training Configuration

5. Data

6. Pipeline

Training

Inference & Evaluation

7. T611 Current Best Results (2026 Dev)

8. T612 — Entity Linking (NERD)

8.1 Task Definition

8.2 Target Ontologies (per entity type)

8.3 Available Resources

8.4 Approach: 3-Layer Entity Linker

8.5 Code & Data Artifacts

8.6 How to Run

Step 1: Build Knowledge Base & FAISS Index (one-time setup)

Step 2: Evaluate Linking Quality (oracle NER)

Step 3: Full NERD Evaluation (NER + NEL)

Step 4: Test-Time Inference (produce submission file)

Full End-to-End (T611 NER → T612 NERD)

8.7 Results (2026 Dev)

8.8 Error Analysis

9. Key Design Decisions

10. Environment