FindMyText is an open-source Python package for efficiently detecting whether a given text appears, in part or in full, within a large text corpus. It is particularly suited for verifying the presence of copyrighted or licensed material in large, web-crawled corpora. This can notably provide important insights into which texts have been used to pre-train LLMs.
The tool builds on standard document fingerprinting techniques, and extends them with a novel mechanism that explicitly captures sequences (chains) of matching fingerprints. This makes it robust to near-verbatim copies — texts that share the same content but with minor differences due to OCR errors, formatting variants, text normalisation, or added boilerplate. Leveraging a distributed, disk-based indexing framework, FindMyText scales to large corpora that cannot be held in memory.
A full step-by-step walkthrough is available in demo.ipynb.
git clone https://github.com/your-org/FindMyText.git
cd FindMyText
pip install -e .The first step is to index your corpus. Fingerprints are extracted in parallel and then merged into a single disk-based index:
from findmytext import index_builder
# `corpus` is any iterable of dicts with "text" and "id" fields
files = index_builder.index_data_parallel(corpus, "my_fingerprints", n_workers=4)
index_builder.merge_indexes_from_dir("my_fingerprints", "my_index")The resulting index is stored on disk and memory-mapped at query time, so it scales to corpora that are too large to fit in RAM.
The index_builder can also be used directly from the command line:
# Step 1: extract fingerprints from a corpus file into intermediate shards
python -m findmytext.index_builder index corpus.jsonl my_fingerprints --nb_workers 4
# Step 2: merge shards into a final disk-based index
python -m findmytext.index_builder merge my_fingerprints my_indexOnce the index is built, detection is a single call:
from findmytext import detectors
detector = detectors.FingerprintChainDetector("my_index")
scores = detector.get_containment_scores(query_text)
# `scores` is a dict mapping document IDs to containment scores
best_match_id = max(scores, key=scores.get)
print(f"Best match: {best_match_id} (score: {scores[best_match_id]:.3f})")The text in query_text does not need to be exactly identical to the one found in the corpus. FindMyText is designed to be robust to small differences between the documents.
To inspect a detected match in detail, use the built-in local alignment:
from findmytext import oracle
alignment = oracle.align(query_text, corpus_document_text)
alignment.show()-
Fingerprinting: each document is tokenised and converted to a set of
$k$ -gram hashes using the winnowing algorithm, which selects a representative subset of hashes from each sliding window. -
Inverted index: fingerprints are stored in a disk-based inverted index mapping each hash to the list of
(doc_id, position)pairs where it was observed. - Chain detection: at query time, the fingerprints of the query are looked up in the index. Rather than simply counting matches, FindMyText clusters matching fingerprints by their relative positions to identify chains — contiguous sequences of matches — and uses the total length of those chains as the containment score.
FindMyText is released under an MIT License.
The MIT License is a short and simple permissive license allowing both commercial and non-commercial use of the software. The only requirement is to preserve the copyright and license notices (see file License). Licensed works, modifications, and larger works may be distributed under different terms and without source code.
If you use FindMyText in your research, please cite our paper:
@article{findmytext2026,
authors = {Lars Henry Berge Olsen and Pierre Lison and Martin Jullum and Mark Anderson},
title = {FindMyText: Robust, Scalable Detection of Text Containment in Large Web-Crawled Corpora},
year = {2026}
}