Skip to content

About

WhatFontIs-Bench — A Synthetic Benchmark for Font Family Identification. v1.0: 600 fonts, 11,995 images of words on real surfaces and scenes

Resources

Stars

1 star

Watchers

0 watching

Forks

Latest commit

 

History

7 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

WhatFontIs-Bench — A Synthetic Benchmark for Font Family Identification

A synthetic test set for font family identification: a word or a short text, set in a known font, printed or painted on real surfaces and in real scenes, with the exact font, the text and the position of every letter recorded for each image.

Made by WhatFontIs, the font finder that identifies fonts from images, to measure how well a tool can find the font in a real-looking photo.

Official page: whatfontis.com/WhatFontIs-Bench.html

Paper: A. Cuibari, WhatFontIs-Bench: A Synthetic Benchmark for Font Family Identification in Real-Looking Images, 2026. doi:10.5281/zenodo.22876579

Versions

Version Fonts Images Data Images download
v2.0 3,000 60,000 v2/ release v2.0
v1.0 600 11,995 v1/ release v1.0

Each version is a separate folder and a separate release, so results on different versions are not mixed. This page describes v2.0 first, then v1.0.

v2.0 contents

Fonts 3,000: 1,059 sans-serif, 1,000 serif, 500 slab serif, 300 script, 141 monospaced
Font sources 1,402 Adobe Fonts, 713 Creative Fabrica, 595 Google Fonts, 290 DaFont (commercial-use license)
Images 60,000 (20 per font)
Text a short text per image, from 1 to 27 characters; its length and case follow real searches on WhatFontIs
Size long side never over 2000 px

What is new in v2.0

In v1.0 every image holds one word of 7–12 letters. Real searches are shorter: a logo, a few capitals, a single letter. In v2.0 the length and the case of the text follow the statistics of real searches on WhatFontIs, so the set measures the case that matters most in practice: identifying a font from only a few letters.

  • Different letters in the text: 1 in 4.5 % of the images, 2 in 6.2 %, 3 in 14.5 %, 4 in 16.0 %, 5 or 6 in 31.8 %, 7 or more in 27.0 %. Repeated letters count once: ANNA has 2 different letters, Anna 3.
  • Case: all capitals 48.0 %, capitalised 18.2 %, mixed 18.5 %, lower case 10.8 %, digits only 4.5 %. Digits appear in 9.7 % of the texts and & in 1.5 %, only with fonts that draw them.
  • Script fonts: 300, a category v1.0 does not have.
  • Five times more fonts: 3,000 instead of 600.

Each font has 20 images, with the same plan as v1.0:

Type Easy Medium Hard Per font In the set
texture: the text printed or painted straight on a close-up photo of a real surface (wood, wall, concrete, brick, metal, fabric, leather, paper, cardboard) 3 2 5 10 30,000
scene: the text painted inside a real photo of a place 1 2 3 6 18,000
object: the text on a printed object (label, box, book cover, poster, menu, shop sign, packaging, business card, painted wall) 1 1 2 4 12,000

Difficulty levels:

  • easy: capital height 130–220 px, little blur and noise, JPEG quality 88–95, almost even light
  • medium: capital height 100–180 px, blur σ 0.2–0.8, noise 1–3.5, JPEG 78–90, uneven light
  • hard: capital height 100–140 px, blur σ 0.5–1.3, noise 2–6, JPEG 65–80, strong uneven light, and often a cast shadow over part of the text (16,986 images), a glare spot (14,015) or light print wear (1,804)

As in v1.0, the images are frontal; texture and scene images add only a small camera deviation (corners moved at most 1.5 %, rotation at most 1°), saved in the labels. Text contrast is at least 70 grey levels (50 on the hard level), no letter is clipped, and fonts that only have capitals (480 of them) are written in capitals.

v2.0 results: WhatFontIs API

The WhatFontIs API on all 60,000 images, October 2026, in its two modes: the default one and mode=pro. The API searches its whole catalogue of over 1.2 million fonts, not only the 3,000 fonts of this set. As for v1.0, a result is correct when it is the right font family, in any weight or from any source.

Images Default: correct at 1 in top 5 in top 20 mode=pro: correct at 1 in top 5 in top 20
All images 60,000 82.2 % 88.0 % 89.9 % 88.1 % 94.3 % 95.6 %
texture 30,000 82.4 % 88.1 % 89.9 % 88.9 % 95.0 % 96.1 %
scene 18,000 79.6 % 85.6 % 87.7 % 86.0 % 92.4 % 94.0 %
object 12,000 85.7 % 91.4 % 93.0 % 89.2 % 95.4 % 96.8 %
easy 15,000 83.0 % 88.1 % 89.7 % 89.2 % 95.1 % 96.5 %
medium 15,000 81.4 % 86.9 % 88.6 % 87.6 % 94.0 % 95.3 %
hard 30,000 82.3 % 88.5 % 90.6 % 87.8 % 94.0 % 95.3 %
sans-serif 21,180 76.7 % 83.1 % 86.0 % 84.1 % 90.5 % 92.5 %
serif 20,000 84.7 % 90.7 % 92.1 % 88.9 % 95.7 % 96.9 %
slab serif 10,000 86.4 % 91.1 % 92.0 % 91.2 % 97.3 % 97.9 %
script 6,000 86.8 % 91.5 % 92.3 % 93.7 % 97.4 % 97.9 %
monospaced 2,820 81.8 % 87.5 % 91.0 % 89.4 % 94.9 % 96.8 %
Median time per image 4.4 s 9.0 s

By the number of different letters in the text, the part where v2.0 differs from v1.0:

Different letters Images Default: correct at 1 in top 5 in top 20 mode=pro: correct at 1 in top 5 in top 20
1 2,697 43.2 % 53.5 % 59.9 % 48.2 % 62.4 % 69.1 %
2 3,713 64.7 % 73.7 % 78.6 % 73.8 % 85.9 % 89.8 %
3 8,695 77.3 % 83.6 % 86.1 % 86.2 % 93.4 % 95.1 %
4 9,625 82.6 % 88.5 % 90.4 % 89.6 % 95.7 % 96.9 %
5-6 19,056 86.8 % 92.0 % 93.2 % 92.0 % 97.2 % 97.8 %
7 or more 16,214 89.8 % 94.4 % 95.2 % 93.5 % 97.8 % 98.2 %

Images without an answer count as misses: 280 in the default mode, where no letters were found, and 50 with mode=pro (37 where the request failed, 13 where no letters were found). Times are the median per image on the live API, one request at a time, measured on 100 of the images.

v2.0 download

The images are in the release v2.0 as 15 zip files of 4,000 images each:

File Size Images
WhatFontIs-Bench_v2.0_images_part01.zip 477 MB 00000–04002
WhatFontIs-Bench_v2.0_images_part02.zip 525 MB 04003–08008
WhatFontIs-Bench_v2.0_images_part03.zip 455 MB 08009–12013
WhatFontIs-Bench_v2.0_images_part04.zip 455 MB 12014–16019
WhatFontIs-Bench_v2.0_images_part05.zip 455 MB 16020–20024
WhatFontIs-Bench_v2.0_images_part06.zip 481 MB 20025–24027
WhatFontIs-Bench_v2.0_images_part07.zip 478 MB 24028–28031
WhatFontIs-Bench_v2.0_images_part08.zip 460 MB 28032–32038
WhatFontIs-Bench_v2.0_images_part09.zip 455 MB 32039–36044
WhatFontIs-Bench_v2.0_images_part10.zip 433 MB 36045–40055
WhatFontIs-Bench_v2.0_images_part11.zip 455 MB 40056–44068
WhatFontIs-Bench_v2.0_images_part12.zip 493 MB 44069–48081
WhatFontIs-Bench_v2.0_images_part13.zip 791 MB 48082–52081
WhatFontIs-Bench_v2.0_images_part14.zip 795 MB 52082–56081
WhatFontIs-Bench_v2.0_images_part15.zip 787 MB 56082–60081

With the GitHub CLI: gh release download v2.0 --repo whatfontis/WhatFontIs-Bench

Unzip all of them into v2/; they create v2/scenes/NNNNN.jpg. The text crops come from python tools/make_crops.py v2.

v2.0 files

File What it holds
v2/labels.jsonl one line per image, with the same fields as v1.0, plus query_spec (the number of different characters and the case the text was drawn with)
v2/annotations.json the same set in COCO format: images, categories (one per font), annotations (word box, text, letters)
v2/fonts.json the 3,000 fonts: title, family, source, category, capitals-only flag, WhatFontIs page, number of images
v2/backgrounds.json the 153 background photos used, with source page, author and license; 1,510 object images have no photo (a painted wall fills the frame)
v2/text_pools.json the strings the texts were drawn from, by case and number of different characters
v2/quick1200.txt 1,200 image ids that give the same score as the whole set (see below)

Font files are not included.

Quick subset

v2/quick1200.txt lists 1,200 images, one font each, with the same mix of type, number of letters and category as the whole set. On every system we measured, its score was within about one point of the score on all 60,000 images (for the WhatFontIs API: 87.9 % against 88.1 % correct at 1 with mode=pro, 82.4 % against 82.2 % in the default mode), so it is a quick first check for a new model; use the whole set for final numbers.

v2.0 known gaps

  • 513 of the 600 fonts of v1.0 are also in v2.0; do not use one version to train and the other to test.
  • A single letter is often drawn almost the same way in several fonts, so one-letter images have no single possible answer; read their results as a lower bound.
  • The scene images reuse a limited number of real scenes, each with a different text, font, position and light.
  • Some fonts in the set are close look-alikes of other fonts that exist under different names.

v1.0 contents

Fonts 600: 200 sans-serif, 200 serif, 100 slab serif, 100 monospaced
Font sources 215 Adobe Fonts, 191 Creative Fabrica, 183 Google Fonts, 11 DaFont (commercial-use license)
Images 11,995 (20 per font; Colt Soft Regular has 15, see v1.0 known gaps)
Text one word per image, 7–12 letters, taken from COCO-Text v2
Size long side 442–1998 px (never over 2000), short side at least 301 px

Each font has 20 images:

Type Easy Medium Hard Per font
texture: the word printed or painted straight on a close-up photo of a real surface (wood, plaster, concrete, brick, metal, fabric, leather, paper, cardboard) 3 2 5 10
scene: the word painted on a wall inside a real photo (rooms, shops, facades, workshops) 1 2 3 6
object: the word on a printed object (label, box, book cover, poster, menu, shop sign, packaging, business card, painted wall) over a real photo 1 1 2 4

Difficulty levels:

  • easy: capital height 160–220 px, little blur and noise, JPEG quality 88–95, almost even light
  • medium: capital height 130–180 px, blur σ 0.3–0.8, noise 1.5–3.5, JPEG 78–90, uneven light
  • hard: capital height 100–140 px, blur σ 0.7–1.3, noise 3–6, JPEG 65–80, strong uneven light, and often a cast shadow over part of the word (3,354 images), a glare spot (2,747) or light print wear (357)

When a scene wall is too small for a long word at the level's size, that image uses the next smaller size range, so a few easy scene images have capitals of 130–160 px and a few medium ones of 100–130 px.

The images are frontal, with no slant or perspective, so an upright font never looks italic. Texture and scene images add only a small camera deviation, as in a real photo taken straight on: corners moved at most 1.5 %, rotation at most 1°. The exact transform is saved in the labels.

Text contrast is at least 70 grey levels (50 on the hard level). No letter is clipped. Fonts that only have capitals (107 of them) are written in capitals.

v1.0 results: WhatFontIs API

The WhatFontIs API on all 11,995 images, October 2026, in its two modes: the default one and mode=pro, which takes about two and a half times as long per image and finds more. The API searches its whole catalogue of over 1.2 million fonts, not only the 600 fonts of this set.

A result counts as correct when it is the right font family, in any weight or from any source: an image in Roboto Bold answered with Roboto Regular is correct, answered with Arial Bold it is wrong.

Images Default: correct at 1 in top 5 in top 20 mode=pro: correct at 1 in top 5 in top 20
All images 11,995 83.2 % 92.2 % 94.6 % 89.6 % 97.1 % 98.5 %
texture 5,997 83.1 % 92.3 % 94.4 % 90.1 % 97.4 % 98.8 %
scene 3,599 81.2 % 90.2 % 93.0 % 87.6 % 95.6 % 97.5 %
object 2,399 86.6 % 95.2 % 97.2 % 91.5 % 98.5 % 99.4 %
easy 2,995 86.0 % 94.1 % 95.8 % 91.3 % 97.9 % 99.1 %
medium 3,000 82.8 % 91.1 % 93.3 % 89.8 % 97.0 % 98.4 %
hard 6,000 82.1 % 91.9 % 94.5 % 88.7 % 96.7 % 98.3 %
sans-serif 4,000 75.2 % 87.5 % 91.6 % 84.1 % 94.7 % 97.3 %
serif 4,000 83.8 % 94.3 % 96.0 % 88.8 % 97.6 % 99.0 %
slab serif 1,995 92.6 % 95.2 % 95.8 % 98.1 % 99.4 % 99.5 %
monospaced 2,000 89.0 % 94.7 % 96.3 % 93.9 % 98.4 % 99.2 %
Median time per image 5.2 s 12.8 s

Images without an answer count as misses: 20 in the default mode, where no letters were found, and 31 with mode=pro, where the request failed. Times are the median per image on the live API, measured on 1,200 of the images. The previous measurement, September 2026, had a single mode: 83.7 % correct at 1, 93.3 % in top 5, 96.5 % in top 20.

v1.0 download

The images are in the release v1.0 as four zip files (they are release downloads, not files in the repository):

File Size Images
WhatFontIs-Bench_v1.0_images_part1.zip 466 MB 00000–02998
WhatFontIs-Bench_v1.0_images_part2.zip 470 MB 02999–05997
WhatFontIs-Bench_v1.0_images_part3.zip 460 MB 05998–08996
WhatFontIs-Bench_v1.0_images_part4.zip 604 MB 08997–11994

With the GitHub CLI: gh release download v1.0 --repo whatfontis/WhatFontIs-Bench

Unzip all four into v1/; they create v1/scenes/NNNNN.jpg. To also get the text crops (the word cut out of each image, unscaled), run:

pip install pillow
python tools/make_crops.py v1

This writes v1/crops/NNNNN.png.

v1.0 files

File What it holds
v1/labels.jsonl one line per image, with everything about it (see below)
v1/annotations.json the same set in COCO format: images, categories (one per font), annotations (word box, text, letters)
v1/fonts.json the 600 fonts: title, family, source, category, capitals-only flag, WhatFontIs page, number of images
v1/backgrounds.json the 153 background photos used, with source page, author and license
words.txt the word list the texts were drawn from (shared by all versions)
v1/specimens/ ten sheets showing all 600 fonts, 60 per sheet
tools/make_crops.py cuts the text crops out of the images

Font files are not included.

labels.jsonl

The main fields of each line:

  • Image and font
    • id, image (scenes/NNNNN.jpg, relative to v1/), width, height
    • text: the word in the image
    • title, family, source, category, caps_only, url: the font
    • type (texture / scene / object), difficulty (easy / medium / hard)
  • Where the text is
    • crop_box: [x0, y0, x1, y1] of the text crop
    • texture and scene images: ink_quad (the word's four corners) and char_quads (four corners per letter)
    • object images: ink_box, word_box and chars (letter boxes), plus object_box
  • How it was made
    • cap_height_px, font_px, ink_rgb, contrast, blur_sigma, noise_sigma, jpeg_quality
    • effects / hard_effects: the hard-level problems applied
    • camera: corner_jitter, rotation_deg, homography
  • Background
    • texture images: surface, texture, texture_source, texture_license
    • scene images: scene_source_file, scene_source, scene_page, scene_license
    • object images: scene (object kind), material, extras, background (file, source, page, license)

The background identifiers match the id field in backgrounds.json.

v1.0 known gaps

  • Colt Soft Regular has 15 images instead of 20: it is so wide that a 7-letter word with 160 px capitals does not fit in 2000 px, so 5 easy images are missing.
  • The scene images reuse a limited number of real scenes (each scene appears many times, with a different word, font, position and light).
  • Some fonts in the set are close look-alikes of other fonts that exist under different names.

Citation

@misc{cuibari2026whatfontisbench,
  title  = {WhatFontIs-Bench: A Synthetic Benchmark for Font Family Identification in Real-Looking Images},
  author = {Cuibari, Alexandru},
  year   = {2026},
  doi    = {10.5281/zenodo.22876579},
  url    = {https://doi.org/10.5281/zenodo.22876579}
}

Licenses and credits

  • Backgrounds: all CC0 1.0. Surface photos from Poly Haven and ambientCG; scene photos from Wikimedia Commons, found through Openverse; panorama views from Poly Haven. Per-file details are in each version's backgrounds.json.
  • Words and texts: from COCO-Text v2 (CC BY 4.0), Veit et al., COCO-Text: Dataset and Benchmark for Text Detection and Recognition in Natural Images, 2016.
  • Fonts: the typefaces belong to their designers and foundries. The images show text rendered in them; the font files are not part of this set.

About

WhatFontIs-Bench — A Synthetic Benchmark for Font Family Identification. v1.0: 600 fonts, 11,995 images of words on real surfaces and scenes

Resources

Stars

1 star

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages