Faces: Report the markers a migration would sample above the clustering bar

A vector drawn from too few pixels is indistinguishable from one that was
not, so an operator whose thumbnails are smaller than their faces need -
or whose migration could not write a rendition - has nothing to read. The
audit now counts the markers whose recorded extent is below the bar, and
how many of those their original could still supply, which separates what
a re-run recovers from what no thumbnail size can.

Author: lastzero

internal/ai/face/README.md

@@ -195,7 +195,7 @@ Two caveats apply to the benchmark-derived recommendations. The measured centroi
 
   **Each detector also carries its own cutoff** (`Detector.MinScore`), because they do not score alike: SCRFD emits one calibrated sigmoid and sits at 50, while YuNet scores as `sqrt(cls x obj)` and sits at the 65 its own corpus measured. Both bars are registered on the **0-100 scale** markers and the `FACE_*` options use, and the engine converts to the 0-1 one its decoder reports, so the registry holds one scale rather than two. **Every one of them is an operator override away** - `FACE_SCORE` replaces `MinScore`, `FACE_CLUSTER_SCORE` replaces `ClusterScore` and `FACE_MIGRATE_SCORE` replaces `MigrateScore`, in either direction - so a threshold can be explored on a running instance rather than through a rebuild. Going below the knee was tried at 9/20, the pair the last stable release ran, and the demo library answered within a day: three "people" on a photograph of a bee, two of them blurred background. That evidence bounds the migration floor as well as the index one, which is why `MigrateScore` sits at 50 rather than at the 9 a migration would otherwise want: the measurement weighs false positives and not **re-detection**, so the floor takes the recall the evidence leaves available and no more. All three remain open until the preview returns attributable data points. `TestDetectorRecall` pins the measured recall at the calibrated cutoff explicitly, so a threshold decision cannot silently rewrite it. A cutoff copied from another detector is not calibration, which `TestDetectorMinScore` states.
 - `SizeThreshold` (`FACE_SIZE`, default 25 px) and `ClusterSizeThreshold` (`FACE_CLUSTER_SIZE`, default 112 px) are the size pair, and they do different jobs: the first decides whether a marker is created at all, the second whether that face may contribute to automatic clustering. **They are also in different units** - `FACE_SIZE` counts detection-thumbnail pixels, because it gates whether a box is emitted before any rendition is chosen, while `FACE_CLUSTER_SIZE` counts the pixels an embedding was sampled from. A face below the second never seeds a person even though it is detected and shown.
-- **The clustering default is `ArcFaceTemplateSize`, and it is a bar on the pixels an embedding was sampled from.** 112 is where embedding quality turns: distance to a per-person centroid averages 0.864 below it against 0.804 at or above, flat thereafter, which agrees with the template geometry. The quantity compared is `markers.thumb_size`, the face's extent in the rendition its embedding was drawn from, recorded where the vector is produced and never recomputed. A marker predating that column, or one placed by hand with no embedding, falls back to `markers.size` - detection-thumbnail pixels, which are a **lower bound** on the sampled extent, since `Fit720` is the narrowest rendition a crop is ever taken from. The fallback therefore rejects well-resolved faces it cannot prove are well-resolved and never admits an invented one. `photoprism faces migrate` treats a missing extent as stale, so a re-run measures it; `photoprism faces audit` reports how many are still without one and never writes the column, since it would have to *predict* it from a vector sampled earlier.
+- **The clustering default is `ArcFaceTemplateSize`, and it is a bar on the pixels an embedding was sampled from.** 112 is where embedding quality turns: distance to a per-person centroid averages 0.864 below it against 0.804 at or above, flat thereafter, which agrees with the template geometry. The quantity compared is `markers.thumb_size`, the face's extent in the rendition its embedding was drawn from, recorded where the vector is produced and never recomputed. A marker predating that column, or one placed by hand with no embedding, falls back to `markers.size` - detection-thumbnail pixels, which are a **lower bound** on the sampled extent, since `Fit720` is the narrowest rendition a crop is ever taken from. The fallback therefore rejects well-resolved faces it cannot prove are well-resolved and never admits an invented one. `photoprism faces migrate` treats a missing extent as stale, so a re-run measures it; `photoprism faces audit` reports how many are still without one and never writes the column, since it would have to *predict* it from a vector sampled earlier. It also counts the markers whose recorded extent is *below* the bar, and how many of those their original could still supply: a vector drawn from too few pixels is indistinguishable from one that was not, so that number exists nowhere else, and it separates what a re-run recovers from what no thumbnail size can.
 - **Both are measured in pixels of the detection thumbnail (`Fit720`), not of the original and not of the crop the embedder receives.** The crop comes from `crop.ImageFromIdealThumb`, which opens the smallest cached rendition wide enough to fill the 112 px template and falls back to the widest one cached, so a marker is compared against one image and embedded from another. A marker at exactly `FACE_CLUSTER_SIZE` needs a source no wider than `Fit720` itself, so that is what the crop path opens and the warp onto the template is 1:1 - on every aspect ratio, because the crop area is square and the required width therefore tracks the thumbnail's own. The bar is a guarantee of no upscaling rather than any headroom above it. A **smaller** marker needs a wider source, and there the aspect ratio decides how much it gets: at the default `THUMB_SIZE` of 1920 a rendition supplies 2.67x the detection size for a 16:9 original, 2.5x for a 3:2 one, 2.22x for a 4:3 one, and 1.67x for a square or 3:4 portrait one - the last two share a factor, because `Fit1920` bounds both by its 1200 px height.
 - **This is the mechanism behind the `THUMB_SIZE` warning in [Advanced Settings](https://docs.photoprism.app/user-guide/settings/advanced/#static-and-dynamic-size-limits).** Lowering the static size limit does not change any face threshold, but it lowers the rendition every crop is drawn from, so each face reaches the embedder with fewer real pixels. The size thresholds keep comparing the same numbers while the crops behind them get worse, which is why the effect is easy to miss.
 - Two detections count as the same face when their area overlap exceeds `OverlapThresholdFloor` (41 %), which is `OverlapThreshold` (42 %) relaxed by one point to absorb rounding. Tests rely on that value (e.g., `Markers.Contains/SameFace`).

internal/entity/query/faces_migrate.go

@@ -605,6 +605,43 @@ func CountMarkersWithoutThumbSize() (n int, err error) {
 	return int(count), err
 }
 
+// FaceSampleShortfall counts the markers whose vector rests on fewer pixels than the clustering
+// bar requires, and how many of those the original could still supply.
+type FaceSampleShortfall struct {
+	Measured    int
+	BelowBar    int
+	Recoverable int
+}
+
+// FaceMarkerSampleShortfall reports how many markers were embedded from too few pixels to be
+// clustered, and how many of them a re-sampling at the resolution of their original would lift
+// over the bar.
+//
+// The two numbers separate the causes an operator can act on from the one nobody can: a crop taken
+// from a rendition narrower than the original is what a migration re-samples, while a face that is
+// small in the original itself stays where it is at any thumbnail size. Only markers that recorded
+// an extent are counted, since the ones that did not are reported on their own.
+func FaceMarkerSampleShortfall(clusterSize int) (result FaceSampleShortfall, err error) {
+	if clusterSize < 1 {
+		return result, fmt.Errorf("faces: clustering size must be positive")
+	}
+
+	stmt := fmt.Sprintf(`SELECT COUNT(*) AS measured,
+		COALESCE(SUM(CASE WHEN m.thumb_size < ? THEN 1 ELSE 0 END), 0) AS below_bar,
+		COALESCE(SUM(CASE WHEN m.thumb_size < ? AND m.w * f.file_width >= ? THEN 1 ELSE 0 END), 0) AS recoverable
+		FROM %s m JOIN %s f ON f.file_uid = m.file_uid
+		WHERE m.marker_type = ? AND m.marker_invalid = 0 AND m.thumb_size >= 1 AND m.w > 0
+		AND LENGTH(m.embeddings_json) > 0
+		AND f.file_width > 0 AND f.file_missing = 0 AND f.deleted_at IS NULL`,
+		entity.Marker{}.TableName(), entity.File{}.TableName())
+
+	if err = Db().Raw(stmt, clusterSize, clusterSize, clusterSize, entity.MarkerFace).Scan(&result).Error; err != nil {
+		return FaceSampleShortfall{}, err
+	}
+
+	return result, nil
+}
+
 // SettleMigrationThumbSize records that a sampling reached these markers and produced no extent, so
 // a migration filling the column does not attempt them again on every future run.
 //

internal/entity/query/faces_migrate_test.go

@@ -867,3 +867,72 @@ func TestFaceMigrationSampleFiles(t *testing.T) {
 		require.Error(t, err)
 	})
 }
+
+// TestFaceMarkerSampleShortfall pins the two numbers an operator acts on: how many markers hold a
+// vector too small to be clustered, and how many of those an original could still supply. Measured
+// as deltas, so the shared fixtures cannot decide the outcome.
+func TestFaceMarkerSampleShortfall(t *testing.T) {
+	const clusterSize = 112
+
+	before, err := FaceMarkerSampleShortfall(clusterSize)
+	require.NoError(t, err)
+
+	file := entity.File{
+		FileUID:   rnd.GenerateUID('f'),
+		PhotoUID:  rnd.GenerateUID('p'),
+		FileName:  "sample-shortfall/large.jpg",
+		FileRoot:  entity.RootOriginals,
+		FileWidth: 4000,
+	}
+	require.NoError(t, Db().Create(&file).Error)
+	t.Cleanup(func() { Db().Unscoped().Delete(&file) })
+
+	// The extents are what each marker's embedding was drawn from, and the widths what its
+	// original could supply: 0.1 of 4000 px is 400, and 0.01 of it is 40.
+	cases := []struct {
+		w         float32
+		thumbSize int
+	}{
+		{0.1, 400}, // sampled above the bar
+		{0.1, 60},  // below it, and its original holds 400 px
+		{0.01, 40}, // below it, and its original holds 40
+	}
+
+	for _, c := range cases {
+		m := entity.Marker{
+			MarkerUID:      rnd.GenerateUID('m'),
+			FileUID:        file.FileUID,
+			MarkerType:     entity.MarkerFace,
+			W:              c.w,
+			H:              c.w,
+			ThumbSize:      c.thumbSize,
+			EmbeddingsJSON: []byte("[[0.1,0.2]]"),
+		}
+		require.NoError(t, Db().Create(&m).Error)
+		t.Cleanup(func() { Db().Unscoped().Delete(&m) })
+	}
+
+	t.Run("Counts", func(t *testing.T) {
+		after, err := FaceMarkerSampleShortfall(clusterSize)
+		require.NoError(t, err)
+
+		assert.Equal(t, before.Measured+3, after.Measured)
+		assert.Equal(t, before.BelowBar+2, after.BelowBar)
+		assert.Equal(t, before.Recoverable+1, after.Recoverable, "only one of the two has an original that could supply the bar")
+	})
+	t.Run("AtALowerBar", func(t *testing.T) {
+		// The bar is what decides both numbers, so a smaller one has to move them.
+		low, err := FaceMarkerSampleShortfall(50)
+		require.NoError(t, err)
+
+		after, err := FaceMarkerSampleShortfall(clusterSize)
+		require.NoError(t, err)
+
+		assert.Less(t, low.BelowBar, after.BelowBar)
+		assert.Equal(t, low.Measured, after.Measured)
+	})
+	t.Run("InvalidSize", func(t *testing.T) {
+		_, err := FaceMarkerSampleShortfall(0)
+		require.Error(t, err)
+	})
+}

internal/photoprism/faces_audit.go

@@ -691,6 +691,40 @@ func (w *Faces) auditProvenance() {
 	w.auditMarkerEmbeddingModels(face.EmbeddingModelName())
 	w.auditMarkerDetectModels()
 	w.auditMarkerThumbSizes()
+	w.auditMarkerSampleShortfall()
+}
+
+// auditMarkerSampleShortfall reports the markers whose vector rests on too few pixels to be
+// clustered, and how many of them the originals could still supply.
+//
+// It is the one state that leaves no trace in the vectors themselves: a marker embedded from an
+// upscaled crop is indistinguishable from one that was not, so without this an operator whose
+// thumbnail cache is smaller than their faces need, or whose migration could not write a rendition,
+// has no number to read and nothing to act on.
+func (w *Faces) auditMarkerSampleShortfall() {
+	shortfall, err := query.FaceMarkerSampleShortfall(face.ClusterSizeThreshold)
+
+	if err != nil {
+		log.Errorf("faces: %s (audit marker sample sizes)", err)
+		return
+	}
+
+	if shortfall.BelowBar == 0 {
+		log.Debugf("faces: every measured marker was sampled at or above the %d px clustering size",
+			face.ClusterSizeThreshold)
+
+		return
+	}
+
+	log.Infof("faces: %s sampled below the %d px clustering size, of %d measured",
+		english.Plural(shortfall.BelowBar, "marker", "markers"), face.ClusterSizeThreshold, shortfall.Measured)
+
+	if shortfall.Recoverable > 0 {
+		log.Infof("faces: %s of those have originals that hold enough detail, so photoprism faces migrate would sample them above it",
+			english.Plural(shortfall.Recoverable, "marker", "markers"))
+	} else {
+		log.Infof("faces: none of them have an original holding enough detail, so no thumbnail size or migration changes it")
+	}
 }
 
 // auditMarkerThumbSizes counts embedded markers with no recorded sample extent, which the size bar

internal/photoprism/faces_audit_test.go

@@ -555,3 +555,60 @@ func TestFaces_auditMarkerThumbSizes(t *testing.T) {
 	assert.Equal(t, 1, reported, "a marker the bar falls back on must be named")
 	assert.NotPanics(t, func() { (*Faces)(nil).auditMarkerThumbSizes() })
 }
+
+// TestFaces_auditMarkerSampleShortfall covers the one state that leaves no trace in the vectors: a
+// marker embedded from an upscaled crop is indistinguishable from one that was not, so the audit is
+// where an operator finds out that their thumbnails, or a migration that could not write one, cost
+// them recognition.
+func TestFaces_auditMarkerSampleShortfall(t *testing.T) {
+	w := NewFaces(config.TestConfig())
+
+	restore := face.ClusterSizeThreshold
+	t.Cleanup(func() { face.ClusterSizeThreshold = restore })
+
+	file := &entity.File{
+		FileUID:   rnd.GenerateUID('f'),
+		PhotoUID:  rnd.GenerateUID('p'),
+		FileName:  "audit-shortfall/large.jpg",
+		FileRoot:  entity.RootOriginals,
+		FileWidth: 4000,
+	}
+	require.NoError(t, entity.Db().Create(file).Error)
+	t.Cleanup(func() { entity.UnscopedDb().Delete(file) })
+
+	// Sampled at 60 px while its original holds 400, so a re-sampling clears a bar of 112.
+	marker := &entity.Marker{
+		MarkerUID:      rnd.GenerateUID('m'),
+		FileUID:        file.FileUID,
+		MarkerType:     entity.MarkerFace,
+		W:              0.1,
+		H:              0.1,
+		ThumbSize:      60,
+		EmbeddingsJSON: []byte("[[0.1,0.2]]"),
+	}
+	require.NoError(t, entity.Db().Create(marker).Error)
+	t.Cleanup(func() { entity.UnscopedDb().Delete(marker) })
+
+	t.Run("NamesWhatAMigrationWouldRecover", func(t *testing.T) {
+		face.ClusterSizeThreshold = 112
+
+		hook := captureLog(t)
+		w.auditMarkerSampleShortfall()
+
+		reported := strings.Join(loggedMessages(hook, logrus.InfoLevel), "\n")
+		assert.Contains(t, reported, "sampled below the 112 px clustering size")
+		assert.Contains(t, reported, "faces migrate")
+	})
+	t.Run("NothingToReport", func(t *testing.T) {
+		// A bar every measured marker clears is not a finding, so it stays out of the report.
+		face.ClusterSizeThreshold = 1
+
+		hook := captureLog(t)
+		w.auditMarkerSampleShortfall()
+
+		assert.Empty(t, loggedMessages(hook, logrus.InfoLevel))
+	})
+	t.Run("NilWorker", func(t *testing.T) {
+		assert.NotPanics(t, func() { (*Faces)(nil).auditMarkerSampleShortfall() })
+	})
+}