Skip to content

Commit 15c7db5

Browse files
ternauscursoragent
ternaus
and
cursoragent
committed
feat(multichannel): add multi-channel benchmark support
- Add --num-channels and --multichannel CLI flags to runner and cli - Add multichannel transform specs for albumentationsx, kornia, torchvision - Add multichannel results output (9ch benchmark data) - Extend update_readme.py and update_docs.sh to include multichannel table - Remove stale docs/images and docs/videos pre-generated artifacts - Update README with multichannel comparison table Co-authored-by: Cursor <cursoragent@cursor.com>
1 parent f2bfb89 commit 15c7db5

20 files changed

+3028
-176
lines changed

‎README.md‎

Lines changed: 46 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -46,10 +46,6 @@ This benchmark suite measures the throughput and performance characteristics of
4646

4747
The image benchmarks compare the performance of various libraries on standard image transformations. All benchmarks are run on a single CPU thread to ensure consistent and comparable results.
4848

49-
[**Detailed Image Benchmark Results**](docs/images/README.md)
50-
51-
![Image Speedup Analysis](docs/images/images_speedup_analysis.webp)
52-
5349
<!-- IMAGE_BENCHMARK_TABLE_START -->
5450
| Transform | albumentationsx 2.0.18 [img/s] | kornia 0.8.2 [img/s] | torchvision 0.25.0 [img/s] | Speedup (albx/fastest other) |
5551
|:---------------------|:---------------------------------|:-----------------------|:-----------------------------|:-------------------------------|
@@ -105,13 +101,55 @@ The image benchmarks compare the performance of various libraries on standard im
105101
| VerticalFlip | **27128 ± 1197** | 2387 ± 58 | 26928 ± 4799 | 1.01x |
106102
<!-- IMAGE_BENCHMARK_TABLE_END -->
107103

108-
### Video Benchmarks
104+
### Multi-Channel Image Benchmarks (9ch)
109105

110-
The video benchmarks compare CPU-based processing (AlbumentationsX) with GPU-accelerated processing (Kornia) for video transformations. The benchmarks use the [UCF101 dataset](https://www.crcv.ucf.edu/data/UCF101.php), which contains realistic videos from 101 action categories.
106+
Benchmarks on 9-channel images (3x stacked RGB) to test OpenCV chunking and library support for >4 channels.
107+
108+
<!-- MULTICHANNEL_BENCHMARK_TABLE_START -->
109+
| Transform | albumentationsx 2.0.18 [img/s] | kornia 0.8.2 [img/s] | torchvision 0.25.0 [img/s] | Speedup (albx/fastest other) |
110+
|:---------------------|:---------------------------------|:-----------------------|:-----------------------------|:-------------------------------|
111+
| Affine | **640 ± 6** | 228 ± 3 | 143 ± 3 | 2.81x |
112+
| AutoContrast | **428 ± 8** | 374 ± 3 | - | 1.14x |
113+
| Blur | **2418 ± 154** | 186 ± 3 | - | 12.97x |
114+
| Brightness | **3650 ± 162** | 1350 ± 40 | - | 2.70x |
115+
| CenterCrop128 | 51247 ± 435 | - | **223574 ± 5049** | 0.23x |
116+
| ChannelDropout | **6387 ± 176** | 2179 ± 95 | - | 2.93x |
117+
| ChannelShuffle | **2427 ± 164** | 929 ± 25 | 1600 ± 41 | 1.52x |
118+
| Contrast | **3837 ± 368** | 1346 ± 31 | - | 2.85x |
119+
| CornerIllumination | **203 ± 2** | 181 ± 3 | - | 1.12x |
120+
| Erasing | **9956 ± 359** | 426 ± 10 | 4321 ± 384 | 2.30x |
121+
| GaussianBlur | **746 ± 9** | 188 ± 2 | 49 ± 6 | 3.97x |
122+
| GaussianIllumination | **240 ± 2** | 212 ± 15 | - | 1.13x |
123+
| GaussianNoise | **95 ± 3** | 65 ± 0 | - | 1.45x |
124+
| HorizontalFlip | 2531 ± 234 | 2286 ± 557 | **15102 ± 3640** | 0.17x |
125+
| Invert | 10141 ± 2159 | 2774 ± 169 | **15806 ± 3070** | 0.64x |
126+
| LinearIllumination | 140 ± 1 | **491 ± 12** | - | 0.29x |
127+
| LongestMaxSize | 350 ± 1 | **376 ± 2** | - | 0.93x |
128+
| MotionBlur | **1443 ± 56** | 63 ± 1 | - | 22.95x |
129+
| Normalize | 387 ± 5 | **1402 ± 64** | 795 ± 22 | 0.28x |
130+
| OpticalDistortion | **457 ± 8** | 157 ± 4 | - | 2.91x |
131+
| Pad | 8358 ± 319 | - | **9112 ± 704** | 0.92x |
132+
| Perspective | **579 ± 2** | 149 ± 1 | 129 ± 2 | 3.90x |
133+
| PlasmaBrightness | **83 ± 1** | 24 ± 1 | - | 3.44x |
134+
| PlasmaContrast | **68 ± 0** | 24 ± 1 | - | 2.82x |
135+
| PlasmaShadow | 125 ± 1 | **224 ± 2** | - | 0.56x |
136+
| Posterize | 4375 ± 187 | 317 ± 16 | **12018 ± 1989** | 0.36x |
137+
| RandomCrop128 | 49313 ± 715 | 2566 ± 75 | **124539 ± 2345** | 0.40x |
138+
| RandomGamma | **4383 ± 62** | 83 ± 0 | - | 53.12x |
139+
| RandomResizedCrop | **369 ± 12** | 309 ± 2 | 297 ± 3 | 1.19x |
140+
| Resize | 287 ± 4 | **297 ± 3** | 194 ± 1 | 0.97x |
141+
| Rotate | **1713 ± 27** | 172 ± 1 | 152 ± 10 | 9.96x |
142+
| Sharpen | **713 ± 9** | 140 ± 6 | - | 5.09x |
143+
| Shear | **642 ± 11** | 250 ± 2 | 163 ± 6 | 2.57x |
144+
| SmallestMaxSize | **253 ± 4** | 187 ± 3 | - | 1.35x |
145+
| Solarize | **4220 ± 26** | 339 ± 4 | 456 ± 11 | 9.26x |
146+
| ThinPlateSpline | **73 ± 2** | 62 ± 0 | - | 1.17x |
147+
| VerticalFlip | 8698 ± 122 | 2296 ± 118 | **15409 ± 890** | 0.56x |
148+
<!-- MULTICHANNEL_BENCHMARK_TABLE_END -->
111149

112-
[**Detailed Video Benchmark Results**](docs/videos/README.md)
150+
### Video Benchmarks
113151

114-
![Video Speedup Analysis](docs/videos/videos_speedup_analysis.webp)
152+
The video benchmarks compare CPU-based processing (AlbumentationsX) with GPU-accelerated processing (Kornia) for video transformations. The benchmarks use the [UCF101 dataset](https://www.crcv.ucf.edu/data/UCF101.php), which contains realistic videos from 101 action categories.
115153

116154
<!-- VIDEO_BENCHMARK_TABLE_START -->
117155
| Transform | albumentationsx (video) 2.0.18 [vid/s] | kornia (video) 0.8.0 [vid/s] | torchvision (video) 0.21.0 [vid/s] | Speedup (albx/fastest other) |
@@ -310,10 +348,6 @@ The benchmark methodology is designed to ensure fair and reproducible comparison
310348
3. **Measurement Phase**: Multiple runs with statistical analysis
311349
4. **Environment Control**: Consistent thread settings and hardware utilization
312350

313-
For detailed methodology, see the specific benchmark READMEs:
314-
- [Image Benchmark Methodology](docs/images/README.md#methodology)
315-
- [Video Benchmark Methodology](docs/videos/README.md#methodology)
316-
317351
## Contributing
318352

319353
Contributions are welcome! If you'd like to add support for a new library, improve the benchmarking methodology, or fix issues, please submit a pull request.

‎benchmark/cli.py‎

Lines changed: 38 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -39,6 +39,12 @@
3939
"kornia": "benchmark/transforms/kornia_impl.py",
4040
}
4141

42+
_MULTICHANNEL_IMAGE_SPECS: dict[str, str] = {
43+
"albumentationsx": "benchmark/transforms/albumentationsx_multichannel_impl.py",
44+
"torchvision": "benchmark/transforms/torchvision_multichannel_impl.py",
45+
"kornia": "benchmark/transforms/kornia_multichannel_impl.py",
46+
}
47+
4248
_VIDEO_SPECS: dict[str, str] = {
4349
"albumentationsx": "benchmark/transforms/albumentationsx_video_impl.py",
4450
"torchvision": "benchmark/transforms/torchvision_video_impl.py",
@@ -125,6 +131,7 @@ def _run_single(
125131
repo_root: Path,
126132
transforms_filter: list[str] | None = None,
127133
verbose: bool = False,
134+
num_channels: int = 3,
128135
) -> None:
129136
python = _ensure_venv(library, media, repo_root)
130137

@@ -153,6 +160,8 @@ def _run_single(
153160
cmd += ["--num-items", str(num_items)]
154161
if max_warmup is not None:
155162
cmd += ["--max-warmup", str(max_warmup)]
163+
if num_channels != 3:
164+
cmd += ["--num-channels", str(num_channels)]
156165

157166
import os
158167

@@ -225,6 +234,12 @@ def cmd_run(args: argparse.Namespace) -> None:
225234
output_dir = Path(args.output)
226235
output_dir.mkdir(parents=True, exist_ok=True)
227236

237+
# --multichannel: use 9ch specs, output to output/multichannel/
238+
if getattr(args, "multichannel", False) and media == "image":
239+
output_dir = output_dir / "multichannel"
240+
output_dir.mkdir(parents=True, exist_ok=True)
241+
args.num_channels = 9
242+
228243
# ------------------------------------------------------------------
229244
# Cloud path: delegate the whole run to a GCP instance
230245
# ------------------------------------------------------------------
@@ -252,11 +267,15 @@ def cmd_run(args: argparse.Namespace) -> None:
252267
repo_root=repo_root,
253268
transforms_filter=args.transforms,
254269
verbose=args.verbose,
270+
num_channels=args.num_channels,
255271
)
256272
return
257273

258274
# Built-in libraries
259-
spec_map = _VIDEO_SPECS if media == "video" else _IMAGE_SPECS
275+
if getattr(args, "multichannel", False) and media == "image":
276+
spec_map = _MULTICHANNEL_IMAGE_SPECS
277+
else:
278+
spec_map = _VIDEO_SPECS if media == "video" else _IMAGE_SPECS
260279
available = list(spec_map.keys())
261280

262281
requested: list[str] = args.libraries or available
@@ -285,6 +304,7 @@ def cmd_run(args: argparse.Namespace) -> None:
285304
repo_root=repo_root,
286305
transforms_filter=args.transforms,
287306
verbose=args.verbose,
307+
num_channels=args.num_channels,
288308
)
289309

290310
logger.info("All benchmarks complete. Results in: %s", output_dir)
@@ -381,6 +401,23 @@ def build_parser() -> argparse.ArgumentParser:
381401
run_p.add_argument("--warmup-window", type=int, default=5)
382402
run_p.add_argument("--warmup-threshold", type=float, default=0.05)
383403
run_p.add_argument("--min-warmup-windows", type=int, default=3)
404+
run_p.add_argument(
405+
"--num-channels",
406+
type=int,
407+
default=3,
408+
help=(
409+
"Number of image channels (must be multiple of 3). Values > 3 stack the RGB source image "
410+
"to synthesize multi-channel data, e.g. 9 for 3x stacked RGB (default: 3)"
411+
),
412+
)
413+
run_p.add_argument(
414+
"--multichannel",
415+
action="store_true",
416+
help=(
417+
"Use multi-channel specs (9ch) and output to <output>/multichannel/. "
418+
"Implies --num-channels 9 for image mode."
419+
),
420+
)
384421

385422
# ------------------------------------------------------------------
386423
# compare

‎benchmark/runner.py‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,7 @@
1818
get_library_versions,
1919
get_system_info,
2020
get_video_loader,
21+
make_multichannel_loader,
2122
time_transform,
2223
verify_thread_settings,
2324
)
@@ -81,6 +82,7 @@ def __init__(
8182
warmup_window: int = 5,
8283
warmup_threshold: float = 0.05,
8384
min_warmup_windows: int = 3,
85+
num_channels: int = 3,
8486
):
8587
self.library = library
8688
self.data_dir = Path(data_dir)
@@ -106,8 +108,11 @@ def __init__(
106108

107109
if media_type == MediaType.IMAGE:
108110
self._loader = get_image_loader(library)
111+
if num_channels != 3:
112+
self._loader = make_multichannel_loader(self._loader, num_channels)
109113
else:
110114
self._loader = get_video_loader(library)
115+
self.num_channels = num_channels
111116

112117
# ------------------------------------------------------------------
113118
# Media loading
@@ -131,6 +136,8 @@ def _load_images(self) -> list[Any]:
131136
img_check = cv2.imread(str(path), cv2.IMREAD_UNCHANGED)
132137
if img_check is None:
133138
continue
139+
# Check the on-disk image (always 3-channel RGB); the loader may
140+
# later stack channels to produce num_channels > 3 in memory.
134141
if img_check.ndim < 3 or img_check.shape[2] < 3:
135142
continue
136143

@@ -400,6 +407,7 @@ def run(self, output_path: Path | None = None) -> dict[str, Any]:
400407
"warmup_window": self.warmup_window,
401408
"warmup_threshold": self.warmup_threshold,
402409
"min_warmup_windows": self.min_warmup_windows,
410+
"num_channels": self.num_channels,
403411
},
404412
}
405413

@@ -515,6 +523,15 @@ def main() -> None:
515523
parser.add_argument("--warmup-window", type=int, default=5, help="Window size for variance check")
516524
parser.add_argument("--warmup-threshold", type=float, default=0.05, help="Variance stability threshold")
517525
parser.add_argument("--min-warmup-windows", type=int, default=3, help="Minimum windows to check")
526+
parser.add_argument(
527+
"--num-channels",
528+
type=int,
529+
default=3,
530+
help=(
531+
"Number of image channels. Must be a multiple of 3. "
532+
"Values > 3 stack the RGB image to synthesize multi-channel data (default: 3)"
533+
),
534+
)
518535

519536
args = parser.parse_args()
520537

@@ -547,6 +564,7 @@ def main() -> None:
547564
warmup_window=args.warmup_window,
548565
warmup_threshold=args.warmup_threshold,
549566
min_warmup_windows=args.min_warmup_windows,
567+
num_channels=args.num_channels,
550568
)
551569

552570
runner.run(args.output)

0 commit comments

Comments
 (0)