docs(benchmarks): add benchmarks to readme (#52)

* docs(benchmarks): add benchmarks to readme

* tables

* minor change in table title

* fix(docs): update hardware specifications in README

* updated tests

* cache

* nitpick

* move benchmark methodology section

* update toc

* charts

Author: s0up4200

Diff for: .gitignore

@@ -7,7 +7,6 @@ coverage.txt
 coverage.html
 test.sh
 .idea
-*.png
 *.pprof
 test_ant/*
 .pgo_profiles/

Diff for: README.md

@@ -82,6 +82,11 @@ mkbrr create path/to/file -t https://example-tracker.com/announce -e
 - [Tracker-Specific Features](#tracker-specific-features)
 - [Incomplete Season Pack Detection](#incomplete-season-pack-detection)
 - [Performance](#performance)
+  - [Benchmark Methodology](#benchmark-methodology)
+  - [Benchmark Results](#benchmark-results)
+  - [Speed Comparison](#speed-comparison)
+  - [Consistency](#consistency)
+  - [Hardware Specifications](#hardware-specifications)
 - [License](#license)
 
 ## Installation
@@ -283,11 +288,158 @@ Wrote title.torrent (elapsed 3.22s)
 
 ## Performance
 
-mkbrr is optimized for speed, and outperforms other popular tools.
-We will post some benchmarks later on.
+mkbrr is optimized for speed and consistently outperforms other popular torrent creation tools in our benchmarks.
+
+### Benchmark Methodology
+
+All tests were performed using [hyperfine](https://github.com/sharkdp/hyperfine) with 5 runs per tool after a warm-up run. Cache was cleared between runs on the servers, but not on the Macbook.
+
+### Benchmark Results
+
+| Hardware | Test Size | mkbrr | mktorrent | torrenttools | torf |
+|----------|---------------|-------|-----------|--------------|------|
+| **Leaseweb Server (SSD)** | 21 GiB | **7.24s** | 45.41s | 9.07s | 8.85s |
+| **Hetzner Server (HDD)** | 14 GiB | **41.02s** | 68.17s | 47.97s | 58.19s |
+| **Macbook Pro M4 (NVME)** | 30 GiB | **9.71s** | 10.90s | - | 9.78s |
+
+![Benchmark Comparison](docs/benchmarks/benchmark_comparison.png)
+
+### Speed Comparison
+
+| Hardware | vs mktorrent | vs torrenttools | vs torf |
+|----------|-------------|----------------|---------|
+| **Leaseweb Server (SSD)** | 6.3× faster | 1.3× faster | 1.2× faster |
+| **Hetzner Server (HDD)** | 1.7× faster | 1.2× faster | 1.4× faster |
+| **Macbook Pro M4 (NVME)** | 1.1× faster | - | Similar |
+
+![Speed Comparison](docs/benchmarks/speed_comparison.png)
+
+### Consistency
+
+Besides raw speed, mkbrr shows more consistent performance between runs, with standard deviation percentages between 0.25-3.7% across platforms compared to much higher variances for other tools (up to 39%). This predictable performance is particularly noticeable on mechanical storage where other tools showed wider fluctuations.
+
+![Consistency Comparison](docs/benchmarks/consistency_comparison.png)
+
+### Hardware Specifications
+
+#### Leaseweb Dedicated Server (SSD)
+- CPU: Intel Xeon E-2274G @ 4.00GHz
+- RAM: 32GB
+- Storage: 1 × SAMSUNG MZQL21T9HCJR-00A07 1.92TB SSD
+
+#### Hetzner Dedicated Server (HDD)
+- CPU: AMD Ryzen 5 3600 (12) @ 4.71GHz
+- RAM: 64GB
+- Storage: 4 × TOSHIBA MG08ACA16TEY in RAID0
+
+#### Macbook Pro M4
+- CPU: Apple M4
+- RAM: 16GB
+- Storage: 512GB NVME
+
+### Full results
+
+<details>
+<summary>View Full Benchmark Commands & Results</summary>
+
+#### Leaseweb Server (21 GiB 1080p season pack)
+
+```bash
+hyperfine --warmup 1 --runs 5 \
+  --setup 'sudo sync && sudo sh -c "echo 3 > /proc/sys/vm/drop_caches"' \
+  --prepare 'sudo sync && sudo sh -c "echo 3 > /proc/sys/vm/drop_caches" && rm -f /home/user/Show.S01.DL.1080p.WEB.H264-GROUP.torrent' \
+  'mkbrr create /home/user/torrents/qbittorrent/Show.S01.DL.1080p.WEB.H264-GROUP' \
+  'mktorrent /home/user/torrents/qbittorrent/Show.S01.DL.1080p.WEB.H264-GROUP' \
+  'torrenttools create --threads 8 ~/torrents/qbittorrent/Show.S01.DL.1080p.WEB.H264-GROUP' \
+  'torf --threads 8 /home/user/torrents/qbittorrent/Show.S01.DL.1080p.WEB.H264-GROUP'
+
+Benchmark 1: mkbrr create /home/user/torrents/qbittorrent/Show.S01.DL.1080p.WEB.H264-GROUP
+  Time (mean ± σ):      7.244 s ±  0.018 s    [User: 31.245 s, System: 7.554 s]
+  Range (min … max):    7.225 s …  7.270 s    5 runs
+
+Benchmark 2: mktorrent /home/user/torrents/qbittorrent/Show.S01.DL.1080p.WEB.H264-GROUP
+  Time (mean ± σ):     45.407 s ±  0.163 s    [User: 36.495 s, System: 4.983 s]
+  Range (min … max):   45.135 s … 45.539 s    5 runs
+
+Benchmark 3: torrenttools create --threads 8 ~/torrents/qbittorrent/Show.S01.DL.1080p.WEB.H264-GROUP
+  Time (mean ± σ):      9.074 s ±  0.093 s    [User: 29.248 s, System: 5.228 s]
+  Range (min … max):    8.908 s …  9.122 s    5 runs
+
+Benchmark 4: torf --threads 8 /home/user/torrents/qbittorrent/Show.S01.DL.1080p.WEB.H264-GROUP
+  Time (mean ± σ):      8.854 s ±  0.077 s    [User: 25.829 s, System: 5.136 s]
+  Range (min … max):    8.771 s …  8.937 s    5 runs
+
+Summary
+  'mkbrr create /home/user/torrents/qbittorrent/Show.S01.DL.1080p.WEB.H264-GROUP' ran
+    1.22 ± 0.01 times faster than 'torf --threads 8 /home/user/torrents/qbittorrent/Show.S01.DL.1080p.WEB.H264-GROUP'
+    1.25 ± 0.01 times faster than 'torrenttools create --threads 8 ~/torrents/qbittorrent/Show.S01.DL.1080p.WEB.H264-GROUP'
+    6.27 ± 0.03 times faster than 'mktorrent /home/user/torrents/qbittorrent/Show.S01.DL.1080p.WEB.H264-GROUP'
+```
+
+#### Hetzner Server (14 GiB 1080p season pack)
+
+```bash
+hyperfine --warmup 1 --runs 5 \
+  --setup 'sudo sync && sudo sh -c "echo 3 > /proc/sys/vm/drop_caches"' \
+  --prepare 'sudo sync && sudo sh -c "echo 3 > /proc/sys/vm/drop_caches" && rm -f /home/user/mkbrr/Show.S01.1080p.SRC.WEB-DL.DDP5.1.H.264-GRP.torrent' \
+  'mkbrr create ~/torrents/qbittorrent/tv/Show.S01.1080p.SRC.WEB-DL.DDP5.1.H.264-GRP' \
+  'mktorrent ~/torrents/qbittorrent/tv/Show.S01.1080p.SRC.WEB-DL.DDP5.1.H.264-GRP' \
+  'torrenttools create --threads 12 ~/torrents/qbittorrent/tv/Show.S01.1080p.SRC.WEB-DL.DDP5.1.H.264-GRP' \
+  'torf --threads 12 ~/torrents/qbittorrent/tv/Show.S01.1080p.SRC.WEB-DL.DDP5.1.H.264-GRP'
+
+Benchmark 1: mkbrr create ~/torrents/qbittorrent/tv/Show.S01.1080p.SRC.WEB-DL.DDP5.1.H.264-GRP
+  Time (mean ± σ):     41.022 s ±  0.979 s    [User: 13.691 s, System: 6.747 s]
+  Range (min … max):   39.938 s … 42.467 s    5 runs
+
+Benchmark 2: mktorrent ~/torrents/qbittorrent/tv/Show.S01.1080p.SRC.WEB-DL.DDP5.1.H.264-GRP
+  Time (mean ± σ):     68.168 s ± 26.654 s    [User: 17.934 s, System: 6.543 s]
+  Range (min … max):   39.268 s … 97.574 s    5 runs
+
+Benchmark 3: torrenttools create --threads 12 ~/torrents/qbittorrent/tv/Show.S01.1080p.SRC.WEB-DL.DDP5.1.H.264-GRP
+  Time (mean ± σ):     47.968 s ± 10.552 s    [User: 7.052 s, System: 6.551 s]
+  Range (min … max):   39.460 s … 66.296 s    5 runs
+
+Benchmark 4: torf --threads 12 ~/torrents/qbittorrent/tv/Show.S01.1080p.SRC.WEB-DL.DDP5.1.H.264-GRP
+  Time (mean ± σ):     58.187 s ±  5.787 s    [User: 7.185 s, System: 6.511 s]
+  Range (min … max):   50.125 s … 65.807 s    5 runs
+
+Summary
+  mkbrr create ~/torrents/qbittorrent/tv/Show.S01.1080p.SRC.WEB-DL.DDP5.1.H.264-GRP ran
+    1.17 ± 0.26 times faster than torrenttools create --threads 12 ~/torrents/qbittorrent/tv/Show.S01.1080p.SRC.WEB-DL.DDP5.1.H.264-GRP
+    1.42 ± 0.15 times faster than torf --threads 12 ~/torrents/qbittorrent/tv/Show.S01.1080p.SRC.WEB-DL.DDP5.1.H.264-GRP
+    1.66 ± 0.65 times faster than mktorrent ~/torrents/qbittorrent/tv/Show.S01.1080p.SRC.WEB-DL.DDP5.1.H.264-GRP
+```
+
+#### Macbook Pro M4 (30 GiB 1080p season pack)
+
+```bash
+hyperfine --warmup 1 --runs 5 \
+  --prepare 'rm -f Show.S01.1080p.SRC.WEB-DL.DDP5.1.H.264-GRP.torrent' \
+  'mkbrr create ~/Desktop/Show.S01.1080p.SRC.WEB-DL.DDP5.1.H.264-GRP' \
+  'mktorrent ~/Desktop/Show.S01.1080p.SRC.WEB-DL.DDP5.1.H.264-GRP' \
+  'torf --threads 10 ~/Desktop/Show.S01.1080p.SRC.WEB-DL.DDP5.1.H.264-GRP'
+
+Benchmark 1: mkbrr create ~/Desktop/Show.S01.1080p.SRC.WEB-DL.DDP5.1.H.264-GRP
+  Time (mean ± σ):      9.708 s ±  0.355 s    [User: 10.823 s, System: 4.297 s]
+  Range (min … max):    9.479 s … 10.323 s    5 runs
+
+Benchmark 2: mktorrent ~/Desktop/Show.S01.1080p.SRC.WEB-DL.DDP5.1.H.264-GRP
+  Time (mean ± σ):     10.897 s ±  0.701 s    [User: 11.021 s, System: 3.038 s]
+  Range (min … max):    9.950 s … 11.620 s    5 runs
+
+Benchmark 3: torf --threads 10 ~/Desktop/Show.S01.1080p.SRC.WEB-DL.DDP5.1.H.264-GRP
+  Time (mean ± σ):      9.779 s ±  0.749 s    [User: 10.776 s, System: 5.253 s]
+  Range (min … max):    9.383 s … 11.112 s    5 runs
+
+Summary
+  mkbrr create ~/Desktop/Show.S01.1080p.SRC.WEB-DL.DDP5.1.H.264-GRP ran
+    1.01 ± 0.09 times faster than torf --threads 10 ~/Desktop/Show.S01.1080p.SRC.WEB-DL.DDP5.1.H.264-GRP
+    1.12 ± 0.08 times faster than mktorrent ~/Desktop/Show.S01.1080p.SRC.WEB-DL.DDP5.1.H.264-GRP
+```
+</details>
 
 ## License
 
 This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.
 
-See [LICENSE](LICENSE) for the full license text.
+See [LICENSE](LICENSE) for the full license text.

Diff for: docs/benchmarks/benchmark_comparison.png

@@ -0,0 +1,115 @@
+import matplotlib.pyplot as plt
+import numpy as np
+import pandas as pd
+import seaborn as sns
+
+# Set style
+plt.style.use('bmh')
+sns.set_theme(style="whitegrid")
+
+# Data
+hardware = ['Leaseweb (SSD)', 'Hetzner (HDD)', 'Macbook (NVME)']
+test_size = [21, 14, 30]  # GiB
+
+# Times in seconds
+mkbrr = [7.24, 41.02, 9.71]
+mktorrent = [45.41, 68.17, 10.90]
+torrenttools = [9.07, 47.97, np.nan]
+torf = [8.85, 58.19, 9.78]
+
+# Create DataFrame
+df = pd.DataFrame({
+    'Hardware': hardware,
+    'Test Size (GiB)': test_size,
+    'mkbrr': mkbrr,
+    'mktorrent': mktorrent,
+    'torrenttools': torrenttools,
+    'torf': torf
+})
+
+plt.rcParams['figure.figsize'] = (10, 6)
+plt.rcParams['font.size'] = 10
+
+# Create horizontal bar plot
+fig, ax = plt.subplots()
+y = np.arange(len(hardware))
+width = 0.2
+
+ax.barh(y - width*1.5, mkbrr, width, label='mkbrr', color='#2ecc71')
+ax.barh(y - width/2, mktorrent, width, label='mktorrent', color='#e74c3c')
+ax.barh(y + width/2, torrenttools, width, label='torrenttools', color='#3498db')
+ax.barh(y + width*1.5, torf, width, label='torf', color='#f1c40f')
+
+# Customize plot
+ax.set_xlabel('Time (seconds)')
+ax.set_title('Torrent Creation Performance Comparison')
+ax.set_yticks(y)
+ax.set_yticklabels(hardware)
+ax.legend(bbox_to_anchor=(1.02, 1), loc='upper left')
+
+# Add test size labels
+for i, size in enumerate(test_size):
+    ax.text(1, i, f'{size} GiB', ha='left', va='center')
+
+# Adjust layout
+plt.tight_layout()
+
+# Save plot
+plt.savefig('benchmark_comparison.png', dpi=300, bbox_inches='tight')
+
+# Create speed comparison plot
+speed_vs_mktorrent = [6.3, 1.7, 1.1]
+speed_vs_torrenttools = [1.3, 1.2, np.nan]
+speed_vs_torf = [1.2, 1.4, 1.0]
+
+fig, ax = plt.subplots()
+y = np.arange(len(hardware))
+width = 0.25
+
+ax.barh(y - width, speed_vs_mktorrent, width, label='vs mktorrent', color='#e74c3c')
+ax.barh(y, speed_vs_torrenttools, width, label='vs torrenttools', color='#3498db')
+ax.barh(y + width, speed_vs_torf, width, label='vs torf', color='#f1c40f')
+
+# Customize plot
+ax.set_xlabel('Speed Multiplier (×)')
+ax.set_title('mkbrr Speed Comparison')
+ax.set_yticks(y)
+ax.set_yticklabels(hardware)
+ax.legend(bbox_to_anchor=(1.02, 1), loc='upper left')
+
+# Add vertical line at 1.0
+ax.axvline(x=1.0, color='black', linestyle='--', alpha=0.3)
+
+# Adjust layout
+plt.tight_layout()
+
+# Save plot
+plt.savefig('speed_comparison.png', dpi=300, bbox_inches='tight')
+
+# Create consistency plot (standard deviation percentages)
+std_mkbrr = [0.25, 2.39, 3.66]
+std_mktorrent = [0.36, 39.10, 6.43]
+std_torrenttools = [1.02, 22.00, np.nan]
+std_torf = [0.87, 9.95, 7.66]
+
+fig, ax = plt.subplots()
+y = np.arange(len(hardware))
+width = 0.2
+
+ax.barh(y - width*1.5, std_mkbrr, width, label='mkbrr', color='#2ecc71')
+ax.barh(y - width/2, std_mktorrent, width, label='mktorrent', color='#e74c3c')
+ax.barh(y + width/2, std_torrenttools, width, label='torrenttools', color='#3498db')
+ax.barh(y + width*1.5, std_torf, width, label='torf', color='#f1c40f')
+
+# Customize plot
+ax.set_xlabel('Standard Deviation (%)')
+ax.set_title('Performance Consistency Comparison')
+ax.set_yticks(y)
+ax.set_yticklabels(hardware)
+ax.legend(bbox_to_anchor=(1.02, 1), loc='upper left')
+
+# Adjust layout
+plt.tight_layout()
+
+# Save plot
+plt.savefig('consistency_comparison.png', dpi=300, bbox_inches='tight') 

Diff for: docs/benchmarks/benchmark_plots.py

@@ -0,0 +1,4 @@
+matplotlib>=3.7.0
+numpy>=1.24.0
+pandas>=2.0.0
+seaborn>=0.12.0