bench: record full knapsack comparison (#66)

Author: bernalde

benchmarks/knapsack/README.md

@@ -119,3 +119,9 @@ drives constrained DMRG cost. Final variance is calculated independently from
 the per-sweep MPS supplied by `on_iteration`. Truncation error remains empty:
 the callback runs after the DMRG sweep has discarded singular values, so that
 error cannot be reconstructed from the retained MPS.
+
+## Recorded results
+
+The repository includes a pinned complete run with its raw CSV, exact command,
+machine and package metadata, quality-aware interpretation, and limitations in
+[`results/README.md`](results/README.md).

benchmarks/knapsack/results/2026-07-22-656491d.csv

@@ -0,0 +1,26 @@
+benchmark_id,julia_version,julia_threads,system,architecture,tensolver_version,itensors_version,itensormps_version,benchmarktools_version,instance,method,nitems,nvariables,capacity,penalty_factor,penalty,exact_value,original_value,feasible,optimality_gap,penalized_objective,solver_reported_objective,solver_seed,requested_iterations,reads,cutoff,time_limit_seconds,timing_samples,formulation_wall_seconds,formulation_gc_seconds,formulation_allocated_bytes,formulation_allocations,solve_wall_seconds,solve_gc_seconds,solve_allocated_bytes,solve_allocations,observer_setup_seconds,observer_setup_gc_seconds,observer_setup_allocated_bytes,sampling_seconds,sampling_gc_seconds,sampling_allocated_bytes,solver_excluding_observer_setup_seconds,end_to_end_wall_seconds,solver_elapsed_seconds,sweeps,time_limit_reached,solution_max_bond,final_variance,truncation_error,objective_mpo_bond,projection_mpo_bond,effective_hamiltonian_bond
+"656491d6daeaaa17ec6cd22b32014a7b5d376374","1.12.6",1,"Linux","x86_64","0.2.3","0.9.30","0.4.1","1.8.0","reference_4","projection",4,4,6,,,13.0,13.0,true,0.0,,13.0,66,6,64,1.0e-10,120.0,3,1.9133e-5,0.0,1216,21,0.057857043,0.032191888,35879920,172431,0.003065788,0.0,4576728,0.002100968,0.0,3812928,0.052690286999999995,0.057876175999999994,0.05497908592224121,6,false,1,0.0,,2,8,3
+"656491d6daeaaa17ec6cd22b32014a7b5d376374","1.12.6",1,"Linux","x86_64","0.2.3","0.9.30","0.4.1","1.8.0","reference_4","penalty_qubo",4,7,6,0.001,0.02,13.0,20.0,false,,-19.28,-19.28,66,6,64,1.0e-10,120.0,3,7.82e-7,0.0,1760,19,0.046468231,0.012489926,55463832,289262,0.000946418,0.0,1378208,0.00396306,0.0,7461632,0.041558753000000004,0.046469012999999997,0.040389060974121094,6,false,9,0.0,,3,,3
+"656491d6daeaaa17ec6cd22b32014a7b5d376374","1.12.6",1,"Linux","x86_64","0.2.3","0.9.30","0.4.1","1.8.0","reference_4","penalty_qubo",4,7,6,0.01,0.2,13.0,17.0,false,,-15.200000000000006,-15.200000000000006,66,6,64,1.0e-10,120.0,3,6.81e-7,0.0,1760,19,0.04681526,0.012379955,53227720,276586,0.001050748,0.0,1376120,0.003878464,0.0,7128832,0.041886048,0.046815941,0.04068899154663086,6,false,9,5.684341886080802e-14,,3,,3
+"656491d6daeaaa17ec6cd22b32014a7b5d376374","1.12.6",1,"Linux","x86_64","0.2.3","0.9.30","0.4.1","1.8.0","reference_4","penalty_qubo",4,7,6,0.1,2.0,13.0,13.0,true,0.0,-13.0,-13.0,66,6,64,1.0e-10,120.0,3,7.44e-7,0.0,1760,19,0.051412114,0.007099745,64894304,324976,0.001083601,0.0,1376120,0.003871539,0.0,6797056,0.046456974,0.051412858,0.03812813758850098,6,false,11,9.094947017729282e-13,,3,,3
+"656491d6daeaaa17ec6cd22b32014a7b5d376374","1.12.6",1,"Linux","x86_64","0.2.3","0.9.30","0.4.1","1.8.0","reference_4","penalty_qubo",4,7,6,1.1,22.0,13.0,9.0,true,4.0,-9.0,-8.0,66,6,64,1.0e-10,120.0,3,8.48e-7,0.0,1760,19,0.061084115,0.011055909,76368616,380811,0.001004124,0.0,1378208,0.003958885,0.0,7191008,0.056121106,0.061084963,0.054817914962768555,6,false,13,1.088863940211013,,3,,3
+"656491d6daeaaa17ec6cd22b32014a7b5d376374","1.12.6",1,"Linux","x86_64","0.2.3","0.9.30","0.4.1","1.8.0","pisinger_uncorrelated_n8","projection",8,8,26,,,41.0,41.0,true,0.0,,41.0,66,6,64,1.0e-10,120.0,3,1.9966e-5,0.0,1440,21,0.103061237,0.020320391,143845848,437563,0.01286354,0.0,21048840,0.004736976,0.0,8684096,0.08546072099999999,0.103081203,0.0968170166015625,6,false,1,0.0,,2,28,13
+"656491d6daeaaa17ec6cd22b32014a7b5d376374","1.12.6",1,"Linux","x86_64","0.2.3","0.9.30","0.4.1","1.8.0","pisinger_uncorrelated_n8","penalty_qubo",8,13,26,0.001,0.06,41.0,49.0,false,,-45.16000000000002,-45.16000000000002,66,6,64,1.0e-10,120.0,3,1.128e-6,0.0,4192,20,0.095275157,0.014115676,138559784,655962,0.003017734,0.0,4739544,0.007523439,0.0,14094144,0.08473398400000001,0.095276285,0.08269691467285156,6,false,13,0.0,,3,,3
+"656491d6daeaaa17ec6cd22b32014a7b5d376374","1.12.6",1,"Linux","x86_64","0.2.3","0.9.30","0.4.1","1.8.0","pisinger_uncorrelated_n8","penalty_qubo",8,13,26,0.01,0.6,41.0,37.0,true,4.0,-37.0,-37.0,66,6,64,1.0e-10,120.0,3,1.505e-6,0.0,4192,20,0.140315042,0.020107159,216960128,810010,0.003124302,0.0,4723120,0.007712437,0.0,14132752,0.12947830300000002,0.140316547,0.12703299522399902,6,false,20,0.18416690808953717,,3,,3
+"656491d6daeaaa17ec6cd22b32014a7b5d376374","1.12.6",1,"Linux","x86_64","0.2.3","0.9.30","0.4.1","1.8.0","pisinger_uncorrelated_n8","penalty_qubo",8,13,26,0.1,6.0,41.0,22.0,true,19.0,-22.0,-22.0,66,6,64,1.0e-10,120.0,3,1.173e-6,0.0,4192,20,0.321576771,0.031372064,246899760,811177,0.008691879,0.0,4739544,0.008121815,0.0,14231824,0.304763077,0.321577944,0.2825629711151123,6,false,27,6.925516746938229,,3,,3
+"656491d6daeaaa17ec6cd22b32014a7b5d376374","1.12.6",1,"Linux","x86_64","0.2.3","0.9.30","0.4.1","1.8.0","pisinger_uncorrelated_n8","penalty_qubo",8,13,26,1.1,66.0,41.0,22.0,true,19.0,-22.0,-22.0,66,6,64,1.0e-10,120.0,3,1.573e-6,0.0,4192,20,0.222079603,0.022157553,258166960,809070,0.004963164,0.0,4723120,0.008704297,0.0,14044032,0.208412142,0.222081176,0.20019221305847168,6,false,31,1400.65363073349,,3,,3
+"656491d6daeaaa17ec6cd22b32014a7b5d376374","1.12.6",1,"Linux","x86_64","0.2.3","0.9.30","0.4.1","1.8.0","pisinger_weakly_correlated_n12","projection",12,12,30,,,36.0,36.0,true,0.0,,36.0,66,6,64,1.0e-10,120.0,3,2.2995e-5,0.0,1936,21,0.243856874,0.041918872,435483440,715400,0.056925007,0.010472206,71471496,0.01005424,0.0,13431360,0.176877627,0.243879869,0.23505806922912598,6,false,3,4.547473508864641e-13,,2,32,38
+"656491d6daeaaa17ec6cd22b32014a7b5d376374","1.12.6",1,"Linux","x86_64","0.2.3","0.9.30","0.4.1","1.8.0","pisinger_weakly_correlated_n12","penalty_qubo",12,17,30,0.001,0.063,36.0,41.0,false,,-39.425,-39.425,66,6,64,1.0e-10,120.0,3,2.531e-6,0.0,6192,22,0.146620493,0.019608789,220426992,980290,0.005336626,0.0,8251704,0.00953884,0.0,18740096,0.131745027,0.146623024,0.12125015258789062,6,false,20,0.0,,3,,3
+"656491d6daeaaa17ec6cd22b32014a7b5d376374","1.12.6",1,"Linux","x86_64","0.2.3","0.9.30","0.4.1","1.8.0","pisinger_weakly_correlated_n12","penalty_qubo",12,17,30,0.01,0.63,36.0,31.0,true,5.0,-30.370000000000005,-29.0,66,6,64,1.0e-10,120.0,3,1.348e-6,0.0,6192,22,0.26146652,0.041521081,470864336,1138511,0.00524406,0.0,8251704,0.011479771,0.0,19708928,0.24474268899999999,0.261467868,0.23497891426086426,6,false,41,5.132681230083108,,3,,3
+"656491d6daeaaa17ec6cd22b32014a7b5d376374","1.12.6",1,"Linux","x86_64","0.2.3","0.9.30","0.4.1","1.8.0","pisinger_weakly_correlated_n12","penalty_qubo",12,17,30,0.1,6.300000000000001,36.0,22.0,true,14.0,-22.00000000000091,-15.00000000000091,66,6,64,1.0e-10,120.0,3,1.927e-6,0.0,6192,22,0.301551215,0.051480054,535936696,1128862,0.007859085,0.0,8251704,0.013441067,0.0,19079840,0.280251063,0.301553142,0.2735910415649414,6,false,53,23.695796847343445,,3,,3
+"656491d6daeaaa17ec6cd22b32014a7b5d376374","1.12.6",1,"Linux","x86_64","0.2.3","0.9.30","0.4.1","1.8.0","pisinger_weakly_correlated_n12","penalty_qubo",12,17,30,1.1,69.30000000000001,36.0,18.0,true,18.0,-18.000000000021828,58.29999999998836,66,6,64,1.0e-10,120.0,3,1.519e-6,0.0,6192,22,0.315270893,0.053545314,528043200,1123765,0.005385221,0.0,8251704,0.010773531,0.0,18545936,0.299112141,0.315272412,0.2906358242034912,6,false,45,994.5305733680725,,3,,3
+"656491d6daeaaa17ec6cd22b32014a7b5d376374","1.12.6",1,"Linux","x86_64","0.2.3","0.9.30","0.4.1","1.8.0","pisinger_strongly_correlated_n16","projection",16,16,40,,,51.0,51.0,true,0.0,,51.0,66,6,64,1.0e-10,120.0,3,2.1975e-5,0.0,2160,21,1.346630832,0.579085034,1986555432,1090212,0.119454062,0.0,275267552,0.012928162,0.0,18739776,1.2142486080000001,1.346652807,1.3238251209259033,6,false,12,0.0,,2,42,54
+"656491d6daeaaa17ec6cd22b32014a7b5d376374","1.12.6",1,"Linux","x86_64","0.2.3","0.9.30","0.4.1","1.8.0","pisinger_strongly_correlated_n16","penalty_qubo",16,22,40,0.001,0.096,51.0,57.0,false,,-54.60000000000002,-54.599999999999966,66,6,64,1.0e-10,120.0,3,2.258e-6,0.0,9520,22,0.264331342,0.043163207,398417944,1484111,0.009239887,0.0,14286392,0.013307363,0.0,25046976,0.241784092,0.2643336,0.22033381462097168,6,false,26,7.275957614183426e-12,,3,,3
+"656491d6daeaaa17ec6cd22b32014a7b5d376374","1.12.6",1,"Linux","x86_64","0.2.3","0.9.30","0.4.1","1.8.0","pisinger_strongly_correlated_n16","penalty_qubo",16,22,40,0.01,0.96,51.0,44.0,true,7.0,-43.039999999999736,-40.999999999999545,66,6,64,1.0e-10,120.0,3,2.128e-6,0.0,9520,22,0.489069621,0.079302856,969823872,1576875,0.009430229,0.0,14286392,0.016954101,0.0,25914224,0.462685291,0.489071749,0.4527759552001953,6,false,57,1.786125825252384,,3,,3
+"656491d6daeaaa17ec6cd22b32014a7b5d376374","1.12.6",1,"Linux","x86_64","0.2.3","0.9.30","0.4.1","1.8.0","pisinger_strongly_correlated_n16","penalty_qubo",16,22,40,0.1,9.600000000000001,51.0,37.0,true,14.0,-37.0,-21.00000000000182,66,6,64,1.0e-10,120.0,3,2.535e-6,0.0,9520,22,0.550872676,0.100512672,1096552624,1567886,0.009571062,0.0,14286392,0.014675211,0.0,25220176,0.526626403,0.550875211,0.49419593811035156,6,false,73,54.594663351774216,,3,,3
+"656491d6daeaaa17ec6cd22b32014a7b5d376374","1.12.6",1,"Linux","x86_64","0.2.3","0.9.30","0.4.1","1.8.0","pisinger_strongly_correlated_n16","penalty_qubo",16,22,40,1.1,105.60000000000001,51.0,27.0,true,24.0,-27.0,-22.0,66,6,64,1.0e-10,120.0,3,2.46e-6,0.0,9520,22,0.631544734,0.090150438,1279942864,1567079,0.013132645,0.0,14286392,0.019606204,0.0,25543776,0.598805885,0.631547194,0.5824029445648193,6,false,87,6499.746471405029,,3,,3
+"656491d6daeaaa17ec6cd22b32014a7b5d376374","1.12.6",1,"Linux","x86_64","0.2.3","0.9.30","0.4.1","1.8.0","pisinger_subset_sum_n16","projection",16,16,44,,,44.0,44.0,true,0.0,,44.0,66,6,64,1.0e-10,120.0,3,6.469e-6,0.0,2160,21,1.466214629,0.61880987,3180177616,1333144,0.066119857,0.007055779,157610256,0.010183218,0.0,17911104,1.3899115539999998,1.466221098,1.4405999183654785,6,false,12,0.0,,2,46,37
+"656491d6daeaaa17ec6cd22b32014a7b5d376374","1.12.6",1,"Linux","x86_64","0.2.3","0.9.30","0.4.1","1.8.0","pisinger_subset_sum_n16","penalty_qubo",16,22,44,0.001,0.088,44.0,50.0,false,,-46.83200000000005,-46.80000000000004,66,6,64,1.0e-10,120.0,3,2.485e-6,0.0,9520,22,0.36939678,0.045507985,682185952,1553356,0.013506043,0.0,14286392,0.017350049,0.0,25067968,0.338540688,0.369399265,0.33707404136657715,6,false,48,0.0013971109292469919,,3,,3
+"656491d6daeaaa17ec6cd22b32014a7b5d376374","1.12.6",1,"Linux","x86_64","0.2.3","0.9.30","0.4.1","1.8.0","pisinger_subset_sum_n16","penalty_qubo",16,22,44,0.01,0.88,44.0,35.0,true,9.0,-34.12000000000012,-32.11999999999966,66,6,64,1.0e-10,120.0,3,1.812e-6,0.0,9520,22,0.575167182,0.097958225,1143150704,1573437,0.009405563,0.0,14286392,0.01566467,0.0,25649696,0.5500969490000001,0.5751689940000001,0.5354421138763428,6,false,71,2.4017899446189404,,3,,3
+"656491d6daeaaa17ec6cd22b32014a7b5d376374","1.12.6",1,"Linux","x86_64","0.2.3","0.9.30","0.4.1","1.8.0","pisinger_subset_sum_n16","penalty_qubo",16,22,44,0.1,8.8,44.0,21.0,true,23.0,-21.0,-20.0,66,6,64,1.0e-10,120.0,3,2.671e-6,0.0,9520,22,0.512100075,0.099561173,956710000,1562372,0.019789647,0.008306125,14286392,0.022166623,0.006352656,24578816,0.47014380499999997,0.5121027459999999,0.466458797454834,6,false,58,5.210645318031311,,3,,3
+"656491d6daeaaa17ec6cd22b32014a7b5d376374","1.12.6",1,"Linux","x86_64","0.2.3","0.9.30","0.4.1","1.8.0","pisinger_subset_sum_n16","penalty_qubo",16,22,44,1.1,96.80000000000001,44.0,21.0,true,23.0,-21.000000000029104,-20.000000000029104,66,6,64,1.0e-10,120.0,3,1.728e-6,0.0,9520,22,0.543657382,0.060013942,1096145128,1562315,0.009400037,0.0,14286392,0.018834377,0.0,24848496,0.515422968,0.54365911,0.5071709156036377,6,false,85,926.1564331054688,,3,,3

benchmarks/knapsack/results/README.md

@@ -0,0 +1,77 @@
+# Knapsack benchmark result: 2026-07-22
+
+This directory records a complete bounded run of the penalty-versus-projection
+benchmark. The raw measurements are in
+[`2026-07-22-656491d.csv`](2026-07-22-656491d.csv).
+
+## Reproduction
+
+- Benchmark code: `656491d6daeaaa17ec6cd22b32014a7b5d376374`
+- Machine: Intel Xeon w5-2565X, Linux x86-64
+- Julia: 1.12.6, one Julia thread
+- Packages: TenSolver 0.2.3, ITensors 0.9.30, ITensorMPS 0.4.1,
+  BenchmarkTools 1.8.0
+- Workload: six DMRG sweeps, 64 reads, three fixed-seed timing samples, and a
+  120-second soft limit per solve
+
+From the repository root, after instantiating the benchmark environment, the
+recorded command was:
+
+```bash
+julia +1.12 --project=benchmarks benchmarks/knapsack/run.jl \
+  --iterations=6 --reads=64 --timing-samples=3 --time-limit=120 \
+  --benchmark-id=656491d6daeaaa17ec6cd22b32014a7b5d376374 \
+  benchmarks/knapsack/results/2026-07-22-656491d.csv
+```
+
+All 25 rows completed all six requested sweeps without reaching the time limit.
+
+## Results
+
+"Best feasible penalty" selects the penalty row with the highest original
+knapsack value. The final column compares projection time with the median time
+over all four penalty factors; it is not an equal-quality speedup.
+
+| Instance | Items | Exact / projection value | Projection (s) | Best feasible penalty: factor, value, gap, time (s) | Projection / median penalty time |
+| --- | ---: | ---: | ---: | --- | ---: |
+| Reference | 4 | 13 / 13 | 0.0579 | 0.1, 13, 0, 0.0514 | 1.18x |
+| Uncorrelated | 8 | 41 / 41 | 0.1031 | 0.01, 37, 4, 0.1403 | 0.57x |
+| Weakly correlated | 12 | 36 / 36 | 0.2439 | 0.01, 31, 5, 0.2615 | 0.87x |
+| Strongly correlated | 16 | 51 / 51 | 1.3467 | 0.01, 44, 7, 0.4891 | 2.59x |
+| Subset sum | 16 | 44 / 44 | 1.4662 | 0.01, 35, 9, 0.5752 | 2.78x |
+
+Projection returned the exact feasible optimum on all five instances. Of the
+20 penalty rows, 14 were feasible and only the reference instance at factor
+0.1 was exact. Every factor-0.001 row was infeasible; larger factors restored
+feasibility but did not guarantee that the six-sweep DMRG run reached the
+encoded global optimum.
+
+The warm, repeated measurements do not reproduce the earlier compilation-heavy
+slowdown. Projection and penalty are comparable on the reference instance,
+projection is faster than the median penalty row at 8 and 12 items, and it
+becomes slower on both 16-item instances. The projected Hamiltonian bond grows
+to 54 and 37 on those two cases, versus 3 for the penalty Hamiltonians. Observer
+setup accounts for only 0.120 and 0.066 seconds respectively; even after
+subtracting it, the projection solver paths take 1.214 and 1.390 seconds. This
+points to the larger effective projected Hamiltonian and its contractions, not
+callback construction or first-use compilation, as the dominant measured cost.
+
+These runtime comparisons must remain paired with solution quality: none of the
+faster 16-item penalty rows reaches the projection result. The data supports a
+quality-versus-cost tradeoff for this bounded workload, not a general claim
+that either formulation is faster.
+
+## Truncation-error limitation
+
+`truncation_error` is intentionally empty. TenSolver's public iteration callback
+runs after a sweep has discarded singular values, so the discarded weight
+cannot be reconstructed from the retained MPS. This limitation is acceptable
+for this benchmark because feasibility, exact optimality gap, independently
+calculated final variance, bond dimensions, allocations, and phase timings are
+all recorded. It does prevent using this dataset to attribute a poor result
+specifically to truncation, so the empty field and that narrower conclusion
+must remain explicit.
+
+This is a single-machine, single-thread, small-instance run with three timing
+samples and a fixed sweep budget. It is reproducible evidence for this PR, not
+a broad performance characterization.