Merge pull request #20 from cwida/dp-sass-variable-m

Add variable-m DP-SAA and reproducible benchmarks

Author: ila

.gitignore

@@ -15,19 +15,27 @@ test/python/__pycache__/
 *.parquet
 
 # Generated benchmark result outputs
+benchmark/**/*.png
+benchmark/**/*.log
+benchmark/**/__pycache__/
+benchmark/results/*
+!benchmark/results/paper/
+!benchmark/results/paper/*.csv
+!benchmark/results/paper/README.md
+benchmark/dp/tmp/
 benchmark/tpch/dp_elastic_*.csv
 benchmark/tpch/dp_sass_*.csv
+benchmark/tpch/as_tpch_*.csv
 benchmark/clickbench/clickbench_*.csv
+benchmark/clickbench/as_clickbench_*_results.csv
 benchmark/clickbench/*.png
+benchmark/dp/*_results.csv
+benchmark/dp/*_summary.csv
+benchmark/dp/tpch_sf30_runtime_perfect_*.csv
 benchmark/dp/sqlstorm_stackoverflow_*stability*.csv
-benchmark/dp/jcch_*_results.csv
 benchmark/sqlstorm/sqlstorm_degradation_*.csv
 benchmark/sqlstorm/data/
-
-# Generated query sets (produced by the stability/scan scripts)
-benchmark/dp/sqlstorm_stackoverflow_stable_queries/
-benchmark/dp/sqlstorm_stackoverflow_stable_aggregate_queries/
-benchmark/dp/uber_like_jcch/
+benchmark/tpch/*_paper_summary.csv
 
 # Serialized privacy-metadata dumps
 pac_metadata_*.json

.gitmodules

@@ -12,6 +12,3 @@
 [submodule ".claude/skills/shared"]
 	path = .claude/skills/shared
 	url = https://github.com/ila/duckdb-claude-skills.git
-[submodule "benchmark/jcch/dbgen.JCC-H"]
-	path = benchmark/jcch/dbgen.JCC-H
-	url = https://github.com/ldbc/dbgen.JCC-H

CLAUDE.md

@@ -26,7 +26,7 @@ Never execute git commands that could lose code. Always ask the user for permiss
 
 - **`pac`** (default): PAC Privacy — empirical MIA resistance. Maintains 64 parallel counters per aggregate (one per "world" bit), adds noise calibrated to the query variance across sub-samples. Provides theoretical mutual-information bounds. Not differential privacy.
 - **`dp_standard`**: Global-sensitivity DP — pure ε-DP for **ungrouped** queries, (ε,δ)-DP for **grouped** ones (see partition selection below). Bounds each privacy unit's *total* contribution to a fixed constant (per-PU contribution clipping), so global sensitivity equals that constant — data-independent. On a single PU table this is per-row clipping (COUNT sensitivity = 1, SUM = `dp_sum_bound`); on a join it pre-aggregates per PU (grouping by the PU-adjacent FK), clips each PU's partial to `dp_sum_bound`/`dp_count_bound`, then sums — so a join with COUNT requires `dp_count_bound`. Laplace noise calibrated to `bound/epsilon`. **Grouped queries apply private partition selection by default** (noised `COUNT(DISTINCT pu) ≥ τ`, the same Wilson/Google mechanism as `dp_sass`): the released key set is data-dependent, so this reserves `ε_η = ε/(c+1)` and needs `δ_η`, making grouped `dp_standard` (ε,δ)-DP and **requiring `dp_delta`**; ungrouped `dp_standard` stays pure ε-DP.
-- **`dp_elastic`**: Elastic sensitivity DP — formal (ε,δ)-DP **at the row level** (FLEX/Chorus neighbor = change one tuple). Same Laplace pipeline as `dp_standard` but uses per-row clipping and the smoothed join-frequency envelope (`2·SES_β`, β derived from ε,δ) for sensitivity. Requires `dp_delta`. **Not user-level**: `∏mf` bounds a single-tuple change, not a whole PU's contribution — a user-level extension would need per-user statistics and a new sensitivity proof (future work). It follows FLEX's public/enumerated-partition model (no *DP* partition selection), unlike the user-level `dp_standard`/`dp_sass`. Its `privacy_min_group_count` gate is a **raw (un-noised) admin support filter, not DP partition selection** — under FLEX's public-partition assumption it's a utility convenience; making it row-level-DP-safe would need a noised τ with `C_u = 1` (Dandan's row-level branch), deferred.
+- **`dp_elastic`**: Elastic sensitivity DP — formal (ε,δ)-DP **at the row level** (FLEX/Chorus neighbor = change one tuple). Same Laplace pipeline as `dp_standard` but uses per-row clipping and the smoothed join-frequency envelope (`2·SES_β`, β derived from ε,δ) for sensitivity. Requires `dp_delta`. **Not user-level**: `∏mf` bounds a single-tuple change, not a whole PU's contribution — a user-level extension would need per-user statistics and a new sensitivity proof (future work). It follows FLEX's public/enumerated-partition model (no *DP* partition selection), unlike the user-level `dp_standard`/`dp_sass`. Its `privacy_min_group_count` gate is a **raw (un-noised) admin support filter, not DP partition selection** — under FLEX's public-partition assumption it's a utility convenience; making it row-level-DP-safe would need a noised τ with `C_u = 1` (the row-level branch), deferred.
 - **`dp_sass`**: Sample-and-aggregate. Computes 64 sample (lane) aggregates per privacy unit and releases either the **median** (`dp_sass_release='median'`, default; smooth-sensitivity Laplace, (ε,δ)-DP, requires `dp_delta`) or the **mean** (`dp_sass_release='average'`; GUPT-style, pure ε-DP, no `dp_delta`). Per-PU contribution is clipped (`dp_count_bound`/`dp_sum_bound`); the cross-group bound `dp_max_groups_contributed` (C_u) enters each aggregate's budget. **Grouped queries apply private partition selection automatically** (per the paper's Algorithm 2): a noised `COUNT(DISTINCT pu)` is released only when `≥ τ = 1 − C_u·log(2−2(1−δ_η)^{1/C_u})/ε_η`, reserving `ε_η = ε/(c+1)` and `δ_η = δ/(c+1)` so each of the c aggregates gets `ε/((c+1)·C_u)`. Grouped `dp_sass` therefore requires `dp_delta` (the pure-ε `average` release is only valid ungrouped). The released value is clamped to a public output domain (`dp_sass_count_output_bound`, `dp_sass_sum_output_bound`, `dp_sass_avg_lower/upper_bound`, `dp_sass_minmax_lower/upper_bound`); with `dp_sass_private_range=true` (average release only) a fraction `dp_sass_range_budget_fraction` of ε privately estimates that range (pure-ε exp-mechanism quantiles). Supports COUNT, COUNT(DISTINCT), SUM, AVG, MIN, MAX.
 
 `dp_standard` and `dp_elastic` share one code path (`CompileDPLaplaceQuery` in `privacy_mechanisms.cpp`); they differ only in how contributions are clipped and how sensitivity is derived (`ClipAndComputeSensitivities`). All four modes share the same DDL (`PRIVACY_KEY`, `PRIVACY_LINK`, `PROTECTED`) and rewrite aggregate plans transparently — users write normal SQL.

CMakeLists.txt

@@ -106,23 +106,30 @@ if(NOT EMSCRIPTEN AND NOT WIN32)
     add_executable(as_clickbench_benchmark benchmark/clickbench/as_clickbench_benchmark.cpp)
     target_link_libraries(as_clickbench_benchmark ${PRIVACY_LINK_LIBS})
 
-    # TPCH Compiler benchmark runner (compares PAC vs regular TPC-H)
-    add_executable(pac_tpch_compiler_benchmark benchmark/tpch/pac_tpch_compiler_benchmark.cpp)
-    target_link_libraries(pac_tpch_compiler_benchmark ${PRIVACY_LINK_LIBS})
-
     # Unified DP utility benchmark runner
+    set(TPCH_STOCK_DP_QUERY_NAMES q01 q05 q06 q14 q19)
+    set(TPCH_STOCK_DP_QUERY_FILES)
+    foreach(QUERY_NAME IN LISTS TPCH_STOCK_DP_QUERY_NAMES)
+        string(TOUPPER ${QUERY_NAME} QUERY_NAME_UPPER)
+        set(QUERY_FILE "${CMAKE_CURRENT_SOURCE_DIR}/benchmark/dp/tpch_queries/${QUERY_NAME}.sql")
+        list(APPEND TPCH_STOCK_DP_QUERY_FILES "${QUERY_FILE}")
+        file(READ
+            "${QUERY_FILE}"
+            "TPCH_${QUERY_NAME_UPPER}_SQL")
+    endforeach()
+    set_property(DIRECTORY APPEND PROPERTY CMAKE_CONFIGURE_DEPENDS
+        ${TPCH_STOCK_DP_QUERY_FILES})
+    set(TPCH_QUERY_GENERATED_DIR "${CMAKE_CURRENT_BINARY_DIR}/generated")
+    file(MAKE_DIRECTORY "${TPCH_QUERY_GENERATED_DIR}")
+    configure_file(
+        benchmark/include/tpch_stock_queries.hpp.in
+        "${TPCH_QUERY_GENERATED_DIR}/tpch_stock_queries.hpp"
+        @ONLY)
     add_executable(dp_benchmark_runner benchmark/dp/dp_benchmark_runner.cpp)
+    target_include_directories(dp_benchmark_runner PRIVATE "${TPCH_QUERY_GENERATED_DIR}")
     target_link_libraries(dp_benchmark_runner ${PRIVACY_LINK_LIBS})
 
-    # ClickHouse/ClickBench benchmark runner
-    add_executable(pac_clickhouse_benchmark benchmark/clickbench/pac_clickhouse_benchmark.cpp)
-    target_link_libraries(pac_clickhouse_benchmark ${PRIVACY_LINK_LIBS})
-
     # IMDb/JOB benchmark runner
     add_executable(imdb_benchmark benchmark/imdb/imdb_benchmark.cpp)
     target_link_libraries(imdb_benchmark ${PRIVACY_LINK_LIBS})
-
-    # SQLStorm benchmark runner
-    add_executable(pac_sqlstorm_benchmark benchmark/sqlstorm/pac_sqlstorm_benchmark.cpp)
-    target_link_libraries(pac_sqlstorm_benchmark ${PRIVACY_LINK_LIBS})
 endif()

README.md

@@ -250,27 +250,3 @@ ALTER PU TABLE t DROP PROTECTED (col2);
 
 For implementation details, see the [docs/](docs/README.md) folder: \
 [Syntax](docs/syntax.md) | [SIMD aggregates](docs/simd-asa/aggregates.md) | [Query Operators](docs/pac/query_operators.md) | [DP mechanisms](docs/dp/dp_mechanisms.md) | [Runtime Checks](docs/pac/runtime_checks.md) | [Tests](docs/test/README.md) | [Benchmarks](docs/benchmark/README.md)
-
-## Literature
-
-I. Battiston, D. Yuan, X. Zhu, P. Boncz. [SIMD-PAC-DB: Pretty Performant PAC Privacy](https://arxiv.org/abs/2603.15023). 2026.
-
-N. Johnson, J.M. Near, D. Song. [Towards Practical Differential Privacy for SQL Queries](https://arxiv.org/abs/1706.09479) (Flex / Elastic Sensitivity). VLDB 2018.
-
-R. Wilson, C. Zhang, W. Lam, D. Desfontaines, D. Simmons-Marengo, B. Gipson. [Differentially Private SQL with Bounded User Contribution](https://arxiv.org/abs/1909.01917). PoPETs 2020.
-
-```bibtex
-@misc{battiston2026simdpacdbprettyperformantpac,
-      title={SIMD-PAC-DB: Pretty Performant PAC Privacy},
-      author={Ilaria Battiston and Dandan Yuan and Xiaochen Zhu and Peter Boncz},
-      year={2026},
-      eprint={2603.15023},
-      archivePrefix={arXiv},
-      primaryClass={cs.DB},
-      url={https://arxiv.org/abs/2603.15023},
-}
-```
-
-## Maintainer
-
-This extension is maintained by **@ila** (ilaria@cwi.nl).

attacks/clip_attack_results.md

@@ -123,9 +123,9 @@ threshold. Only the outlier level (1 contributor) is affected.
 
 ---
 
-## Attack 6: Clip-after-filter vs clip-full-table (Dandan's hypothesis)
+## Attack 6: Clip-after-filter vs clip-full-table
 
-Dandan's concern: clipping applied after filtering may leak more than clipping
+The concern is that clipping applied after filtering may leak more than clipping
 applied to the entire dataset, because the filter changes which users contribute
 to the bitmap, affecting which levels appear "supported."
 
@@ -137,7 +137,7 @@ to the bitmap, affecting which levels appear "supported."
 | clip-after-filter (pac_clip_support=2) | **72.5%** | 613,511 | 122,026 | 5.0x |
 | clip-full-table (pre-clip to mu+3sigma) | **56.9%** | 180,457 | 124,641 | 1.4x |
 
-**Finding: Dandan is correct.** Pre-clipping the full table then filtering gives
+**Finding: the concern is valid.** Pre-clipping the full table then filtering gives
 significantly better protection (56.9% vs 72.5%). The reasons:
 
 1. **Full-table pre-clipping** clamps the billionaire to 13,661 BEFORE PAC sees it.
@@ -149,7 +149,7 @@ significantly better protection (56.9% vs 72.5%). The reasons:
    The attenuation is only ~16x (one level), leaving a 5x variance gap.
 
 However, clip-after-filter is still much better than no clipping (72.5% vs 96%),
-confirming Dandan's second point: "this approach is still significantly better
+confirming the second point: "this approach is still significantly better
 than not applying clipping at all."
 
 ---
@@ -266,7 +266,7 @@ Best threshold accuracy: **100.0%**. Attack fully succeeds.
 "supported." The clipping mechanism assumes outlier levels have few contributors.
 Collusion (or any scenario with 2+ users at the same extreme level) defeats it.
 
-### TEST 5: Dandan's filter probing
+### TEST 5: Filter probing
 
 Attacker uses two queries with different filters to probe clipping behavior.
 
@@ -276,7 +276,7 @@ Attacker uses two queries with different filters to probe clipping behavior.
 | Filter<=999 (wide) | 51.7% |
 | Cross-filter differential | **51.0%** |
 
-**Dandan's concern is NOT exploitable with hard-zero.** The narrow query zeroes the
+**The filter-probing concern is NOT exploitable with hard-zero.** The narrow query zeroes the
 outlier level, giving identical counter distributions for in/out. The wide query
 has the outlier's level zeroed too (1 contributor < threshold). The cross-filter
 differential reveals nothing.
@@ -307,7 +307,7 @@ Even the minimum level-3 value is zeroed when it's the sole contributor.
 3. **Two colluding outliers defeat the clipping** by making their level "supported"
    (2 contributors >= threshold 2). Attack accuracy: 100%.
 
-4. **Dandan's filter-probing concern does not apply with hard-zero.** The zeroed level
+4. **Filter probing does not apply with hard-zero.** The zeroed level
    contributes nothing regardless of filter, so different filters reveal no info.
 
 5. **The pre-aggregation step remains essential** — 20K small items are correctly

attacks/clip_attack_test.sh

@@ -117,14 +117,14 @@ for CLIP in off 50 100; do
     run_scenario "atk5" 1000 999999 999 "$CLIP" 15
 done
 
-# --- Attack 6: Clip after filter vs clip on full table (Dandan's concern) ---
+# --- Attack 6: Clip after filter vs clip on full table ---
 # pac_clip_sum clips AFTER filtering (only filtered rows enter the aggregate).
 # An adversary might exploit this: the clipping behavior differs depending on
 # which users are in the filter. Compare filter-then-clip (what pac_clip_sum does)
 # vs clip-all-then-filter (manual pre-clipping of the full table, then query).
 echo "## ATTACK 6: Clip-after-filter vs clip-full-table"
 echo "N=1000, target=999999, filter<=3, $NT trials"
-echo "Tests Dandan's hypothesis: clipping after filtering leaks more than"
+echo "Tests whether clipping after filtering leaks more than"
 echo "clipping the entire dataset. We compare pac_clip_sum (clips filtered rows)"
 echo "vs manual pre-clipping of all rows then querying without clip_support."
 echo ""

attacks/clip_hardzero_stress.sh

@@ -209,9 +209,9 @@ analyze "Two outliers (0 and -1), filt<=3" "$IN_F" "$OUT_F" "$FBG" 999999
 rm -f "$IN_F" "$OUT_F"
 
 # ---------------------------------------------------------------
-# TEST 5: Dandan's filter probing attack
+# TEST 5: Filter probing attack
 # ---------------------------------------------------------------
-echo "## TEST 5: Dandan's filter probing"
+echo "## TEST 5: Filter probing"
 echo "Attacker tries different filters to see if clipping behavior changes."
 echo "If the outlier is present, the bitmap at level 3 has a bit set."
 echo "Query 1: filter<=3 (includes user 0 if present)"

benchmark/clickbench/README.md

@@ -1,67 +1,18 @@
-# ClickBench privacy benchmark
+# ClickBench AS Benchmark
 
-`pac_clickhouse_benchmark` runs the ClickHouse `hits` (ClickBench) queries through the
-`privacy` extension and measures runtime (and, where applicable, utility) for several
-privacy mechanisms against a non-private baseline.
-
-## Running
+`as_clickbench_benchmark` compares plain DuckDB with handwritten stochastic-aggregation forms of the aggregate
+ClickBench queries. Queries without an aggregate are skipped.
 
 ```bash
-# build first: GEN=ninja make
-build/release/extension/privacy/pac_clickhouse_benchmark [options]
+cmake --build build/release --target as_clickbench_benchmark
+build/release/extension/privacy/as_clickbench_benchmark \
+  --config benchmark/configs/utility/as_clickbench_graviton.json --dry-run
+build/release/extension/privacy/as_clickbench_benchmark \
+  --config benchmark/configs/utility/as_clickbench_graviton.json
 ```
 
-Options:
-
-| Flag | Meaning |
-|---|---|
-| `--micro` | Use a smaller dataset (`clickbench_micro.db`) for quick testing |
-| `--db <path>` | DuckDB database file |
-| `--queries <dir>` | Directory with `create.sql`, `load.sql`, `queries.sql`, `setup.sql` |
-| `--out <csv>` | Output CSV path (auto-named if omitted) |
-| `--modes <csv>` | Which mechanisms to measure: subset of `pac,dp_standard,dp_elastic,dp_sass` (default: all four) |
-| `--run-naive` | Also run the explicit sample-table-join "naive" variants for the sampling mechanisms in `--modes` |
-
-The non-private **`baseline`** always runs regardless of `--modes`.
-
-Each selected DP/PAC mechanism runs **once per query** at a single operating point —
-`dp_epsilon = 1.0` and the inferred "perfect" sensitivity bound (no epsilon/sensitivity sweep).
-The DP mechanisms use a common ClickBench support set for reporting; queries outside that
-intersection are recorded as failed for all DP modes instead of letting one mechanism report a
-larger successful-query set than another.
-
-## Modes in the output `mode` column
-
-- `baseline` — plain DuckDB, no privacy.
-- `PAC`, `dp_standard`, `dp_elastic`, `dp_sass` — the four mechanisms (vectorized rewrite path).
-- `naive_pac`, `naive_dp` — explicit-SQL sample-and-aggregate baselines (only with `--run-naive`).
-
-## Naive queries
-
-The naive variants live in:
-
-- `clickbench_naive_pac_queries/qNN.sql`
-- `clickbench_naive_dp_queries/qNN.sql`
-
-They exist only for the **privatizable aggregate** queries (`SUM`/`COUNT`/`AVG` over `hits`
-grouping on non-protected, non-`UserID` columns). Queries with no naive form — `COUNT(DISTINCT
-UserID)`, `MIN`/`MAX`, row dumps (`ORDER BY … LIMIT`), and `GROUP BY UserID` — have no file and
-are skipped automatically by the runner.
-
-Both families share one skeleton and differ only in the terminal UDF:
-
-- **PAC naive**: 128 random sub-samples per privacy unit (`WHERE random() < 0.5`), per-sub-sample
-  `SUM`/`COUNT`/`AVG`, then `pac_aggregate(answers, counts, mi, k)` → mean-of-sub-samples + PAC noise.
-- **DP naive**: 64 lanes, each privacy unit assigned to ~8 of them via an **approximate** SQL hash
-  (`hash(UserID, lane_id) % 64 < 8`), per-lane answers rescaled by `64/8`, then
-  `dp_aggregate(answers, counts, eps, delta, lanes, lower, upper)` → clipped median of the 64
-  lane answers + smooth-sensitivity Laplace noise (wraps `dp_smooth_median_noise`; the `counts`
-  argument is present only for signature symmetry with `pac_aggregate` and is unused by the
-  median path).
+If the configured database does not exist, the runner downloads `hits.parquet` and loads `hits`. It records the median
+of five hot runs in `query,mode,m,median_ms` format. Use `benchmark/configs/sanity/as_clickbench_m_sweep.json` for a
+focused `m=64,512` check.
 
-> **The DP naive variant is a runtime / structural baseline, not a utility-faithful one.** Its
-> lane assignment is approximate (the real `DpSampleHash` 6-bit-chunk bitmask is C++-only and
-> cannot be reproduced in plain SQL), and its domain bounds are hand-set generous constants
-> rather than the per-query inferred bounds used by the real `dp_sass` rewrite. The runner records
-> only **timing + success** for naive rows; their numeric values are expected to be noise-dominated
-> and should not be compared to the vectorized path for accuracy.
+See [repro.md](../../repro.md) for the reported machine settings and plotting command.