Revisiting the LiRA Membership Inference Attack Under Realistic Assumptions

This repository provides the official implementation of the paper "Revisiting the LiRA Membership Inference Attack Under Realistic Assumptions" (Jebreel, Khalil, Sánchez, Domingo-Ferrer — PoPETs 2026).

We show that prior work often overestimated attack success by using overconfident models, target-data threshold calibration, and balanced membership priors.

Overview

Key Contributions:

Evaluation under anti-overfitting (AoF) and transfer-learning (TL) settings
Shadow-only threshold calibration and precision under skewed priors (π ≤ 10%)
Per-sample reproducibility analysis across architectures and configurations
Comprehensive implementation of LiRA variants across 4 datasets and 10 benchmarks

Main Findings:

AoF and TL defenses reduce LiRA success while maintaining model utility
Shadow-calibrated thresholds and realistic priors substantially lower attack precision
Membership predictions are unstable; reproducibility requires support thresholding

Installation

Option A — Conda (recommended):

Pinned reproduction environment:

conda env create -f environment.yml
conda activate lira-repro

Quick development install:

conda create -n lira python=3.11
conda activate lira
pip install .          # or: pip install .[dev]

Notes:

Python 3.10+ is required.
The default configs favor deterministic reruns for paper reproduction.
On servers or environments where the open file descriptor limit has not been raised automatically (e.g., non-Slurm setups), run ulimit -n 65536 in your shell before executing any pipeline commands.

Option B — Docker:

# Build once (~10–15 min, ~6 GB)
docker build -t lira-analysis:main .

# Launch an interactive shell (data dirs are bind-mounted)
docker run --gpus all --rm -it \
    -v $(pwd)/data:/workspace/data \
    -v $(pwd)/experiments:/workspace/experiments \
    -v $(pwd)/analysis_results:/workspace/analysis_results \
    lira-analysis:main bash

All python commands below work unchanged inside the container. Outputs written to the mounted directories persist on the host after the container exits.

Pipeline

The pipeline has five phases. All commands are run from the repo root. The full command reference is in cli_commands.txt.

Phase 1 — Train shadow models

python train.py --config configs/train_image.yaml
# or fine-tune from ImageNet pretrained weights:
python train.py --config configs/finetune.yaml

Use experiment.run_name to give the experiment a readable directory name instead of a timestamp — this is the recommended convention for all named runs:

python train.py --config configs/train_image.yaml \
  --override seed=1 experiment.run_name=seed1

Output: experiments/<dataset>/<arch>/<run_name>/model_0/ … model_255/, train_test_stats.csv (per-model train/test accuracy → Table 2), shadow_metrics_aggregate.csv (aggregate loss ratio → Table 2).

If experiment.run_name is not set, the directory falls back to a timestamp (experiments/<dataset>/<arch>/<timestamp>/).

Shard across machines by splitting the model index range (all shards must share the same keep_indices.npy from the first shard):

python train.py --config configs/train_image.yaml \
  --override seed=1 experiment.run_name=seed1 \
             training.start_shadow_model_idx=0 \
             training.end_shadow_model_idx=63

Phase 2 — LiRA attack

python attack.py --config configs/attack.yaml

# Point to a specific run on the fly
python attack.py --config configs/attack.yaml \
  --override experiment.checkpoint_dir=experiments/cifar10/resnet18/seed1

# Skip recomputing cached logits or scores
python attack.py --config configs/attack.yaml \
  --override attack.compute_logits=false \
             attack.compute_scores=false

Output inside experiments/<dataset>/<arch>/<run>/:

online_scores_leave_one_out.npy, offline_scores_leave_one_out.npy, … — raw attack scores
membership_labels.npy — ground-truth member/non-member labels
roc_curve_single.pdf — ROC curve visual sanity check
attack_results_leave_one_out_summary.csv — AUC, TPR@FPR per attack → Tables 3–6

Phase 3 — Per-run post-analysis

python comprehensive_analysis/run_analysis.py \
  --exp-path experiments/cifar10/resnet18/seed1 \
  --target-fprs 0.00001 0.001 \
  --priors 0.01 0.1 0.5

# Skip image grid generation on headless machines
python comprehensive_analysis/run_analysis.py \
  --exp-path experiments/cifar10/resnet18/seed1 \
  --skip-visualization

Output: analysis_results/<dataset>/<arch>/<run>/

summary_statistics_two_modes.csv + summary_<mode>_prior<p>.tex — TPR/PPV under shadow thresholds and skewed priors → Tables 7–10
per_model_metrics_two_modes.csv — full per-model detail
samples_vulnerability_ranked_*.csv, samples_highly_vulnerable_*.csv — per-sample vulnerability
top9_vulnerable_online_shadow_*.png — top-9 vulnerable sample grids → Figure 4 (assembled in Phase 5)

Phase 4 — Cross-run reproducibility and rank stability

The script auto-discovers all completed Phase 3 runs under --analysis-root (any subdirectory containing summary_statistics_two_modes.csv). Subdirectories matching seed\d+ are treated as baseline runs; everything else as variants. Any number of runs is supported — the paper used 12 seeds + 4 variants, but 3–4 seeds already show the reproducibility trend.

# Uses default analysis-root (analysis_results/cifar10/resnet18)
python comprehensive_analysis/reproducibility_analysis.py

# Point to a different set of runs
python comprehensive_analysis/reproducibility_analysis.py \
  --analysis-root analysis_results/cifar10/resnet18

# Skip individual sections
python comprehensive_analysis/reproducibility_analysis.py --skip-heatmaps
python comprehensive_analysis/reproducibility_analysis.py --skip-rank

Output:

analysis_results/figures/ — panels, heatmaps, and rank-stability figures → Figures 2, 3, 5, 6 (main); 9, 10, 12, 13, 14, 15 (appendix)
analysis_results/tables/ — reproducibility_*runs_*variants_*.csv and rank-stability LaTeX tables → Table 11 (appendix); Table 13 (appendix)

Phase 5 — Paper figure scripts

All scripts read Phase 3/4 outputs and write PDFs to analysis_results/figures/.

Script	Paper output	Output file
`threshold_distribution.py`	Figure 1	`analysis_results/figures/thresh_boxplot_single.pdf`, `thresh_boxplot_multi.pdf`
`compose_top_vulnerable.py`	Figure 4 (collect)	`analysis_results/figures/topk_vulnerable_images/<dataset>/<arch>/<run>/`
`loss_ratio_tpr.py`	Figure 7	`analysis_results/figures/lossratio_tpr.pdf`
`plot_benchmark_distribution.py`	Figure 8	`analysis_results/figures/sample_inout_score.pdf`, `sample_inout_ratio.pdf`
(manual assembly)	Figure 11 (appendix)	Assemble manually — see note below

python comprehensive_analysis/threshold_distribution.py

python comprehensive_analysis/compose_top_vulnerable.py

python comprehensive_analysis/loss_ratio_tpr.py

python comprehensive_analysis/plot_benchmark_distribution.py \
  --config configs/figure_panels/figure8_scores.yaml
python comprehensive_analysis/plot_benchmark_distribution.py \
  --config configs/figure_panels/figure8_ratios.yaml

Figure 11 (appendix — manual): Run Phase 3 for seeds 1–3 with --top-k 16 --nrow 4. This produces analysis_results/cifar10/resnet18/seed{1,2,3}/top16_vulnerable_online_shadow_{0p001pct,0p1pct}.png. Place the six images in a 2 × 3 grid (rows = FPR threshold, columns = seed) using any image editor or LaTeX \includegraphics.

Figures 2, 3, 5, 6 and appendix Figures 9, 10, 12, 13, 14, 15 are produced by Phase 4 (reproducibility_analysis.py). Tables 11 and 13 (appendix) are also produced by Phase 4 (rank-stability .tex outputs).

Estimated Reproduction Time

All benchmarks used 256 shadow models. Timings measured on the authors' machine: NVIDIA GeForce RTX 4080 (16 GB), Ubuntu 20.04 (WSL2), CUDA 12.6, Windows 11 Education. Training ran sequentially on a single GPU; Phase 2 used aug=18 for all image benchmarks.

Times marked † are from the paper (Appendix C, full early-stopping run on RTX 4080); times marked ★ are extrapolated from 1-epoch timing runs on the same machine. ★† = upper bound; FCN converges fast, early stopping reduces this substantially.

Benchmark	Arch	Epochs	Phase 1 training	Phase 2 attack	Total
`purchase100_baseline`	FCN (tabular)	100	≤ ~16 h ★†	< 30 min ★	≤ ~16 h
`purchase100_aof`	FCN (tabular)	100	≤ ~15 h ★†	< 30 min ★	≤ ~15 h
`cifar10_baseline`, `cifar10_aof`	ResNet-18	100	~29 h † (≈ 7 min/model)	~4–5 h † (15–17% overhead)	~33–34 h
`cifar100_baseline`, `cifar100_aof`	WRN-28-2	100	~33 h † (≈ 8 min/model)	~5–6 h † (15–17% overhead)	~38–39 h
`cifar10_tl`, `cifar100_tl`, `gtsrb_tl`	EfficientNetV2 (pretrained)	5	~4–8 h	~3–5 h	~8–13 h
`gtsrb_baseline`	ResNet-18	100	~20–28 h	~3–4 h	~23–32 h

Phases 3–4 (post-analysis, reproducibility figures) take < 1 h combined. If logits are already cached, Phase 2 re-runs in < 30 min (Purchase) or ~1 h (image datasets).

Benchmark Manifests (recommended for paper runs)

Each manifest under configs/benchmarks/ encodes the exact train + attack + analysis config for one Table 1 row. This is the preferred way to reproduce paper results.

# List available benchmarks
python scripts/run_benchmark.py --list

# Full run (train → attack → analyze)
python scripts/run_benchmark.py --benchmark cifar10_baseline

# Dry-run: print commands without executing
python scripts/run_benchmark.py --benchmark cifar10_baseline --dry-run

# Skip stages whose output markers already exist
python scripts/run_benchmark.py --benchmark cifar10_baseline --skip-existing

# Run only specific stages
python scripts/run_benchmark.py --benchmark cifar10_baseline --stages attack analysis

Available benchmarks: cifar10_baseline, cifar10_aof, cifar10_tl, cifar100_baseline, cifar100_aof, cifar100_tl, gtsrb_baseline, gtsrb_tl, purchase100_baseline, purchase100_aof

Project Structure

lira_analysis/
├── train.py
├── attack.py
├── attacks/
├── utils/
│   ├── cli.py
│   └── benchmark_cli.py
├── configs/
│   ├── train_image.yaml
│   ├── train_tabular.yaml
│   ├── finetune.yaml
│   ├── attack.yaml
│   ├── benchmarks/               # one YAML per Table 1 row
│   └── figure_panels/            # Figure 8 panel manifests
├── scripts/
│   └── run_benchmark.py
├── comprehensive_analysis/
│   ├── run_analysis.py           # Phase 3: per-run post-analysis
│   ├── reproducibility_analysis.py  # Phase 4: cross-run analysis
│   ├── threshold_distribution.py    # Figure 1
│   ├── compose_top_vulnerable.py    # collect per-run top-vulnerable grids
│   ├── loss_ratio_tpr.py            # Figure 7
│   └── plot_benchmark_distribution.py  # Figure 8
└── experiments/

Datasets and Models

Dataset	Type	Classes	Samples	Models
CIFAR-10 / CIFAR-100	image	10 / 100	60K	ResNet-18, WideResNet, EfficientNet-V2
GTSRB	image	43	51K	ResNet-18, EfficientNet-V2
Purchase-100	tabular	100	197K	FCN

Notes:

Image datasets download automatically on first use.
Purchase-100 must be staged under data/purchase/, as documented in data/Readme.md.

Attack Variants

LiRA (online) — uses both in- and out-of-distribution shadow statistics
LiRA (online, fixed var) — online with global variance
LiRA (offline) — uses only out-of-distribution statistics
LiRA (offline, fixed var) — offline with global variance
Global threshold — score baseline, no shadow statistics

Analysis

All paper-facing post analysis is under comprehensive_analysis/.

Scripts:

run_analysis.py — per-run TPR, PPV, vulnerability analysis
reproducibility_analysis.py — Jaccard, heatmaps, rank stability
threshold_distribution.py — Figure 1
compose_top_vulnerable.py — collect per-run top-vulnerable grids (Figure 4 assembled manually)
loss_ratio_tpr.py — Figure 7
plot_benchmark_distribution.py — Figure 8
Figure 11 (appendix) — assembled manually from per-run top-16 grids (see Phase 5 note)

For the paper-to-artifact map and badge requirements, see ARTIFACT-APPENDIX.md. For the expected output path index, see RESULTS_INDEX.md. For the complete CLI command reference, see cli_commands.txt.

Future Work

Multi-GPU training: Shadow-model training currently runs on a single GPU. A natural extension would be to parallelise it across multiple GPUs on a single node via torch.nn.DataParallel or DistributedDataParallel, which would significantly reduce Phase 1 wall-clock time. This is planned as a future improvement and is outside the scope of the current artifact.

Citation

@misc{jebreel2026revisitingliramembershipinference,
  title         = {Revisiting the {LiRA} Membership Inference Attack Under Realistic Assumptions},
  author        = {Najeeb Jebreel and Mona Khalil and David S\'{a}nchez and Josep Domingo-Ferrer},
  year          = {2026},
  eprint        = {2603.07567},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CR},
  url           = {https://arxiv.org/abs/2603.07567},
}

The proceedings citation (PoPETs 2026 volume/issue/DOI) will be added after publication.

Acknowledgments

Partial support to this work has been received from the Government of Catalonia (ICREA Acadèmia Prizes to J. Domingo-Ferrer and to D. Sánchez), MCIN/AEI under grant PID2024-157271NB-I00 "CLEARING-IT", and the European Commission under project HORIZON-101292277 "SoBigData IP".

We used Claude Sonnet 4.5 to optimize code implementation and WriteFull and ChatGPT to correct typos, grammatical errors, and awkward phrasing throughout the article.

Original LiRA: Carlini et al., 2022

License

Released under the MIT License. See LICENSE.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Revisiting the LiRA Membership Inference Attack Under Realistic Assumptions

Overview

Installation

Pipeline

Phase 1 — Train shadow models

Phase 2 — LiRA attack

Phase 3 — Per-run post-analysis

Phase 4 — Cross-run reproducibility and rank stability

Phase 5 — Paper figure scripts

Estimated Reproduction Time

Benchmark Manifests (recommended for paper runs)

Project Structure

Datasets and Models

Attack Variants

Analysis

Future Work

Citation

Acknowledgments

License

FilesExpand file tree

README.md

Latest commit

History

README.md

File metadata and controls

Revisiting the LiRA Membership Inference Attack Under Realistic Assumptions

Overview

Installation

Pipeline

Phase 1 — Train shadow models

Phase 2 — LiRA attack

Phase 3 — Per-run post-analysis

Phase 4 — Cross-run reproducibility and rank stability

Phase 5 — Paper figure scripts

Estimated Reproduction Time

Benchmark Manifests (recommended for paper runs)

Project Structure

Datasets and Models

Attack Variants

Analysis

Future Work

Citation

Acknowledgments

License