|
| 1 | +# Security Policy |
| 2 | + |
| 3 | +## Supported Versions |
| 4 | + |
| 5 | +We ship security fixes in the latest release. Please reproduce on the current released version before reporting — we do not backport fixes to older releases. |
| 6 | + |
| 7 | +## Reporting a Vulnerability |
| 8 | + |
| 9 | +Report privately — **do not open a public issue or PR for a suspected vulnerability.** |
| 10 | + |
| 11 | +- **Preferred:** GitHub private vulnerability reporting — the **"Report a vulnerability"** button under this repo's **Security** tab. This routes to the maintainers, keeps the report private until a fix is ready, and lets us issue a CVE through GitHub if warranted. |
| 12 | +- **Email:** security@huggingface.co |
| 13 | + |
| 14 | +We acknowledge valid, in-scope reports and will keep you updated on remediation. Please give us a reasonable window to fix before any public disclosure. |
| 15 | + |
| 16 | +## Recognition |
| 17 | + |
| 18 | +We do not offer a monetary bounty. For a valid, in-scope report we credit you on the published GitHub Security Advisory and name you as the reporter in the associated CVE. Let us know how you'd like to be credited (name or handle). |
| 19 | + |
| 20 | +## What your report must include |
| 21 | + |
| 22 | +We receive a high volume of reports. To be triaged, a report **must** follow the structure below. Copy this block into your submission and fill in every field. Reports missing the version, the proof of concept, or the impact are returned as incomplete and are not investigated until provided. |
| 23 | + |
| 24 | +```markdown |
| 25 | +### Summary |
| 26 | +One sentence: what the vulnerability is and where. |
| 27 | + |
| 28 | +### Affected version / commit |
| 29 | +Exact released version or commit SHA you reproduced on (e.g. v0.7.0 / a1b2c3d). |
| 30 | +Not "latest" or "main". |
| 31 | + |
| 32 | +### Affected component |
| 33 | +The public API, module, or entry point involved (e.g. `EvaluationTracker.save`, a specific model backend config). |
| 34 | + |
| 35 | +### Vulnerability class |
| 36 | +Type and CWE if known (e.g. deserialization / CWE-502, information exposure / CWE-200). |
| 37 | + |
| 38 | +### Attack vector & preconditions |
| 39 | +- How is the vulnerable code reached? (which API call / CLI flag / config) |
| 40 | +- Who is the attacker and what do they control? |
| 41 | +- What must be true for the attack to work? (a non-default setting, a specific |
| 42 | + backend, a malicious task/dataset being loaded, etc.) |
| 43 | + |
| 44 | +### Proof of concept |
| 45 | +A minimal, self-contained script or step sequence that runs on a clean install |
| 46 | +of the version above. Include: |
| 47 | +- the exact commands / code to run, |
| 48 | +- any input files needed (attach them, or give a script that generates them), |
| 49 | +- the **expected** behavior vs. the **actual** behavior you observed. |
| 50 | +A snippet showing that a function *exists* or *could* be misused is not a PoC. |
| 51 | + |
| 52 | +### Impact |
| 53 | +What an attacker gains in a realistic deployment. "Could theoretically…" |
| 54 | +without a working chain is not an impact. |
| 55 | + |
| 56 | +### Scope |
| 57 | +Which trust boundary (see below) does this cross? If your finding touches |
| 58 | +anything in the "Out of scope" list, name which item and explain why it is |
| 59 | +nonetheless a violation of a guarantee we make. |
| 60 | + |
| 61 | +### Suggested severity (optional) |
| 62 | +We assign the final severity. Include a CVSS v3.1 vector only if you have one. |
| 63 | + |
| 64 | +### Suggested fix (optional) |
| 65 | +``` |
| 66 | + |
| 67 | +> The bar is a **reproducible PoC against a supported version, with a concrete impact that crosses a trust boundary we actually defend** (see scope below). Reports that are theoretical, auto-generated by a scanner or LLM, or that restate documented behavior will be closed without detailed review. |
| 68 | +
|
| 69 | +## Threat model & trust boundaries |
| 70 | + |
| 71 | +Understanding these saves everyone time — most of what we close as not-a-vulnerability falls inside one of them. |
| 72 | + |
| 73 | +**Loading tasks, datasets, and models you did not create is a trust decision.** Lighteval evaluates using community task definitions, prompt functions, and metrics pulled from the Hugging Face Hub or a local path you point it at, and can load model weights and custom code the same way `transformers`/`accelerate` do. That a maliciously crafted task, dataset, or model can execute code or otherwise act when you load and run it is the **documented risk of running untrusted content** — not a vulnerability in this library. Protect yourself by reviewing custom task/prompt code before running it and by pinning a specific, reviewed revision when loading from the Hub. |
| 74 | + |
| 75 | +We *will* treat as a vulnerability anything that breaks one of these protections, or that exposes data you did not intend to share — e.g. a model config's credentials or another user's data ending up in locally saved or Hub-pushed results. |
| 76 | + |
| 77 | +**CLI and developer utilities are operator tools.** Scripts you run yourself against inputs you chose (the `lighteval` CLI, examples, dev helpers) are not a library API attack surface. Behaviors such as a script reading or writing a file you pointed it at are within the operator's trust. |
| 78 | + |
| 79 | +## In scope |
| 80 | + |
| 81 | +We treat as vulnerabilities issues in the **published package code** — the library's own API surface — that an attacker can trigger without the victim having opted into a documented risk. For example: |
| 82 | + |
| 83 | +- code execution, memory corruption, or file access reachable through a normal API call on input that is **not** an untrusted task/model/dataset the user chose to load and run; |
| 84 | +- exposure or mishandling of credentials, tokens, or another user's data by the library — including data written to local output files or pushed to the Hugging Face Hub; |
| 85 | +- a control we advertise being bypassed (e.g. a pinned revision being ignored); |
| 86 | +- CI/CD or supply-chain issues in this repository. |
| 87 | + |
| 88 | +## Out of scope |
| 89 | + |
| 90 | +The following are **not** treated as vulnerabilities in `lighteval`. If your finding touches one of these, the report must explain why it is nonetheless a violation of a guarantee we make — otherwise it will be closed. |
| 91 | + |
| 92 | +- Issues that require loading and running an untrusted task, prompt function, metric, or model and amount to the documented run-time risk above. |
| 93 | +- Findings in `examples/`, documentation, tests, or other non-packaged reference material. |
| 94 | +- Local denial-of-service from feeding pathological input to a function on your own machine (high memory, slow parse, panic), absent a multi-tenant or remote-service impact. |
| 95 | +- Model behavior: jailbreaks, alignment failures, prompt injection, or harmful generations from the model being evaluated. Model weights are authored by their uploaders; report these to the model owner. |
| 96 | +- Vulnerabilities in third-party dependencies or inference backends we do not vendor (e.g. vLLM, TGI, LiteLLM, individual API providers) — report upstream; we'll bump once fixed. |
| 97 | +- Theoretical issues without a working proof of concept, and reports auto-generated from scanners or LLMs without a verified, reproducible chain. |
| 98 | +- Best-practice or hardening suggestions with no demonstrated impact. |
| 99 | + |
| 100 | +## Safe harbor |
| 101 | + |
| 102 | +Good-faith research that respects these guidelines, avoids privacy violations and service disruption, and gives us a reasonable disclosure window will not be pursued by us. Do not access data that isn't yours and do not run tests against Hugging Face production infrastructure. |
0 commit comments