Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -123,6 +123,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

### Fixed

- **security:** flag risky import-only pickle references for `torch.jit`, `torch._dynamo`, `torch._inductor`, `torch.compile`, `torch.storage._load_from_bytes`, `numpy.f2py`, and `numpy.distutils` while preserving safe state-dict reconstruction paths
- **security:** add low-severity pickle structural tamper findings for duplicate or misplaced `PROTO` opcodes while avoiding benign binary-tail false positives
Comment on lines 124 to 127
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Duplicate ### Fixed section violates Keep a Changelog format.

There are two ### Fixed sections under [Unreleased] (lines 82 and 124). The Keep a Changelog format requires each section type to appear only once per version. Merge this section's entries into the existing ### Fixed section at line 82.

🧰 Tools
🪛 markdownlint-cli2 (0.21.0)

[warning] 124-124: Multiple headings with the same content

(MD024, no-duplicate-heading)

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@CHANGELOG.md` around lines 124 - 127, The duplicate "### Fixed" header and
its two bullet entries should be merged into the existing "### Fixed" section
(the other "### Fixed" under [Unreleased]); move the two bullets about security
(the torch/jit/numpy import-only pickle references entry and the pickle
structural tamper findings entry) into the existing "### Fixed" list and then
remove this duplicate "### Fixed" header and its empty block so each section
type appears only once under [Unreleased]; preserve the exact bullet text and
ordering when appending to the original "### Fixed" section.

- **security:** scan OCI layer members based on registered file extensions so embedded ONNX, Keras H5, and other real-path scanners are no longer skipped inside tar layers
- **security:** resolve bare-module TorchServe handler references like `custom_handler` to concrete archive members so malicious handler source is no longer skipped by static analysis
Expand Down
38 changes: 36 additions & 2 deletions modelaudit/config/explanations.py
Original file line number Diff line number Diff line change
Expand Up @@ -99,6 +99,34 @@
"The 'dill' module extends pickle's capabilities to serialize almost any Python object, including lambda "
"functions and code objects. This significantly increases the attack surface for code execution."
),
"torch.jit": (
"The 'torch.jit' module can load and execute serialized TorchScript artifacts. In untrusted model files, "
"this can introduce unsafe runtime behavior and should be treated as a high-risk import surface."
),
"torch._dynamo": (
"The 'torch._dynamo' internals drive dynamic graph capture and compilation. Importing these internals from "
"untrusted pickle payloads is suspicious because they enable advanced runtime execution pathways."
),
"torch._inductor": (
"The 'torch._inductor' compiler backend can generate and execute optimized kernels at runtime. In model "
"artifacts, this is a risky import surface that should be reviewed as potentially unsafe."
),
"torch.compile": (
"The 'torch.compile' API triggers runtime compilation and execution pipelines. In untrusted serialized "
"payloads, this can be used to reach risky execution paths and should be flagged."
),
"torch.storage._load_from_bytes": (
"The 'torch.storage._load_from_bytes' function reconstructs storages from raw bytes and can be abused in "
"malicious pickle chains. References from untrusted payloads should be treated as dangerous."
),
"numpy.f2py": (
"The 'numpy.f2py' toolchain bridges Python and compiled Fortran extensions. References in untrusted "
"pickles are risky because they can touch native-code compilation/loading paths."
),
"numpy.distutils": (
"The 'numpy.distutils' build utilities are tied to extension module compilation and setup workflows. "
"Importing them from serialized model payloads is suspicious and may indicate unsafe behavior."
),
"numpy.load": (
"The 'numpy.load' function can recursively deserialize object arrays via pickle support, enabling "
"second-stage payload loading from attacker-controlled files."
Expand Down Expand Up @@ -428,8 +456,14 @@ def get_import_explanation(module_name: str) -> str | None:
"""Get explanation for a dangerous import/module."""
if module_name in DANGEROUS_IMPORTS:
return get_explanation("import", module_name)
# Handle module.function format (e.g., "os.system")
base_module = module_name.split(".")[0]

parts = module_name.split(".")
for i in range(len(parts) - 1, 0, -1):
parent = ".".join(parts[:i])
if parent in DANGEROUS_IMPORTS:
return get_explanation("import", parent)

base_module = parts[0]
return get_explanation("import", base_module)


Expand Down
58 changes: 57 additions & 1 deletion modelaudit/scanners/pickle_scanner.py
Original file line number Diff line number Diff line change
Expand Up @@ -655,6 +655,31 @@ def _compute_pickle_length(path: str) -> int:
"glob",
}

# Risky ML-specific import surfaces that must be flagged even when they appear
# as import-only GLOBAL/STACK_GLOBAL references (without immediate REDUCE).
RISKY_ML_MODULE_PREFIXES: tuple[str, ...] = (
"torch.jit",
"torch._dynamo",
"torch._inductor",
"numpy.f2py",
"numpy.distutils",
)

RISKY_ML_EXACT_REFS: set[tuple[str, str]] = {
("torch", "compile"),
("torch.storage", "_load_from_bytes"),
}


def _split_parent_child_ref(prefix: str) -> tuple[str, str]:
parent, _separator, child = prefix.rpartition(".")
return parent, child


RISKY_ML_PARENT_CHILD_REFS: frozenset[tuple[str, str]] = frozenset(
_split_parent_child_ref(prefix) for prefix in RISKY_ML_MODULE_PREFIXES
)
Comment on lines +658 to +681
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical

🧩 Analysis chain

🌐 Web query:

In CPython 3.10-3.13, does pickle.Unpickler.find_class()resolve dottednamevalues (for exampleOrderedDict.fromkeys) by traversing attributes after importing the module?

💡 Result:

Yes—for the C-accelerated pickle.Unpickler used by CPython 3.10–3.13, find_class() imports the module and then (for pickle protocols >= 4) resolves a dotted name by splitting on '.' and repeatedly looking up attributes (i.e., module → OrderedDictfromkeys). (raw.githubusercontent.com)

For protocols < 4, it does not do dotted traversal; it only looks up a single attribute named exactly name. (raw.githubusercontent.com)

Citations:


🏁 Script executed:

cd modelaudit && wc -l scanners/pickle_scanner.py

Repository: promptfoo/modelaudit

Length of output: 95


🏁 Script executed:

cd modelaudit && sed -n '658,681p' scanners/pickle_scanner.py

Repository: promptfoo/modelaudit

Length of output: 776


🏁 Script executed:

cd modelaudit && sed -n '1933,1950p' scanners/pickle_scanner.py

Repository: promptfoo/modelaudit

Length of output: 772


🏁 Script executed:

cd modelaudit && sed -n '1940,1960p' scanners/pickle_scanner.py

Repository: promptfoo/modelaudit

Length of output: 791


🏁 Script executed:

cd modelaudit && grep -n "pickle.HIGHEST_PROTOCOL\|protocols\|protocol" scanners/pickle_scanner.py | head -30

Repository: promptfoo/modelaudit

Length of output: 2266


🏁 Script executed:

cd modelaudit && sed -n '2000,2010p' scanners/pickle_scanner.py

Repository: promptfoo/modelaudit

Length of output: 615


🏁 Script executed:

cd modelaudit && find . -name "*test*pickle*" -type f 2>/dev/null | head -10

Repository: promptfoo/modelaudit

Length of output: 46


🏁 Script executed:

cd modelaudit && rg "_is_risky_ml_import" --type py -A 5 -B 5

Repository: promptfoo/modelaudit

Length of output: 2216


🏁 Script executed:

cd modelaudit && rg "torch.*compile\.__globals__\|torch\.storage\._load_from_bytes\.__" tests/ --type py

Repository: promptfoo/modelaudit

Length of output: 110


🏁 Script executed:

cd modelaudit && grep -r "compile\.__globals__\|_load_from_bytes\.__" tests/ --include="*.py"

Repository: promptfoo/modelaudit

Length of output: 103


🏁 Script executed:

git ls-files | grep -i test | grep -i pickle | head -20

Repository: promptfoo/modelaudit

Length of output: 1047


🏁 Script executed:

git ls-files | grep -i test | head -30

Repository: promptfoo/modelaudit

Length of output: 1512


🏁 Script executed:

cd modelaudit && rg "torch.*compile" --type py | grep -i test | head -10

Repository: promptfoo/modelaudit

Length of output: 46


🏁 Script executed:

find . -path ./node_modules -prune -o -name "*test*pickle*scanner*" -type f -print 2>/dev/null

Repository: promptfoo/modelaudit

Length of output: 103


🏁 Script executed:

cd modelaudit && find .. -name "*test*" -type f -path "*/scanners/*" 2>/dev/null | head -10

Repository: promptfoo/modelaudit

Length of output: 492


🏁 Script executed:

git ls-files | grep -E "test.*pickle.*scanner|scanner.*test.*pickle" | head -10

Repository: promptfoo/modelaudit

Length of output: 101


🏁 Script executed:

cd modelaudit && git ls-files | grep test | grep scanner

Repository: promptfoo/modelaudit

Length of output: 46


🏁 Script executed:

wc -l tests/scanners/test_pickle_scanner.py

Repository: promptfoo/modelaudit

Length of output: 106


🏁 Script executed:

grep -n "torch.*compile\|_is_risky_ml_import\|RISKY_ML" tests/scanners/test_pickle_scanner.py | head -30

Repository: promptfoo/modelaudit

Length of output: 352


🏁 Script executed:

grep -n "def test.*risky\|def test.*ml" tests/scanners/test_pickle_scanner.py

Repository: promptfoo/modelaudit

Length of output: 972


🏁 Script executed:

sed -n '1,100p' tests/scanners/test_pickle_scanner.py

Repository: promptfoo/modelaudit

Length of output: 3367


🏁 Script executed:

sed -n '1603,1643p' tests/scanners/test_pickle_scanner.py

Repository: promptfoo/modelaudit

Length of output: 2057


🏁 Script executed:

sed -n '1750,1779p' tests/scanners/test_pickle_scanner.py

Repository: promptfoo/modelaudit

Length of output: 1190


🏁 Script executed:

sed -n '1644,1670p' tests/scanners/test_pickle_scanner.py

Repository: promptfoo/modelaudit

Length of output: 1248


🏁 Script executed:

sed -n '1590,1610p' tests/scanners/test_pickle_scanner.py

Repository: promptfoo/modelaudit

Length of output: 944


🏁 Script executed:

grep -n "compile.__\|__globals__\|__code__" tests/scanners/test_pickle_scanner.py

Repository: promptfoo/modelaudit

Length of output: 46


🏁 Script executed:

rg "torch.*compile.__" --type py

Repository: promptfoo/modelaudit

Length of output: 46


🏁 Script executed:

sed -n '2732,2760p' modelaudit/scanners/pickle_scanner.py

Repository: promptfoo/modelaudit

Length of output: 1430


🏁 Script executed:

grep -n "def is_dangerous_reduce_pattern\|_is_dangerous_ref" modelaudit/scanners/pickle_scanner.py | head -20

Repository: promptfoo/modelaudit

Length of output: 386


🏁 Script executed:

sed -n '2643,2660p' modelaudit/scanners/pickle_scanner.py

Repository: promptfoo/modelaudit

Length of output: 768


Treat exact risky refs as dotted prefixes—fix bypass via torch.compile.__globals__ and similar nested attributes.

The current _is_risky_ml_import() implementation only matches exact (module, function) tuples and fails to catch dotted children of exact refs. For example, a pickle with GLOBAL "torch" "compile.__globals__" would not be flagged because the loop checks ("torch", "compile.__globals__") against RISKY_ML_EXACT_REFS, which expects only ("torch", "compile"). Since CPython 3.10–3.13 resolves dotted names via attribute traversal for protocols ≥ 4, this is a real security bypass.

Add a check before the tuple split to test whether any prefix of the full dotted reference matches an exact ref:

+RISKY_ML_EXACT_FULL_REFS: frozenset[str] = frozenset(
+    f"{module}.{name}" for module, name in RISKY_ML_EXACT_REFS
+)
+
 def _is_risky_ml_import(mod: str, func: str) -> bool:
     """Return True when module/function matches risky ML import policy."""
     full_ref = f"{mod}.{func}" if func else mod
     parts = full_ref.split(".")
 
+    for i in range(1, len(parts) + 1):
+        candidate = ".".join(parts[:i])
+        if candidate in RISKY_ML_EXACT_FULL_REFS:
+            return True
+
     for i in range(1, len(parts)):
         candidate_mod = ".".join(parts[:i])
         candidate_func = ".".join(parts[i:])
         if (candidate_mod, candidate_func) in RISKY_ML_EXACT_REFS:
             return True

Add a test case for ("torch", "compile.__globals__") and ("torch.storage", "_load_from_bytes.__code__") to prevent regression.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@modelaudit/scanners/pickle_scanner.py` around lines 658 - 681, The
_is_risky_ml_import logic must treat exact risky refs as dotted prefixes: update
the function that checks RISKY_ML_EXACT_REFS (the code path that receives module
and name like ("torch","compile.__globals__")) to first test whether any leading
dotted prefix of the full name (e.g. "compile" from "compile.__globals__")
matches an entry in RISKY_ML_EXACT_REFS before splitting into parent/child; in
practice modify _is_risky_ml_import to iterate name.split(".") prefixes (or
progressively rpartition on ".") and check (module, prefix) against
RISKY_ML_EXACT_REFS, falling back to the existing RISKY_ML_PARENT_CHILD_REFS and
RISKY_ML_MODULE_PREFIXES checks; add unit tests asserting that
("torch","compile.__globals__") and
("torch.storage","_load_from_bytes.__code__") are flagged to prevent
regressions.



def _is_dangerous_module(mod: str) -> bool:
"""Check if module is in ALWAYS_DANGEROUS_MODULES (exact or prefix match).
Expand Down Expand Up @@ -1905,6 +1930,26 @@ def _is_safe_ml_global(mod: str, func: str) -> bool:
return False


def _is_risky_ml_import(mod: str, func: str) -> bool:
"""Return True when module/function matches risky ML import policy."""
full_ref = f"{mod}.{func}" if func else mod
parts = full_ref.split(".")

for i in range(1, len(parts)):
candidate_mod = ".".join(parts[:i])
candidate_func = ".".join(parts[i:])
if (candidate_mod, candidate_func) in RISKY_ML_EXACT_REFS:
return True
if (candidate_mod, candidate_func) in RISKY_ML_PARENT_CHILD_REFS:
return True
if any(
candidate_mod == prefix or candidate_mod.startswith(f"{prefix}.") for prefix in RISKY_ML_MODULE_PREFIXES
):
return True

return False


def _is_copyreg_extension_ref(mod: str) -> bool:
"""Return True when a reference came from an EXT opcode extension lookup."""
return mod == COPYREG_EXTENSION_MODULE
Expand Down Expand Up @@ -1954,6 +1999,13 @@ def _is_actually_dangerous_global(mod: str, func: str, ml_context: dict) -> bool
logger.warning(f"Extension-registry callable detected via EXT opcode: {full_ref}")
return True

# STEP 0.5: Risky ML imports should be flagged even in import-only payloads.
# These are intentionally separate from the broad ML safe allowlist because
# they map to runtime loading/compilation pathways with elevated risk.
if _is_risky_ml_import(mod, func):
logger.warning(f"Risky ML import detected: {full_ref}")
return True
Comment on lines +2002 to +2007
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Mirror the risky-ML short-circuit inside is_dangerous_reduce_pattern().

Line 2748 still returns early on _is_safe_ml_global(), so GLOBAL torch jit + REDUCE is missed here because torch.jit remains allowlisted. That leaves a misleading passed Reduce Pattern Analysis check even though the per-opcode REDUCE path now flags the same ref as risky.

🔧 Proposed fix
     def _is_dangerous_ref(mod: str, func: str) -> bool:
         """Check if a module.function reference is dangerous enough to flag."""
         if _is_copyreg_extension_ref(mod):
             return True
+        if _is_risky_ml_import(mod, func):
+            return True
 
         full_ref = f"{mod}.{func}"

Based on learnings, "Preserve or strengthen security detections; test both benign and malicious samples when adding scanner/feature changes".

Also applies to: 2734-2751

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@modelaudit/scanners/pickle_scanner.py` around lines 2002 - 2007,
is_dangerous_reduce_pattern() currently short-circuits on _is_safe_ml_global(),
letting allowed globals like torch.jit bypass REDUCE checks; replicate the
risky-ML short-circuit used elsewhere by calling _is_risky_ml_import(mod, func)
before the _is_safe_ml_global() early return and, if it returns True, log the
risky import (use the same logger.warning message format) and return True so
GLOBAL torch jit + REDUCE is flagged; update the block around
is_dangerous_reduce_pattern() (the logic handling mod/func and the early-return)
to mirror the treatment in the import-only path.


# STEP 1: ALWAYS flag dangerous functions first (no exceptions, no allowlist override)
# This MUST come before the ML_SAFE_GLOBALS check to prevent bypass attacks
# where an attacker places dangerous functions (e.g., operator.attrgetter) in a
Expand Down Expand Up @@ -2588,6 +2640,10 @@ def is_suspicious_global(mod: str, func: str) -> bool:
First checks against ML_SAFE_GLOBALS allowlist to reduce false positives
for legitimate ML framework operations.
"""
# STEP 0: Always flag risky ML imports before any allowlist checks.
if _is_risky_ml_import(mod, func):
return True

# STEP 1: Check ML_SAFE_GLOBALS allowlist first
# If the module.function is in the safe list, it's not suspicious
if mod in ML_SAFE_GLOBALS:
Expand Down Expand Up @@ -5055,7 +5111,7 @@ def get_depth(x):
0,
),
},
why=get_import_explanation(module_name)
why=get_import_explanation(f"{module_name}.{func_name}")
if module_name
else "A dangerous pattern was detected that could execute arbitrary code during unpickling.",
)
Expand Down
Loading
Loading