Custom scanner plugin for AI-agent threats (prompt injection / MCP / skill manifests)

[eeee2345]

ASH ships scanners for code and IaC (bandit, semgrep, checkov, grype, cdk-nag, syft, etc.), but there is no detector today for adversarial content in AI-agent assets — prompt injection, tool poisoning, and MCP/skill manipulation that live in skill manifests, MCP tool descriptions, and agent prompt files rather than in source logic. As repos add agents and MCP servers, those files are a scan surface ASH doesn't cover yet.

This fits the existing plugin model cleanly, as an external custom scanner — no core changes:

• a class subclassing ScannerPluginBase[T] decorated with @ash_scanner_plugin (from automated_security_helper.plugins.decorators)
• implementing scan(self, target, target_type, ...), returning a SarifReport (or ScanResultsContainer) like the built-in scanners
• tool_type = ScannerToolType.CUSTOM, discovered via ash_plugin_modules in .ash/.ash.yaml

The detection engine would be pyatr (https://pypi.org/project/pyatr/) — the Python reference engine for Agent Threat Rules, an open MIT ruleset of deterministic regex patterns for agent attacks. It runs in-process (import pyatr, no subprocess), and each rule carries MITRE ATT&CK + ATLAS mappings, which map naturally to SARIF rule metadata and taxonomies. So findings would surface in the existing aggregated SARIF report with technique IDs attached.

Before writing anything, I wanted to check direction:

1. Is an agent/skill/MCP threat scanner in scope for ASH, or out of scope by design?
2. Would you prefer it as a standalone external plugin package (referenced from docs), or contributed into the repo as a bundled community plugin module like ash_ferret_plugins / ash_trivy_plugins?

Happy to follow whichever path you point to. Standard examples/ash_plugins_example shape, with tests.

Disclosure: I maintain ATR.

[rafaelpereyra]

Thanks @eeee2345 let me investigate a bit more into pyatr and I'll get back to you on this.

[nikmeiser]

@eeee2345 it looks like ATR/pyatr scans events rather than static code. How do you envisage this working?

[eeee2345]

@nikmeiser good distinction to draw, and it's worth being precise about it since it affects the integration shape.

pyatr's core type is called AgentEvent, but "event" here doesn't mean a runtime signal (no process/syscall/network monitoring). AgentEvent is just {content: str, event_type: str, fields: dict, metadata: dict} — a plain data container. ATREngine.evaluate() runs regex conditions against event.content, which is any string you hand it. There's no live-agent hook, no subprocess trace, nothing that requires an agent to actually be running.

For a static scanner like ASH, that maps directly: read the file (SKILL.md, an MCP tool-description JSON, an agent prompt template), wrap it as AgentEvent(content=file_bytes.decode(), event_type="skill_manifest"), call engine.evaluate(event). It's the same "read file → treat content as scannable text" flow ASH's other scanners already use for source code — pyatr doesn't need a running agent or an event stream to fire.

@rafaelpereyra — hope this saves you some digging. Concretely, the loader shape from the issue body still holds: ScannerPluginBase[T].scan() reads the target file, constructs an AgentEvent from its content, runs it through ATREngine in-process (import pyatr, no subprocess), and each match's rule metadata (MITRE ATT&CK/ATLAS mapping) drops into SarifReport rule properties. Happy to sketch the ~5-line adapter here if that helps evaluate scope before deciding whether this fits ASH's plugin model.

[nikmeiser]

@eeee2345 thank-you for this. Yes, a 5-line adapter sketch would be appreciated.

[eeee2345]

Here's the adapter. It drops into ASH's scanner-plugin model the same way CustomRegexScanner does in the dev guide — same scan() shape, just swapping the raw-regex loop for ATREngine.evaluate() so each hit carries the rule's id / title / severity / confidence instead of a bare pattern match.

The 5-line core:

from pyatr import ATREngine, AgentEvent

engine = ATREngine()
engine.load_default_rules()   # bundled ATR rule pack

def atr_hits(content: str):
    seen = {}
    for et in ("llm_input", "tool_call", "tool_response"):     # see note below on why 3
        for m in engine.evaluate(AgentEvent(content=content, event_type=et)):
            seen[m.rule_id] = m                                 # dedup across surfaces
    return list(seen.values())   # ATRMatch: rule_id, title, severity, confidence, matched_patterns

Wrapped as a ScannerPluginBase (mirroring your CustomRegexScanner):

from pathlib import Path
from pyatr import ATREngine, AgentEvent
from automated_security_helper.base.scanner_plugin import ScannerPluginBase

_ATR_SEV = {"critical": "CRITICAL", "high": "HIGH", "medium": "MEDIUM", "low": "LOW"}

class ATRScanner(ScannerPluginBase):
    def __init__(self, config):
        super().__init__(config)
        self.engine = ATREngine()
        self.engine.load_default_rules()

    def scan(self, target_path: str) -> dict:
        findings = []
        for fp in self._get_files(target_path):          # ASH's own file walk
            content = Path(fp).read_text(errors="ignore")
            seen = {}
            for et in ("llm_input", "tool_call", "tool_response"):
                for m in self.engine.evaluate(AgentEvent(content=content, event_type=et)):
                    seen[m.rule_id] = m
            for m in seen.values():
                findings.append({
                    "file": fp, "rule_id": m.rule_id, "title": m.title,
                    "severity": _ATR_SEV.get(m.severity, "MEDIUM"),
                    "confidence": m.confidence,
                    "match": m.matched_patterns[0] if m.matched_patterns else "",
                })
        return {"findings": findings}

One correction worth flagging, because it changes recall a lot: in my earlier reply I wrapped the file as event_type="skill_manifest". That runs, but it undercounts. pyatr routes content to a rule field by event_type (llm_input->user_input, tool_call->tool_args, tool_response->tool_response, ...). An unmapped type like skill_manifest only satisfies rules whose condition field is literally content, so every user_input / tool_args / tool_response-keyed rule stays silent. A static artifact has no runtime "direction", so the right move is to feed it under each input-bearing type and union the hits. Measured on one adversarial manifest against the bundled pack: skill_manifest -> 8 matches, the 3-type union -> 22. Same file, 2.75x the coverage, with no extra FP surface beyond what those rules already gate on.

(evaluate() skips draft/deprecated rules by design, and severity/confidence come straight off each rule, so they map cleanly onto ASH's SARIF levels.)

If this shape looks right, I'm happy to open a PR with it as a first-class plugin — a config class, pyatr as an optional dependency, and a couple of fixture tests (one malicious manifest, one clean) to pin the recall / no-FP behavior. Just say the word and I'll put it up for review.