Writing GuardDog Rules

This document explains how to write effective detection rules for GuardDog that follow our risk-based detection model.

Core Philosophy

GuardDog uses a two-layer detection model where capabilities and threats work together to identify actual risks:

Capabilities: "CAN DO"

Capabilities detect what code can do - the technical ability to perform an action.

Focus on: Function calls to known methods that enable an action

Example: .readFile() or .readFileSync() shows a Node.js app can read files
Example: requests.get() shows a Python app can make HTTP requests
Example: subprocess.Popen() shows code can spawn processes

Avoid:

❌ Relying solely on imports (e.g., import fs alone is too broad and leads to high false positive rates)
❌ Detecting threat indicators (that's what threat rules do)

Good Capability Examples:

# ✅ capability.filesystem.read
$py_read = /\.read\s*\(/          # Detects .read() method
$py_readfile = /\.readFile\s*\(/  # Detects .readFile() method

# ✅ capability.network.outbound
$py_requests = /requests\.get\s*\(/  # Detects HTTP GET capability
$js_fetch = /\bfetch\s*\(/           # Detects fetch() API

Bad Capability Examples:

# ❌ Too broad - just importing doesn't mean it's used
$py_import_fs = /import\s+os/

# ❌ This is a threat indicator, not a capability
$sensitive_file = "/etc/passwd"

Threats: "SUSPICIOUS"

Threats detect suspicious indicators - patterns that suggest malicious intent.

Focus on: Clear threat indicators that attackers target

Example: /etc/passwd is a file attackers commonly target
Example: bit.ly domains are commonly used for malicious redirection
Example: base64.decode() + exec() is an obfuscation technique used by malware

Avoid:

❌ Relying on function calls alone (that's what capability rules do)
❌ Generic patterns that appear in legitimate code

Good Threat Examples:

# ✅ threat.filesystem.read (targets sensitive files)
$passwd = "/etc/passwd"
$shadow = "/etc/shadow"
$ssh_key = ".ssh/id_rsa"

# ✅ threat.network.outbound (suspicious domains)
$pastebin = /pastebin\.com/
$discord_webhook = /discord\.com\/api\/webhooks/

# ✅ threat.runtime.obfuscation (evasion technique)
# Requires BOTH decode AND exec together
condition: ($base64_decode and $exec)

Bad Threat Examples:

# ❌ Just a capability, not suspicious on its own
$file_read = /\.read\s*\(/

# ❌ Too generic - common in legitimate code
$env_read = "process.env"

Risks: Capability + Threat

Risks form when a capability and threat are found in the same file with matching categories.

Example Risk Formation:

capability.filesystem.read (can read files)
    +
threat.filesystem.read (reads /etc/passwd)
    =
risk.filesystem.read (credential-access)

Why This Works:

Capability alone isn't malicious (file system libraries should read files)
Threat indicator alone might be a false positive (test fixtures, documentation)
Capability + Threat together indicates actual risk (code that can and will do something malicious)

Rule Structure

Identifies Field Format

{type}.{category}[.{detail}]

Type: capability or threat

Category:

network - Network operations
filesystem - File system operations
process - Process/command execution
runtime - Runtime operations (no capability needed)
system - System information APIs
metadata - Package metadata indicators (no capability needed)

Detail (optional specificity):

Examples: outbound, read, write, obfuscation, collection, typosquatting

Auto-Risk Categories:

threat.runtime.* and threat.metadata.* rules automatically form risks without needing a capability. Use these for:

Runtime: Obfuscation techniques, install hooks, standalone suspicious patterns
Metadata: Supply chain indicators like typosquatting, compromised maintainer emails, repository tampering

Risk Formation Rules

Capabilities and threats form risks when:

Same category: Both must have matching category
- ✅ capability.network + threat.network → risk.network
- ✅ capability.process.hooks + threat.process.hooks → risk.process.hooks
- ❌ capability.filesystem + threat.network → NO RISK (categories don't match)
- ❌ capability.process.hooks + threat.network.outbound → NO RISK (categories don't match)
Detail compatibility:
- General matches specific: threat.network + capability.network.outbound ✅
- Exact match: threat.network.outbound + capability.network.outbound ✅
- Conflict: threat.network.outbound + capability.network.inbound ❌
Exception: threat.runtime.* and threat.metadata.* rules skip this - they auto-form risks

CRITICAL: Risks only form when categories match. A capability.process.* rule can only form risks with threat.process.* rules, never with threat.network.* or other categories. This is by design to ensure accurate risk assessment.

Metadata Schema

Required Fields

identifies: "capability.{category}[.{detail}]"  # or "threat.{category}[.{detail}]"
severity: "low" | "medium" | "high"
description: "Human-readable description"

For threat rules only:

mitre_tactics: "tactic"  # Single MITRE ATT&CK tactic

Optional Fields (default to `medium`)

specificity: "low" | "medium" | "high"      # Pattern specificity
sophistication: "low" | "medium" | "high"    # Technique advancement level

YARA-Specific Fields

max_hits: 3                                  # Limit findings per file
path_include: "*.py,*.js,*/package.json"    # File patterns to scan (see below)

path_include Pattern Matching:

GuardDog uses Python's fnmatch to match file paths. Patterns are matched against the relative path from the scan root:

*.py - Matches .py files at any depth (e.g., foo.py, src/foo.py, a/b/c/foo.py)
setup.py - Matches only in root directory (not src/setup.py)
*/setup.py - Matches in subdirectories only, not root (e.g., src/setup.py, a/b/setup.py)
setup.py,*/setup.py - Matches at any depth including root

Field Definitions

Severity (impact of finding):

low: Minor concern, common patterns
medium: Moderately suspicious
high: Clearly malicious or high impact

Specificity (how specific the pattern is to malware vs legitimate code):

low: Generic patterns that appear in legitimate code frequently
medium: Reasonably specific, some false positives expected
high: Very specific to malicious behavior, minimal false positives

Sophistication (technique advancement level):

low: Basic/common techniques used by script kiddies
medium: Moderate evasion or complexity
high: Advanced techniques, APT-level tradecraft

MITRE Tactics (attack stage - single tactic only):

Early stage: initial-access, execution, reconnaissance, resource-development
Mid stage: defense-evasion, credential-access, persistence, privilege-escalation, discovery
Late stage: command-and-control, exfiltration, impact, lateral-movement, collection

Note: Only threats have MITRE tactics. Choose the primary tactic that best represents the malicious intent.

Rule Examples

YARA Capability Rule

rule capability_filesystem_read
{
    meta:
        author = "GuardDog Team, Datadog"
        description = "Detects file reading capabilities"
        identifies = "capability.filesystem.read"
        severity = "low"
        specificity = "low"
        sophistication = "low"

        max_hits = 1
        path_include = "*.py,*.pyx,*.pyi,*.js,*.ts,*.jsx,*.tsx,*.mjs,*.cjs"

    strings:
        // Python - file reading methods
        $py_read = /\.read\s*\(/
        $py_readfile = /\.readFile\s*\(/
        $py_open = /\bopen\s*\(/

        // JavaScript - file reading methods
        $js_readfile = /\.readFile(Sync)?\s*\(/
        $js_read = /\.read\s*\(/

    condition:
        any of them
}

YARA Threat Rule

rule threat_filesystem_read
{
    meta:
        author = "GuardDog Team, Datadog"
        description = "Detects access to sensitive files (credentials, configs, keys)"
        identifies = "threat.filesystem.read"
        severity = "high"
        mitre_tactics = "credential-access"
        specificity = "high"
        sophistication = "low"

        max_hits = 5

    strings:
        // Sensitive system files
        $passwd = "/etc/passwd"
        $shadow = "/etc/shadow"

        // SSH keys - use regex to match .ssh structure
        $ssh_rsa = ".ssh/id_rsa"
        $ssh_ed25519 = ".ssh/id_ed25519"

        // Cloud credentials
        $aws = ".aws/credentials"
        $gcp = "gcloud/credentials"

    condition:
        any of them
}

Pattern Writing Best Practices

1. Use Word Boundaries

Avoid matching partial words and ensure proper spacing:

// ❌ Bad - matches "myrequests.get()" or "requests.getall()"
$bad = /requests\.get/

// ✅ Good - only matches "requests.get()" with word boundaries
$good = /\brequests\.get\s*\(/

2. Match Method Calls, Not Object Names

The object can be named anything:

// ❌ Bad - only works if object is named "dns"
$bad = "dns.lookup("

// ✅ Good - matches .lookup() on any object
$good = /\.lookup\s*\(/

3. Require Context for Strings

Match patterns within quoted strings when appropriate:

// ❌ Bad - matches "curl" anywhere, even in variable names
$bad = "curl"

// ✅ Good - matches "curl" only within strings
$good = /['"][^'"]*\bcurl\b[^'"]*['"]/

4. Detect Patterns in Specific Context

Use private rules to establish context and avoid false positives:

// ❌ Bad - matches LOLBAS anywhere (shebangs, READMEs, code files)
rule threat_process_hooks {
    strings: $node = /\bnode\b/
    condition: $node
}

// ✅ Good - only matches LOLBAS within install hooks
include "hooks.meta"
include "lolbas-proc.meta"
rule threat_process_hooks {
    condition: (has_npm_hook or has_python_hook) and lolbas_proc
}

Key principle: Establish context with private rules, then detect threats within that context.

5. Use Private Rules and Meta Files for Reusable Logic (DRY Principle)

Don't Repeat Yourself (DRY): Extract common patterns into reusable private rules in .meta files.

Create a .meta file for shared patterns:

// lolbas-net.meta
rule lolbas_net
{
    strings:
        $curl = /\bcurl\b/ nocase
        $wget = /\bwget\b/ nocase
        $nc = /\bnc\b/ nocase
        $netcat = /\bnetcat\b/ nocase
    condition:
        any of them
}

Use in multiple rules:

// capability-network-lolbas.yar
include "lolbas-net.meta"

rule capability_network_lolbas
{
    ...
    condition:
        has_process_spawn and lolbas_net
}

// threat-process-hooks.yar
include "lolbas-net.meta"
include "hooks.meta"

rule threat_process_hooks
{
    ...
    condition:
        (has_npm_hook or has_python_hook) and lolbas_net
}

Benefits:

Single source of truth for common patterns
Easier maintenance (update once, applies everywhere)
Consistent detection logic across rules
Prevents drift between similar patterns

When to use .meta files:

Patterns used in 2+ rules
Common detection building blocks (LOLBAS, hooks, spawning)
Shared context validators (e.g., detecting if we're in a hook)

Testing Rules

Unit Testing

The YARA test harness lives in tests/analyzer/sourcecode/test_sourcecode_yara.py. It does three things for every rule: compiles it, asserts it matches each positive test file, and asserts it does not match the benign files. Tests are matched to a rule by filename prefix, so test files are flat (no per-rule subdirectory) and named after the rule id.

The YARA rule's internal name must be the rule id with hyphens replaced by underscores. The no-false-positive check filters matches to that exact name, so a mismatch silently drops coverage:

// file: capability-network-outbound.yar
rule capability_network_outbound { ... }

Positive tests — add one file per language the rule should flag. The harness asserts the rule matches it:

tests/analyzer/sourcecode/
  capability-network-outbound.py    # requests.get("https://example.com")
  capability-network-outbound.js    # fetch("https://example.com")
  capability-network-outbound.go    # http.Get("https://example.com")

Negative tests — add legitimate code that must NOT trigger the rule under benign/. Every new or changed rule should have one to lock in against false positives:

tests/analyzer/sourcecode/benign/
  capability-network-outbound.py    # import requests   (declared but never called)

Run the suite:

make test-yara-rules
# or just one rule:
uv run pytest tests/analyzer/sourcecode -k capability-network-outbound

Manual Testing

# Test on a specific package
uv run guarddog pypi scan package-name --rules my-rule

# Test with specific rules only
uv run guarddog pypi scan package-name --rules rule1 --rules rule2

# Exclude specific rules
uv run guarddog pypi scan package-name --exclude-rules my-rule

# Test on local directory
uv run guarddog pypi scan /path/to/package/

# Output JSON for analysis
uv run guarddog pypi scan package-name --output-format json

Validation Checklist

Before submitting a rule:

Advanced Patterns

Install Hooks: Capability vs Threat

Install hooks are a special case that demonstrates the capability/threat separation:

✅ Correct Approach:

Install hooks themselves are a capability, not a threat:

// capability-process-hooks.yar
include "hooks.meta"

rule capability_process_hooks
{
    meta:
        identifies = "capability.process.hooks"
        severity = "low"
    condition:
        has_npm_hook or has_python_hook
}

Why? Many legitimate packages use install hooks for:

Compiling native modules (node-gyp rebuild)
Building assets (npm run build)
Setting up git hooks (husky install)
Post-install messages

The threat is what LOLBAS tools are being used in the hooks:

// threat-process-hooks.yar
include "hooks.meta"
include "lolbas-net.meta"

rule threat_process_hooks
{
    meta:
        identifies = "threat.process.hooks"
        severity = "medium"
        mitre_tactics = "execution"
    condition:
        (has_npm_hook or has_python_hook) and lolbas_net
}

Risk Formation:

capability.process.hooks (has install hook)
    +
threat.process.hooks (uses curl/wget)
    =
risk.process.hooks (malicious install hook)

Key Insight: Install hooks are equivalent to subprocess.call() - they're a process spawning capability. The same threats that apply to capability.process.spawn should also apply to capability.process.hooks.

LOLBAS Detection Pattern

Living Off The Land Binaries and Scripts (LOLBAS) are legitimate tools used maliciously:

Split by Purpose:

lolbas-proc.meta - Execution tools (bash, python, node, perl, ruby)
lolbas-net.meta - Network tools (curl, wget, nc, netcat)

Why split? YARA has limitations using multiple private rules from the same include. Splitting allows flexible composition:

// Use execution tools only
include "lolbas-proc.meta"

// Use network tools only
include "lolbas-net.meta"

// Use both if needed
include "lolbas-proc.meta"
include "lolbas-net.meta"

Pattern:

// lolbas-net.meta
rule lolbas_net
{
    strings:
        $curl = /\bcurl\b/ nocase
        $wget = /\bwget\b/ nocase
        $nc = /\bnc\b/ nocase
        $netcat = /\bnetcat\b/ nocase
        $telnet = /\btelnet\b/ nocase
    condition:
        any of them
}

Use word boundaries (\b) to avoid false positives from substring matches.

Rule Format

Source-code rules are written in YARA (.yar):

Language-agnostic: every YARA rule is loaded regardless of ecosystem
Binary-safe: can scan any file type
Pattern matching: great for byte patterns, strings, and regex
Best for: cross-language patterns, obfuscation detection, binary analysis

Shared private rules go in .meta files (see the DRY section above).

File Organization

Place rules under guarddog/analyzer/sourcecode/:

guarddog/analyzer/sourcecode/
  # Capability rules
  capability-filesystem-read.yar
  capability-network-outbound.yar
  capability-network-lolbas.yar
  capability-process-hooks.yar
  capability-process-spawn.yar

  # Threat rules
  threat-filesystem-read.yar
  threat-network-outbound-shady-links.yar
  threat-process-hooks.yar
  threat-runtime-obfuscation-base64exec.yar
  threat-runtime-system-info.yar
  threat-runtime-environment-read.yar

  # Meta files (reusable private rules)
  hooks.meta
  lolbas-net.meta
  lolbas-proc.meta

Naming conventions:

Rule files (.yar): {type}-{category}-{detail}.yar

Examples:
- capability-network-outbound.yar
- threat-filesystem-read.yar
- threat-runtime-obfuscation-base64exec.yar

Meta files (.meta): {pattern-name}.meta

Examples:
- hooks.meta - Install hook detection patterns
- lolbas-net.meta - LOLBAS network tools (curl, wget, nc)
- lolbas-proc.meta - LOLBAS execution tools (bash, python, node)

Meta file guidelines:

Use .meta extension for files containing only private rules
Name describes what patterns it contains
Single responsibility - one concern per meta file
Split related patterns if they can't be used together (e.g., lolbas-proc.meta and lolbas-net.meta instead of combined lolbas.meta)

Key Takeaways

Rule Writing Principles

Capabilities detect CAN DO - Focus on function calls, not imports
Threats detect SUSPICIOUS - Focus on attacker indicators, not generic patterns
Categories must match - capability.process.* + threat.process.* = risk
Runtime and metadata bypass capabilities - threat.runtime.* and threat.metadata.* auto-form risks
Context matters - Detect patterns where they matter (hooks + LOLBAS, not just LOLBAS anywhere)

DRY Principle

Extract common patterns to .meta files
Split incompatible private rules (lolbas-proc.meta vs lolbas-net.meta)
Single source of truth prevents pattern drift

Install Hooks Pattern

Hooks are capabilities (like subprocess.call())
LOLBAS in hooks are threats (process category)
Same threats apply to both capability.process.spawn and capability.process.hooks

Avoid False Positives

Use word boundaries (\b)
Require context (private rules for "where" detection)
Test against shebangs, READMEs, legitimate code

Questions?

If you're unsure whether a pattern should be a capability or threat, ask:

Does this pattern show what code CAN do? → Capability
Does this pattern show suspicious/malicious indicators? → Threat
Does this pattern work standalone without needing other capabilities? → Runtime threat
Will it form risks with the right category? → Check category matching

When in doubt, err on the side of being more specific. It's better to have separate capability and threat rules that form risks correctly than to have overly broad rules with high false positive rates.

Uh oh!

FilesExpand file tree

WRITING_RULES.md

Latest commit

History