docs: add blocklist documentation to docs-site

Copilot · Mossaka · Copilot · commit a4217aba83e1 · 2025-12-19T22:22:42.000Z
Co-authored-by: Mossaka &lt;5447827+Mossaka@users.noreply.github.com&gt;
diff --git a/docs-site/src/content/docs/guides/domain-filtering.md b/docs-site/src/content/docs/guides/domain-filtering.md
@@ -0,0 +1,263 @@
+---
+title: Domain Filtering
+description: Control network access with allowlists, blocklists, and wildcard patterns.
+---
+
+Control which domains your AI agents can access using allowlists and blocklists. This guide covers all domain filtering options including wildcard patterns and file-based configuration.
+
+## How domain matching works
+
+Domains automatically match all subdomains:
+
+```bash
+# Allowing github.com permits:
+# ✓ github.com
+# ✓ api.github.com  
+# ✓ raw.githubusercontent.com
+# ✗ example.com (not in allowlist)
+
+sudo awf --allow-domains github.com -- curl https://api.github.com
+```
+
+:::tip
+You don't need to list every subdomain. Adding the base domain covers all subdomains automatically.
+:::
+
+## Allowlist options
+
+### Command-line flag
+
+Use `--allow-domains` with a comma-separated list:
+
+```bash
+sudo awf --allow-domains github.com,npmjs.org,googleapis.com -- <command>
+```
+
+### File-based allowlist
+
+Use `--allow-domains-file` for managing large domain lists:
+
+```bash
+# Create a domains file
+cat > allowed-domains.txt << 'EOF'
+# GitHub domains
+github.com
+api.github.com
+
+# NPM registry  
+npmjs.org, registry.npmjs.org
+
+# Wildcard patterns
+*.googleapis.com
+EOF
+
+# Use the file
+sudo awf --allow-domains-file allowed-domains.txt -- <command>
+```
+
+**File format:**
+- One domain per line or comma-separated
+- Comments start with `#` (full line or inline)
+- Empty lines are ignored
+- Whitespace is trimmed
+
+### Combining methods
+
+You can use both flags together - domains are merged:
+
+```bash
+sudo awf \
+  --allow-domains github.com \
+  --allow-domains-file my-domains.txt \
+  -- <command>
+```
+
+## Wildcard patterns
+
+Use `*` to match multiple domains:
+
+```bash
+# Match any subdomain of github.com
+--allow-domains '*.github.com'
+
+# Match api-v1.example.com, api-v2.example.com, etc.
+--allow-domains 'api-*.example.com'
+
+# Combine plain domains and wildcards
+--allow-domains 'github.com,*.googleapis.com,api-*.example.com'
+```
+
+:::caution
+Use quotes around patterns to prevent shell expansion of `*`.
+:::
+
+**Pattern matching rules:**
+
+| Pattern | Matches | Does Not Match |
+|---------|---------|----------------|
+| `*.github.com` | `api.github.com`, `raw.github.com` | `github.com` |
+| `api-*.example.com` | `api-v1.example.com`, `api-test.example.com` | `api.example.com` |
+| `github.com` | `github.com`, `api.github.com` | `notgithub.com` |
+
+**Security restrictions:**
+- Overly broad patterns like `*`, `*.*`, or `*.*.*` are rejected
+- Patterns are case-insensitive (DNS is case-insensitive)
+
+## Blocklist options
+
+Block specific domains while allowing others. **Blocked domains take precedence over allowed domains.**
+
+### Basic blocklist usage
+
+```bash
+# Allow example.com but block internal.example.com
+sudo awf \
+  --allow-domains example.com \
+  --block-domains internal.example.com \
+  -- curl https://api.example.com  # ✓ allowed
+
+sudo awf \
+  --allow-domains example.com \
+  --block-domains internal.example.com \
+  -- curl https://internal.example.com  # ✗ blocked
+```
+
+### Blocklist with wildcards
+
+```bash
+# Allow all of example.com except internal-* subdomains
+sudo awf \
+  --allow-domains example.com \
+  --block-domains 'internal-*.example.com' \
+  -- curl https://api.example.com  # ✓ allowed
+
+# Allow broad pattern, block sensitive subdomains
+sudo awf \
+  --allow-domains '*.example.com' \
+  --block-domains '*.secret.example.com' \
+  -- curl https://api.example.com  # ✓ allowed
+```
+
+### File-based blocklist
+
+```bash
+# Create a blocklist file
+cat > blocked-domains.txt << 'EOF'
+# Internal services that should never be accessed
+internal.example.com
+admin.example.com
+
+# Block all subdomains of sensitive.org
+*.sensitive.org
+EOF
+
+# Use the blocklist file
+sudo awf \
+  --allow-domains example.com,sensitive.org \
+  --block-domains-file blocked-domains.txt \
+  -- <command>
+```
+
+### Combining all options
+
+```bash
+sudo awf \
+  --allow-domains github.com \
+  --allow-domains-file allowed.txt \
+  --block-domains internal.github.com \
+  --block-domains-file blocked.txt \
+  -- <command>
+```
+
+## Common use cases
+
+### AI agent with API access
+
+Allow an AI agent to access specific APIs while blocking internal services:
+
+```bash
+sudo awf \
+  --allow-domains 'api.openai.com,*.github.com' \
+  --block-domains 'internal.github.com,admin.github.com' \
+  -- npx @github/copilot@latest --prompt "Analyze this code"
+```
+
+### CI/CD pipeline restrictions
+
+Restrict network access during builds:
+
+```bash
+sudo awf \
+  --allow-domains npmjs.org,registry.npmjs.org,github.com \
+  --block-domains-file ci-blocklist.txt \
+  -- npm install && npm test
+```
+
+### MCP server isolation
+
+Test MCP servers with controlled network access:
+
+```bash
+sudo awf \
+  --allow-domains arxiv.org,api.github.com \
+  -- npx @github/copilot@latest \
+    --mcp-server ./my-mcp-server.js \
+    --prompt "Search for papers"
+```
+
+## Normalization
+
+Domains are normalized before matching:
+
+- **Case-insensitive**: `GitHub.COM` = `github.com`
+- **Whitespace trimmed**: `" github.com "` = `github.com`  
+- **Trailing dots removed**: `github.com.` = `github.com`
+- **Protocols stripped**: `https://github.com` = `github.com`
+
+```bash
+# These are all equivalent
+--allow-domains github.com
+--allow-domains " GitHub.COM. "
+--allow-domains "https://github.com"
+```
+
+## Debugging domain filtering
+
+### Enable debug logging
+
+See which domains are being allowed or blocked:
+
+```bash
+sudo awf \
+  --allow-domains github.com \
+  --block-domains internal.github.com \
+  --log-level debug \
+  -- <command>
+```
+
+### Check Squid logs
+
+View traffic decisions after execution:
+
+```bash
+# Find blocked requests
+sudo grep "TCP_DENIED" /tmp/squid-logs-*/access.log
+
+# Find allowed requests  
+sudo grep "TCP_TUNNEL" /tmp/squid-logs-*/access.log
+```
+
+### Use the logs command
+
+```bash
+# View recent traffic with formatting
+awf logs
+
+# Filter to blocked requests only
+awf logs --format json | jq 'select(.isAllowed == false)'
+```
+
+## See also
+
+- [CLI Reference](/gh-aw-firewall/reference/cli-reference) - Complete option documentation
+- [Security Architecture](/gh-aw-firewall/reference/security-architecture) - How filtering works
diff --git a/docs-site/src/content/docs/index.md b/docs-site/src/content/docs/index.md
@@ -14,7 +14,7 @@ This project is part of GitHub Next's explorations of [Agentic Workflows](https:
 When AI agents like GitHub Copilot CLI run with access to tools and MCP servers, they can make network requests to any domain. This firewall provides **L7 (HTTP/HTTPS) egress control** using domain whitelisting, ensuring agents can only access approved domains while blocking all unauthorized network traffic.
 
 **Key Capabilities:**
-- **Domain Whitelisting**: Allow only specific domains (automatically includes subdomains)
+- **Domain Allowlist & Blocklist**: Allow specific domains and block exceptions with wildcard pattern support
 - **Docker-in-Docker Enforcement**: Spawned containers inherit firewall restrictions
 - **Host-Level Protection**: Uses iptables DOCKER-USER chain for defense-in-depth
 - **Zero Trust**: Block all traffic by default, allow only what you explicitly permit
@@ -174,36 +174,42 @@ The firewall uses a containerized architecture with three security layers:
 
 <div class="sl-steps">
 
-1. **Understand Security**
+1. **Learn Domain Filtering**
    
-   Review the [Security Architecture](/gh-aw-firewall/reference/security-architecture/) to learn how the firewall protects against attacks.
+   Master [allowlists, blocklists, and wildcards](/gh-aw-firewall/guides/domain-filtering/) for fine-grained network control.
 
-2. **Read Full Documentation**
+2. **Understand Security**
    
-   Check the [README](https://github.com/githubnext/gh-aw-firewall#readme) for detailed usage examples and configuration options.
+   Review the [Security Architecture](/gh-aw-firewall/reference/security-architecture/) to learn how the firewall protects against attacks.
 
-3. **Debug Issues**
+3. **CLI Reference**
    
-   See the [troubleshooting guide](https://github.com/githubnext/gh-aw-firewall/blob/main/docs/troubleshooting.md) for common problems and solutions.
+   See the [CLI Reference](/gh-aw-firewall/reference/cli-reference/) for all available options.
 
-4. **Explore Examples**
+4. **Debug Issues**
    
-   Browse the [examples directory](https://github.com/githubnext/gh-aw-firewall/tree/main/examples) for real-world use cases.
+   Check the [troubleshooting guide](https://github.com/githubnext/gh-aw-firewall/blob/main/docs/troubleshooting.md) for common problems and solutions.
 
 </div>
 
 ## Key Features
 
 ### Domain Whitelisting
 
-Domains automatically match all subdomains:
+Domains automatically match all subdomains. Use blocklist for fine-grained control:
 
 ```bash
 # Whitelisting github.com allows:
 # ✓ github.com
 # ✓ api.github.com
 # ✓ raw.githubusercontent.com
 # ✗ example.com (not whitelisted)
+
+# Block specific subdomains while allowing parent domain:
+sudo awf \
+  --allow-domains example.com \
+  --block-domains internal.example.com \
+  -- curl https://api.example.com  # ✓ allowed
 ```
 
 ### Host-Level Enforcement
@@ -238,6 +244,9 @@ sudo awf --allow-domains github.com,arxiv.org,npmjs.org -- <command>
 
 # From file
 sudo awf --allow-domains-file domains.txt -- <command>
+
+# With blocklist for fine-grained control
+sudo awf --allow-domains '*.example.com' --block-domains 'internal.example.com' -- <command>
 ```
 
 ## Architecture Highlights
diff --git a/docs-site/src/content/docs/reference/cli-reference.md b/docs-site/src/content/docs/reference/cli-reference.md
@@ -234,7 +234,5 @@ Log sources are auto-discovered in this order: running containers, `AWF_LOGS_DIR
 
 ## See Also
 
-- [Quick Start Guide](/gh-aw-firewall/quickstart) - Getting started with examples
-- [Usage Guide](/gh-aw-firewall/usage) - Detailed usage patterns and examples
-- [Troubleshooting](/gh-aw-firewall/troubleshooting) - Common issues and solutions
+- [Domain Filtering Guide](/gh-aw-firewall/guides/domain-filtering) - Allowlists, blocklists, and wildcards
 - [Security Architecture](/gh-aw-firewall/reference/security-architecture) - How the firewall works internally
diff --git a/docs-site/src/content/docs/reference/security-architecture.md b/docs-site/src/content/docs/reference/security-architecture.md
@@ -95,7 +95,13 @@ graph TB
 
 **Container iptables (NAT table)** — Inside the agent container, NAT rules intercept outbound HTTP (port 80) and HTTPS (port 443) traffic, rewriting the destination to Squid at `172.30.0.10:3128`. This handles traffic from the agent process itself and any child processes (including stdio MCP servers).
 
-**Squid ACL** — The primary control point. Squid receives CONNECT requests, extracts the target domain from SNI (for HTTPS) or Host header (for HTTP), and checks against the allowlist. Unlisted domains get `403 Forbidden`. No SSL inspection—we read SNI from the TLS ClientHello without decrypting traffic.
+**Squid ACL** — The primary control point. Squid receives CONNECT requests, extracts the target domain from SNI (for HTTPS) or Host header (for HTTP), and checks against the allowlist and blocklist. The evaluation order is:
+
+1. **Blocklist check first**: If domain matches a blocked pattern, deny immediately
+2. **Allowlist check second**: If domain matches an allowed pattern, permit
+3. **Default deny**: All other domains get `403 Forbidden`
+
+This allows fine-grained control like allowing `*.example.com` while blocking `internal.example.com`. No SSL inspection—we read SNI from the TLS ClientHello without decrypting traffic.
 
 ---
 
@@ -117,9 +123,13 @@ sequenceDiagram
     NAT->>Squid: TCP to proxy port
 
     Squid->>Squid: Parse CONNECT api.github.com:443
-    Squid->>Squid: Check domain against ACL
+    Squid->>Squid: Check blocklist first
+    Squid->>Squid: Check allowlist second
 
-    alt api.github.com in allowlist
+    alt Domain in blocklist
+        Squid-->>Agent: HTTP 403 Forbidden
+        Note over Agent: Blocked by blocklist
+    else Domain in allowlist
         Squid->>Host: Outbound to api.github.com:443
         Note over Host: Source is Squid IP (172.30.0.10)<br/>→ ACCEPT (unrestricted)
         Host->>Net: TCP connection
@@ -128,7 +138,7 @@ sequenceDiagram
         Note over Agent,Net: End-to-end encrypted tunnel
     else Domain not in allowlist
         Squid-->>Agent: HTTP 403 Forbidden
-        Note over Agent: Connection refused
+        Note over Agent: Not in allowlist
     end
 ```
 
@@ -305,6 +315,5 @@ Use `sudo -E` to preserve environment variables (like `GITHUB_TOKEN`) through su
 
 ## Related Documentation
 
-- [Architecture Overview](/reference/architecture) — Component details and code structure
-- [CLI Reference](/reference/cli-options) — Complete command-line options
-- [Logging](/guides/logging) — Audit trail configuration and analysis
+- [Domain Filtering](/gh-aw-firewall/guides/domain-filtering/) — Allowlists, blocklists, and wildcard patterns
+- [CLI Reference](/gh-aw-firewall/reference/cli-reference/) — Complete command-line options
