Skip to content

waf-detection-research.md

Latest commit

 

History

History
421 lines (330 loc) · 11.3 KB

waf-detection-research.md

File metadata and controls

421 lines (330 loc) · 11.3 KB
Raw

🔍 WAF Detection Research & Improvements

Research Summary

Based on analysis of industry-leading WAF detection tools (WAFW00F, IdentYwaf) and security research, I've identified key improvements for our detection engine.

🎯 Current Detection Methods (What We Have)

1. HTTP Header Analysis

  • Vendor-specific headers (cf-ray, x-amz-cf-id, akamai-grn)
  • Server headers
  • Custom headers

2. Cookie Analysis

  • Vendor-specific cookies (__cfduid, incap_ses, awsalb)
  • Cookie patterns

3. Response Content Analysis

  • Error page content
  • Vendor-specific text patterns

4. Status Code Analysis

  • 403, 406, 503 responses
  • Blocking behavior patterns

🚀 Recommended Improvements (Industry Best Practices)

1. Enhanced Response Code Analysis

Current Limitation: We only check if status code is in a list Improvement: Analyze unique response code patterns per vendor

# Enhanced response code patterns
'response_code_patterns': {
    'Cloudflare': {
        'common_codes': [403, 503, 520, 521, 522, 523, 524, 525, 526, 527],
        'unique_codes': [520, 521, 522, 523, 524],  # Cloudflare-specific
        'error_pages': True
    },
    'Akamai': {
        'common_codes': [403, 503],
        'reference_id_pattern': r'Reference #[\d\.]+',  # Akamai error format
    }
}

2. Error Page Fingerprinting

Current Limitation: Simple text matching Improvement: Analyze error page structure and unique identifiers

# Error page patterns
'error_page_patterns': {
    'Cloudflare': {
        'title_pattern': r'Attention Required.*Cloudflare',
        'ray_id_pattern': r'Ray ID: [a-f0-9]+',
        'html_structure': ['cf-error-details', 'cf-wrapper']
    },
    'Imperva': {
        'incident_id_pattern': r'Incident ID: [a-zA-Z0-9]+',
        'support_id_pattern': r'Support ID: [0-9]+',
    },
    'F5 BIG-IP': {
        'error_text': 'The requested URL was rejected. Please consult with your administrator.',
        'support_id_pattern': r'Support ID: [0-9]+'
    }
}

3. Cookie Attribute Analysis

Current Limitation: Only check cookie names Improvement: Analyze cookie attributes (domain, path, flags)

# Enhanced cookie analysis
'cookie_patterns': {
    'Cloudflare': {
        'names': ['__cfduid', '__cflb', 'cf_clearance'],
        'domain_pattern': r'\.cloudflare\.com',
        'attributes': ['SameSite=None', 'Secure']
    },
    'Akamai': {
        'names': ['ak_bmsc', 'bm_sv', 'bm_sz'],
        'path_pattern': r'^/',
        'bot_manager': True  # Akamai Bot Manager cookies
    }
}

4. Timing-Based Detection (Side-Channel)

New Technique: Analyze response timing patterns

# Timing analysis
def analyze_timing_patterns(self, target):
    """Detect WAF based on response timing"""
    timings = []
    
    # Send normal request
    normal_time = self._measure_response_time(target, normal=True)
    
    # Send malicious request
    malicious_time = self._measure_response_time(target, malicious=True)
    
    # WAFs often have different timing for malicious requests
    time_diff = malicious_time - normal_time
    
    if time_diff > 0.5:  # Significant delay
        return {'timing_anomaly': True, 'delay': time_diff}

5. HTTP/2 and HTTP/3 Fingerprinting

New Technique: Analyze protocol-specific features

# Protocol fingerprinting
'protocol_features': {
    'Cloudflare': {
        'http2_support': True,
        'http3_support': True,
        'alt_svc_header': 'h3=":443"'
    },
    'Fastly': {
        'http2_support': True,
        'via_header_pattern': r'.*fastly.*'
    }
}

6. TLS/SSL Fingerprinting

New Technique: Analyze TLS handshake and certificates

# TLS fingerprinting
'tls_patterns': {
    'Cloudflare': {
        'issuer_pattern': r'Cloudflare Inc',
        'san_pattern': r'sni\.cloudflaressl\.com',
        'cipher_suites': ['TLS_AES_128_GCM_SHA256', 'TLS_AES_256_GCM_SHA384']
    },
    'Akamai': {
        'issuer_pattern': r'Akamai',
        'organization': 'Akamai Technologies'
    }
}

7. Behavioral Analysis (Multiple Requests)

New Technique: Send sequence of requests to identify blocking patterns

# Behavioral detection
def behavioral_detection(self, target):
    """Detect WAF based on blocking behavior"""
    
    # 1. Normal request (should pass)
    normal_response = self._send_request(target, payload=None)
    
    # 2. Mildly suspicious request
    mild_response = self._send_request(target, payload="<script>")
    
    # 3. Highly malicious request
    malicious_response = self._send_request(target, payload="' OR 1=1--")
    
    # Analyze progression of responses
    if normal_response.status == 200 and mild_response.status == 403:
        return {'progressive_blocking': True}

8. Header Order and Case Sensitivity

New Technique: Some WAFs have specific header ordering

# Header analysis
'header_patterns': {
    'order_sensitive': {
        'Cloudflare': ['CF-RAY', 'CF-Cache-Status', 'Server'],
        'Akamai': ['Akamai-GRN', 'Server']
    },
    'case_patterns': {
        'custom_capitalization': True
    }
}

9. Rate Limiting Detection

New Technique: Identify WAF by rate limiting behavior

# Rate limiting detection
def detect_rate_limiting(self, target):
    """Detect WAF based on rate limiting"""
    
    responses = []
    for i in range(10):
        response = self._send_request(target)
        responses.append(response.status_code)
        time.sleep(0.1)
    
    # Check for rate limiting (429, 503, etc.)
    if 429 in responses or responses.count(503) > 3:
        return {'rate_limiting': True, 'threshold': i}

10. CAPTCHA/Challenge Detection

New Technique: Identify challenge pages

# Challenge detection
'challenge_patterns': {
    'Cloudflare': {
        'challenge_text': ['Checking your browser', 'Just a moment'],
        'javascript_challenge': 'cf-challenge',
        'captcha_challenge': 'cf-captcha-container'
    },
    'Imperva': {
        'challenge_text': ['Verifying you are human'],
        'captcha_provider': 'reCAPTCHA'
    }
}

🔧 Implementation Priority

High Priority (Immediate)

  1. ✅ Enhanced error page fingerprinting
  2. ✅ Cookie attribute analysis
  3. ✅ Response code pattern analysis
  4. ✅ CAPTCHA/Challenge detection

Medium Priority (Next Phase)

  1. ⚠️ Timing-based detection
  2. ⚠️ Behavioral analysis (multiple requests)
  3. ⚠️ Rate limiting detection

Low Priority (Future)

  1. 📋 TLS/SSL fingerprinting (requires SSL library)
  2. 📋 HTTP/2 and HTTP/3 analysis
  3. 📋 Header order analysis

📊 Vendor-Specific Improvements

Cloudflare

  • ✅ Add Cloudflare-specific error codes (520-527)
  • ✅ Detect "Checking your browser" challenge page
  • ✅ Add cf-challenge detection
  • ✅ Detect Ray ID format in error pages

Akamai

  • ✅ Add "Reference #" pattern detection
  • ✅ Detect Akamai Bot Manager cookies (bm_*)
  • ✅ Add akamai-grn header variations
  • ✅ Detect Akamai error page structure

Imperva (Incapsula)

  • ✅ Add "Incident ID" pattern detection
  • ✅ Detect "Support ID" in error pages
  • ✅ Add incap_ses cookie variations
  • ✅ Detect Imperva challenge pages

AWS WAF

  • ✅ Better differentiate CloudFront from AWS WAF
  • ✅ Add AWS WAF-specific error patterns
  • ✅ Detect X-Amzn-WAF-Action header
  • ✅ Add AWSALB cookie variations

Azure WAF

  • ✅ Add Azure Front Door specific headers
  • ✅ Detect Azure WAF error pages
  • ✅ Add ARRAffinity cookie detection
  • ✅ Detect x-azure-ref patterns

Fastly (Signal Sciences)

  • ✅ Add x-sigsci-* header detection
  • ✅ Detect Signal Sciences error pages
  • ✅ Add Varnish-specific patterns
  • ✅ Detect x-served-by patterns

🎯 Expected Improvements

Accuracy Gains

  • Cloudflare: 88.3% → 92%+ (add challenge detection)
  • Akamai: 83.7% → 87%+ (add Reference # pattern)
  • Imperva: 90.4% → 93%+ (add Incident ID pattern)
  • AWS WAF: 39.6% → 55%+ (better CloudFront differentiation)
  • Azure WAF: 61.1% → 70%+ (add Front Door patterns)
  • Fastly: 68.4% → 75%+ (add more Signal Sciences patterns)

Detection Rate

  • Current: 82.0%
  • Expected: 85%+

False Positive Reduction

  • Better vendor differentiation
  • More specific signatures
  • Multi-factor confirmation

🔍 Tools & Resources Referenced

Industry Tools

  1. WAFW00F - https://github.com/EnableSecurity/wafw00f

    • Largest WAF fingerprint database
    • Multi-stage detection approach
    • 150+ WAF signatures

  2. IdentYwaf - Blind WAF detection

    • Timing-based detection
    • Behavioral analysis

  3. w3af - Web Application Attack Framework

    • Comprehensive WAF detection
    • Automated testing

  4. WhatWaf - WAF detection and bypass

    • Multiple detection methods
    • Bypass technique database

Research Papers

  • "WAFFLED: Exploiting Parsing Discrepancies to Bypass WAFs"
  • WAF bypass techniques and fingerprinting methods
  • Side-channel timing attacks

Security Resources

  • Hacken.io WAF bypass cheat sheet
  • The Hacker Recipes - WAF fingerprinting
  • GitHub Awesome-WAF repository

📝 Implementation Notes

Code Structure

class EnhancedWAFDetector:
    def __init__(self):
        self.signatures = self._load_enhanced_signatures()
        self.timing_analyzer = TimingAnalyzer()
        self.behavioral_analyzer = BehavioralAnalyzer()
    
    def detect(self, target):
        # Stage 1: Passive detection (headers, cookies)
        passive_results = self._passive_detection(target)
        
        # Stage 2: Active detection (malicious payloads)
        active_results = self._active_detection(target)
        
        # Stage 3: Behavioral analysis
        behavioral_results = self._behavioral_detection(target)
        
        # Stage 4: Timing analysis
        timing_results = self._timing_detection(target)
        
        # Combine all results with weighted scoring
        return self._combine_results([
            passive_results,
            active_results,
            behavioral_results,
            timing_results
        ])

Weighted Scoring

DETECTION_WEIGHTS = {
    'unique_header': 40,      # Vendor-specific header
    'unique_cookie': 35,      # Vendor-specific cookie
    'error_page_pattern': 30, # Specific error page
    'server_header': 30,      # Server identification
    'challenge_page': 25,     # CAPTCHA/challenge
    'timing_anomaly': 20,     # Timing pattern
    'behavioral': 15,         # Blocking behavior
    'generic_header': 10,     # Generic header
    'status_code': 5          # Status code only
}

✅ Conclusion

By implementing these improvements, we can:

  1. Increase overall detection accuracy by 5-10%
  2. Reduce false positives significantly
  3. Better differentiate between similar vendors
  4. Achieve 90%+ confidence for top vendors
  5. Improve AWS/Azure WAF detection substantially

Next Steps:

  1. Implement high-priority improvements
  2. Test against 600-site dataset
  3. Validate accuracy improvements
  4. Document new detection methods
  5. Update vendor signatures

Research Date: March 1, 2026
Sources: WAFW00F, IdentYwaf, Hacken.io, Security Research Papers
Status: Ready for Implementation