fix(terraform): ensure consistent JSON output format when no checks are evaluated

[daily-dreamer]

Changes:

• Modified runner_registry.py line 535 to use Report.get_dict() instead of Report.get_summary()
• Ensures consistent JSON structure across all scenarios output (empty results, file not found, no matches)

Impact:

• API consumers no longer need special handling for empty results
• JSON output structure is now consistent regardless of check evaluation status
• Maintains backward compatibility by keeping summary data within the response

Fixes: Inconsistent JSON output format when no checks are evaluated

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

PR Title: docs(general): add documentation for nominal success JSON output

Description

This PR adds a new documentation file JSON_SUMMARY_README.md that clarifies the JSON output format when Checkov runs but evaluates zero checks (Nominal Success).

Motivation and Context

Users were confused about the difference between a "successful" scan that returns a full JSON vs one that returns only a summary dictionary.

• Nominal Success (Summary Only): Occurs when passed: 0 and failed: 0 (e.g., file not found, no scannable resources, or no check matches).
• Evaluation Success (Full JSON): Occurs when passed > 0.

This documentation helps developers understand and parse the JSON output correctly by accounting for the "Empty Report" edge case.

Checklist:

• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have added tests that prove my feature, policy, or fix is effective and works
• New and existing tests pass locally with my changes

[daily-dreamer]

Test Scenarios for JSON Output Format Fix

---

Prerequisites

# Install checkov (if not already installed)
pip install checkov

# Or install from your fork for testing
cd /path/to/your/fork/checkov
pip install -e .

---

Test Scenario 1: Empty Directory (Nothing to Scan)

Purpose

Verify that when scanning an empty directory, Checkov returns a consistent JSON structure.

Setup

# Create a clean test directory
mkdir -p /tmp/checkov-test-empty
cd /tmp/checkov-test-empty

# Ensure it's completely empty
rm -rf *
ls -la  # Should show only . and ..

Test Command

checkov -d /tmp/checkov-test-empty --output json

Expected Behavior BEFORE Fix

{
    "passed": 0,
    "failed": 0,
    "skipped": 0,
    "parsing_errors": 0,
    "resource_count": 0,
    "checkov_version": "3.2.509"
}

❌ Problem: Only summary object, missing check_type, results, and url fields.

Expected Behavior AFTER Fix

{
    "check_type": "terraform",
    "results": {
        "passed_checks": [],
        "failed_checks": [],
        "skipped_checks": [],
        "parsing_errors": []
    },
    "summary": {
        "passed": 0,
        "failed": 0,
        "skipped": 0,
        "parsing_errors": 0,
        "resource_count": 0,
        "checkov_version": "3.2.509"
    },
    "url": "https://www.checkov.io/"
}

✅ Fixed: Full JSON structure with all expected fields.

Verification

# Save output to file
checkov -d /tmp/checkov-test-empty --output json > /tmp/output1.json

# Check if it has the 'check_type' field (should exist after fix)
cat /tmp/output1.json | grep -q "check_type" && echo "✅ PASS: Full structure" || echo "❌ FAIL: Summary only"

# Check if it has the 'results' field (should exist after fix)
cat /tmp/output1.json | grep -q "results" && echo "✅ PASS: Has results field" || echo "❌ FAIL: Missing results"

# Check if it has the 'summary' field (should exist after fix)
cat /tmp/output1.json | grep -q "summary" && echo "✅ PASS: Has summary field" || echo "❌ FAIL: Missing summary"

---

Test Scenario 2: File Not Present

Purpose

Verify that when scanning a non-existent file, Checkov returns a consistent JSON structure.

Setup

# Ensure the file doesn't exist
rm -f /tmp/nonexistent-file.tf
ls /tmp/nonexistent-file.tf  # Should show "No such file or directory"

Test Command

checkov -f /tmp/nonexistent-file.tf --output json 2>/dev/null

Expected Behavior BEFORE Fix

{
    "passed": 0,
    "failed": 0,
    "skipped": 0,
    "parsing_errors": 0,
    "resource_count": 0,
    "checkov_version": "3.2.509"
}

❌ Problem: Only summary object.

Expected Behavior AFTER Fix

{
    "check_type": "terraform",
    "results": {
        "passed_checks": [],
        "failed_checks": [],
        "skipped_checks": [],
        "parsing_errors": []
    },
    "summary": {
        "passed": 0,
        "failed": 0,
        "skipped": 0,
        "parsing_errors": 0,
        "resource_count": 0,
        "checkov_version": "3.2.509"
    },
    "url": "https://www.checkov.io/"
}

✅ Fixed: Full JSON structure.

Verification

# Save output to file (suppress stderr)
checkov -f /tmp/nonexistent-file.tf --output json 2>/dev/null > /tmp/output2.json

# Verify structure
cat /tmp/output2.json | python3 -c "
import json, sys
data = json.load(sys.stdin)
assert 'check_type' in data, 'Missing check_type'
assert 'results' in data, 'Missing results'
assert 'summary' in data, 'Missing summary'
print('✅ PASS: All required fields present')
"

---

Test Scenario 3: File with No Scannable Resources

Purpose

Verify that when scanning a file with no recognizable resources, Checkov returns a consistent JSON structure.

Setup

# Create a file with only comments (no resources)
cat > /tmp/empty-content.tf << 'EOF'
# This is just a comment file
# No actual Terraform resources here

# Another comment
# Still no resources
EOF

Test Command

checkov -f /tmp/empty-content.tf --output json

Expected Behavior BEFORE Fix

{
    "passed": 0,
    "failed": 0,
    "skipped": 0,
    "parsing_errors": 0,
    "resource_count": 0,
    "checkov_version": "3.2.509"
}

❌ Problem: Only summary object.

Expected Behavior AFTER Fix

{
    "check_type": "terraform",
    "results": {
        "passed_checks": [],
        "failed_checks": [],
        "skipped_checks": [],
        "parsing_errors": []
    },
    "summary": {
        "passed": 0,
        "failed": 0,
        "skipped": 0,
        "parsing_errors": 0,
        "resource_count": 0,
        "checkov_version": "3.2.509"
    },
    "url": "https://www.checkov.io/"
}

✅ Fixed: Full JSON structure.

Verification

# Save output to file
checkov -f /tmp/empty-content.tf --output json > /tmp/output3.json

# Verify using jq (if available)
if command -v jq &> /dev/null; then
    echo "Checking with jq..."
    jq -e '.check_type' /tmp/output3.json && echo "✅ Has check_type"
    jq -e '.results' /tmp/output3.json && echo "✅ Has results"
    jq -e '.summary' /tmp/output3.json && echo "✅ Has summary"
else
    echo "Checking with grep..."
    grep -q "check_type" /tmp/output3.json && echo "✅ Has check_type"
    grep -q "results" /tmp/output3.json && echo "✅ Has results"
    grep -q "summary" /tmp/output3.json && echo "✅ Has summary"
fi

---

Test Scenario 4: Comparison with Normal Scan (Control Test)

Purpose

Verify that the fix doesn't break normal scanning behavior when resources ARE found.

Setup

# Create a valid Terraform file with a resource
cat > /tmp/valid-resource.tf << 'EOF'
resource "aws_s3_bucket" "example" {
  bucket = "my-test-bucket"
  
  tags = {
    Name        = "My bucket"
    Environment = "Dev"
  }
}
EOF

Test Command

checkov -f /tmp/valid-resource.tf --output json

Expected Behavior (BEFORE and AFTER Fix - Should be Same)

{
    "check_type": "terraform",
    "results": {
        "passed_checks": [
            {
                "check_id": "CKV_AWS_...",
                "check_name": "...",
                "check_result": {
                    "result": "passed"
                },
                "file_path": "/tmp/valid-resource.tf",
                "resource": "aws_s3_bucket.example"
            }
        ],
        "failed_checks": [
            {
                "check_id": "CKV_AWS_...",
                "check_name": "...",
                "check_result": {
                    "result": "failed"
                },
                "file_path": "/tmp/valid-resource.tf",
                "resource": "aws_s3_bucket.example"
            }
        ],
        "skipped_checks": [],
        "parsing_errors": []
    },
    "summary": {
        "passed": 5,
        "failed": 10,
        "skipped": 0,
        "parsing_errors": 0,
        "resource_count": 1,
        "checkov_version": "3.2.509"
    },
    "url": "https://www.checkov.io/"
}

✅ Expected: Normal scans should work exactly as before.

Verification

# Save output
checkov -f /tmp/valid-resource.tf --output json > /tmp/output4.json

# Verify it has the full structure
python3 << 'PYTHON'
import json
with open('/tmp/output4.json') as f:
    data = json.load(f)
    
# Check structure
assert 'check_type' in data, "Missing check_type"
assert 'results' in data, "Missing results"
assert 'summary' in data, "Missing summary"

# Check that checks were actually run
summary = data['summary']
total_checks = summary['passed'] + summary['failed'] + summary['skipped']
assert total_checks > 0, "No checks were run!"

print(f"✅ PASS: Normal scan works correctly")
print(f"   - Passed: {summary['passed']}")
print(f"   - Failed: {summary['failed']}")
print(f"   - Skipped: {summary['skipped']}")
print(f"   - Total checks: {total_checks}")
PYTHON

---

Test Scenario 5: Multiple Frameworks (Comprehensive Test)

Purpose

Verify the fix works across different framework types (Terraform, Kubernetes, etc.).

Setup

# Create test directory
mkdir -p /tmp/checkov-multi-framework-test

# Create empty Kubernetes file
cat > /tmp/checkov-multi-framework-test/empty.yaml << 'EOF'
# Just comments
# No actual Kubernetes resources
EOF

# Create empty Dockerfile
cat > /tmp/checkov-multi-framework-test/Dockerfile << 'EOF'
# Empty Dockerfile
# No actual instructions
EOF

Test Commands

# Test Kubernetes framework
checkov -f /tmp/checkov-multi-framework-test/empty.yaml --framework kubernetes --output json > /tmp/output5-k8s.json

# Test Dockerfile framework
checkov -f /tmp/checkov-multi-framework-test/Dockerfile --framework dockerfile --output json > /tmp/output5-docker.json

Expected Behavior AFTER Fix

Both outputs should have full JSON structure:

{
    "check_type": "kubernetes",  // or "dockerfile"
    "results": {
        "passed_checks": [],
        "failed_checks": [],
        "skipped_checks": [],
        "parsing_errors": []
    },
    "summary": {
        "passed": 0,
        "failed": 0,
        "skipped": 0,
        "parsing_errors": 0,
        "resource_count": 0,
        "checkov_version": "3.2.509"
    },
    "url": "https://www.checkov.io/"
}

Verification

# Verify both outputs
for file in /tmp/output5-k8s.json /tmp/output5-docker.json; do
    echo "Checking $file..."
    python3 << PYTHON
import json
with open('$file') as f:
    data = json.load(f)
    assert 'check_type' in data, "Missing check_type in $file"
    assert 'results' in data, "Missing results in $file"
    assert 'summary' in data, "Missing summary in $file"
    print(f"✅ PASS: $file has correct structure")
PYTHON
done

---

Manual Verification Checklist

Use this checklist to manually verify the fix:

• Test 1: Empty directory returns full JSON structure
• Test 2: Non-existent file returns full JSON structure
• Test 3: File with no resources returns full JSON structure
• Test 4: Normal scan with resources still works correctly
• Test 5: Multiple frameworks (K8s, Docker) work correctly
• Consistency: All empty scenarios return same structure format
• Backward Compatibility: Summary data is still present in summary field
• No Regressions: Existing functionality not broken

---

Expected Test Results

BEFORE Fix Applied

Test 1: Empty Directory          ❌ FAIL (summary only)
Test 2: Non-existent File        ❌ FAIL (summary only)
Test 3: File with No Resources   ❌ FAIL (summary only)
Test 4: Valid Resource           ✅ PASS (works normally)

AFTER Fix Applied

Test 1: Empty Directory          ✅ PASS (full structure)
Test 2: Non-existent File        ✅ PASS (full structure)
Test 3: File with No Resources   ✅ PASS (full structure)
Test 4: Valid Resource           ✅ PASS (works normally)

---

Troubleshooting

Issue: Tests still failing after applying fix

Solution:

1. Ensure you edited the correct file: checkov/common/runners/runner_registry.py
2. Verify the change is on line 535 (or nearby)
3. Reinstall checkov: pip install -e . from your fork directory
4. Clear any cached bytecode: find . -type d -name __pycache__ -exec rm -rf {} +

Issue: Cannot find checkov command

Solution:

# Install from your fork
cd /path/to/your/fork/checkov
pip install -e .

# Verify installation
which checkov
checkov --version

Issue: JSON output is malformed

Solution:

• Check for syntax errors in your code change
• Ensure proper indentation (Python uses 4 spaces)
• Verify all parentheses and brackets are balanced

---

[daily-dreamer]

Hi maintainers! 👋

I've implemented a fix for JSON output format consistency. This PR includes:

• Minimal code change (3 lines in runner_registry.py)
• Complete documentation in the issue
• Backward compatible

The fix ensures empty reports return the same JSON structure as non-empty reports, making it easier for tools to parse Checkov output.

Happy to address any feedback! 🙏
@gruebel @mblonder @mayblo