Cloud Security Assessment Report (CDRL A008): runner, findings tracker and report

[erikrozi]

Task prompt

Repository: COG-GTM/aws-cloudformation-templates (a fork of the AWS CloudFormation sample templates — ~300 CloudFormation YAML/JSON templates organized by service directory: `APIGateway/`, `EC2/`, `RDS/`, `S3/`, `VPC/`, `EKS/`, `Lambda/`, `Solutions/`, etc., plus a small Terraform module under `EC2/terraform/`). Lint command: `cfn-lint <template.yaml>`. Terraform check: `terraform -chdir=EC2/terraform init -backend=false && terraform -chdir=EC2/terraform validate`.

You are acting as the Cloud Computing Specialist / Enterprise Architect on a Department of the Air Force cloud-security engineering task. Treat this repository as the infrastructure-as-code baseline for a set of Government cloud workloads and produce a **Cloud Security Assessment Report** in the form the Government expects from CDRL A008 ("Cloud Security Assessment Report — MS Word/PDF report and MS Excel findings tracker"), then remediate the highest-severity findings in a second PR.

## Performance standard you must meet (from the tasking)
"Findings shall be risk-ranked, evidence-based, traceable to the applicable baseline or requirement, and include recommended remediation or disposition."

## PR 1 — Assessment (no template changes)

Create `security-assessment/` at the repo root containing:

1. `scripts/cloud_security_assessment.py` (repo-root `scripts/`) — a reusable Python 3 runner that:
   - Runs `checkov` (install with pip) across every CloudFormation template and the Terraform module, and `cfn_nag` if it is installable (Ruby gem; skip gracefully if not) and `cfn-lint`.
   - Adds its own rule pack for items those tools miss or that the Government baseline emphasizes: encryption at rest for every storage/database/queue/topic/log resource (S3, EBS, RDS, DynamoDB, EFS, SQS, SNS, CloudWatch Logs, Kinesis/Firehose), KMS customer-managed keys vs AWS-managed, TLS-in-transit enforcement (S3 bucket policies with `aws:SecureTransport`, ELB/ALB listener protocols and TLS security policies, RDS `rds.force_ssl`, API Gateway TLS 1.2 minimum), public exposure (0.0.0.0/0 or ::/0 ingress, public S3 access blocks, `PubliclyAccessible` RDS, public subnets for databases), logging and monitoring (S3 access logging, VPC Flow Logs, CloudTrail, ALB access logs, API Gateway execution logs, RDS/EKS control-plane logs), IAM least privilege (wildcard actions/resources, inline policies, missing permission boundaries, `iam:PassRole` with `*`), IMDSv2 enforcement (`HttpTokens: required`), EBS encryption, secrets in `Parameters` without `NoEcho`, hardcoded credentials in templates, Lambda without VPC/tracing where relevant, EKS public endpoint, backup/retention settings, and deletion protection for stateful resources.
   - Normalizes every finding into one schema and maps each to: **NIST SP 800-53 Rev 5 control(s)** (e.g. SC-28 encryption at rest, SC-8/SC-8(1) transit, AC-3/AC-6 least privilege, AU-2/AU-12 logging, SI-4 monitoring, CM-6 configuration settings, CP-9 backups, IA-5 authenticator management, SC-7 boundary protection), **CIS AWS Foundations Benchmark v3.0 recommendation ID** where one exists, **DoD Cloud Computing SRG** topic area (identify the relevant SRG section by name; verify the current SRG version and section names via web search and cite only what you confirm), and a **DISA STIG/SRG-style severity** (CAT I / CAT II / CAT III) with a one-line justification. Also emit a `checkov_id`/`cfn_nag_id`/`custom_rule_id` for tool traceability.
   - Writes `security-assessment/findings.json`, `security-assessment/Cloud-Security-Findings-Tracker.xlsx` (openpyxl; sheets: `Findings` — Finding ID, Template path, Resource logical ID, Resource type, Finding title, Description, Evidence (file:line snippet), Severity (CAT I/II/III), Risk rank (1–N), NIST 800-53 Rev 5 control(s), CIS AWS v3.0 ID, DoD Cloud SRG area, Tool/rule ID, Recommended remediation, Disposition (Open / Remediated in PR / Risk acceptance recommended / Not applicable), Owner (role, not a person), Target date placeholder — with frozen header, autofilter, conditional fill by severity; `Summary` — counts by severity, by service directory, by control family, scan date, commit SHA; `Control-Coverage` — for each NIST control referenced, number of findings and number of templates assessed; `POAM-Draft` — one row per open CAT I/II finding in a POA&M-style layout (Weakness, Control, Severity, Scheduled completion placeholder, Milestones, Resources required, Status); `Method` — tools, versions, rule pack, known limits).
   - Prints a console summary.

2. `security-assessment/Cloud-Security-Assessment-Report.md` — narrative report (6–10 pages rendered): Executive summary with counts by severity; Scope and method (which templates, tools and versions, commit SHA); Risk-ranked findings (top 20 with evidence and traceability); Findings by NIST control family; Systemic patterns (e.g. "encryption at rest absent in N of M storage resources") and recommended secure-by-default automation (pre-commit `cfn-lint`/`checkov` gate, CI policy-as-code, reusable secure module patterns, Rain modules); Zero Trust and identity-integration observations relevant to IaC; Remediation plan and what was fixed in PR 2 vs. what needs Government disposition; Decisions requiring Government action; Appendix: full control crosswalk table. Plain language; short sentences; no marketing adjectives.

3. `security-assessment/README.md` — regeneration steps and artifact descriptions.

Commit the generated `findings.json` and XLSX so reviewers can open them without running anything.

## PR 2 — Remediation (based on PR 1's branch)

Fix the **top 10 CAT I findings by risk rank** (and any trivially-adjacent CAT II in the same resource) directly in the templates: enable encryption at rest, add S3 public access blocks and `aws:SecureTransport` deny policies, require IMDSv2, enforce TLS 1.2+ listener policies, remove 0.0.0.0/0 admin ingress in favour of a parameterized CIDR, add logging where missing, and so on. Keep each template deployable and `cfn-lint` clean. Do not change template intent (a public web ALB stays public; a bastion stays a bastion). Re-run the assessment script on the remediated branch and commit the updated tracker with those rows marked `Remediated in PR` and a before/after count in `Summary`. If a fix is not safe without knowing the Government's environment, do not guess — leave it Open and record it under "Decisions requiring Government action".

## Constraints

- Every finding must have evidence (file:line) and at least one NIST 800-53 Rev 5 control ID. No finding without traceability.
- Verify external references (CIS benchmark version/IDs, SRG version) via web search before citing; never fabricate an ID or URL.
- Do not use the word "demo" anywhere in the PRs, commit messages, or artifacts, and do not name any customer, agency program office, or partner company inside repository artifacts (use "the Government", "the system owner").
- Both PR descriptions must start with a `## Task prompt` section containing this entire prompt verbatim in a fenced code block, followed by `## Summary`, `## Top findings` (table), `## Traceability` (how controls were mapped), and `## How to regenerate`.
- Branch names: `devin/<unix-timestamp>-cloud-security-assessment` and `devin/<unix-timestamp>-cloud-security-remediation` (the second branched from the first; open PR 2 against PR 1's branch or against the default branch — state which).

## Done criteria

- `python3 scripts/cloud_security_assessment.py` runs end to end and the XLSX loads with openpyxl (print sheet names and row counts).
- `cfn-lint` passes on every template you modified in PR 2; `terraform validate` passes if you touched Terraform.
- Final message: both PR URLs, counts by severity before and after remediation, and attach the XLSX (both versions), `findings.json`, and the report rendered to PDF (pandoc or LibreOffice) via message attachments. Provide structured output with both PR URLs and a 5-sentence summary.

Summary

Adds a reusable Cloud Security Assessment runner and commits the generated CDRL A008-style artifacts for the CloudFormation baseline at commit a0f43bc6. No CloudFormation templates are changed in this PR.

• scripts/cloud_security_assessment.py — discovers every CloudFormation template (160; 138 generated JSON twins are compared with their YAML source and only drift is reported) and any Terraform (.tf) files (none exist in this revision; the prompt's EC2/terraform/ module is absent on main, so that part of the method is recorded as not applicable). Runs Checkov 3.3.17, cfn_nag 0.8.10 and cfn-lint 1.56.3 (per-template so an unparseable template cannot abort the scan; cfn-lint follows scripts/lint-single.sh conventions — macro examples and Rain fragments excluded, !Rain:: templates packaged first), plus a 55-rule Government-oriented rule pack (CSA-ENC-*, CSA-TLS-*, CSA-NET-*, CSA-LOG-*, CSA-IAM-*, CSA-SEC-*, CSA-CFG-*, CSA-BKP-*, …). Every hit is normalized into one Finding schema with file:line evidence, a snippet, CAT I/II/III severity with justification, deterministic finding ID, risk score/rank, NIST SP 800-53 Rev. 5 controls, CIS AWS v3.0 ID where one exists, DoD Cloud Computing SRG area and tool IDs. Tool-only (heuristic) findings are capped at CAT II; corroborating Checkov/cfn_nag hits are merged into the custom finding rather than duplicated. Writes findings.json, the XLSX tracker (Findings, Summary, Control-Coverage, POAM-Draft, Method) and a console summary. --baseline carries forward findings that disappear as Remediated in PR only when every scanner that produced them ran again on that template (otherwise the prior disposition is kept with a Not re-evaluated note) and adds before/after counts (used by PR 2). --fail-on-incomplete exits 2 for CI when a scanner was skipped, failed or left templates unprocessed.
• scripts/cloud_security_report.py — renders security-assessment/Cloud-Security-Assessment-Report.md from findings.json (every number in the report is derived from the data).
• security-assessment/ — findings.json, Cloud-Security-Findings-Tracker.xlsx, Cloud-Security-Assessment-Report.md, README.md.

Result at baseline: 1,192 findings / 1,174 open — CAT I 9, CAT II 371, CAT III 794 (18 recommended for risk acceptance: intentional public-website buckets and CloudFront origins). 149 of 160 templates have at least one open finding. Every finding has file:line evidence and at least one NIST control; all 1,192 finding IDs are unique.

Scanner coverage recorded in the artifacts: checkov did not process 7 template(s); cfn_nag did not process 12 template(s); cfn-lint did not process 2 template(s) (Checkov/cfn_nag parser or schema errors and the two StackSets templates whose !Rain::Embed target is a build artifact not in the repository); those templates are covered by the custom rule pack and the remaining scanners and are listed in the Method sheet. No malformed template files and no custom-rule failures were recorded. Provenance: the report states the assessment checkout SHA and templates_commit_sha (last commit touching any assessed template, a0f43bc6) separately, because commits on this branch add assessment files only — the artifacts are necessarily generated one commit before the commit that contains them.

Review follow-up (second commit): bucket-wide aws:SecureTransport deny semantics for CSA-TLS-001; CloudFront MinimumProtocolVersion missing/parameter-default detection; per-hit stable finding IDs; coverage-gated baseline carry-forward; report text for scanner coverage, Terraform scope, systemic-pattern numerators (open findings only) and planned fixes now derived from data.

Review follow-up (third commit): BucketPolicy.Bucket given as a literal/!Sub name is associated with the bucket whose BucketName is the same expression (bucket_policy_targets); a cfn-lint batch that exits with a non-result code, no JSON list, or an invocation-level error (Filename: null) marks the whole batch unprocessed instead of silently dropping it; finding IDs use a content key (CIDR/port/protocol for ingress, Sid or a digest of Effect/Action/Resource for IAM statements) so IDs do not shift when sibling hits are added or removed; Checkov summary.parsing_errors > 0 and cfn_nag FATAL pseudo-violations mark the template unprocessed for that tool; a generated JSON twin counts as assessed only when both it and its source parse (generated_json_not_compared otherwise); the executive-summary coverage sentence is built from tool status/unparsed/excluded lists.

Review follow-up (fourth commit): a custom rule that raises on a parsed template is recorded in metadata.custom_rule_failures, fails --fail-on-incomplete, and blocks the baseline carry-forward from marking that rule's findings on that template as remediated; CSA-TLS-005 inspects ordered CacheBehaviors as well as DefaultCacheBehavior; CSA-LOG-006 reads Logging.ClusterLogging in its CloudFormation list form; generated-twin drift comparison keeps scalar types and null/empty distinctions (only Fn::GetAZs null vs "" is treated as equivalent) while identity digests keep their text canonicalisation so finding IDs are unchanged; discovery distinguishes well-formed non-template files (skipped) from malformed JSON/YAML (metadata.malformed_files, a coverage gap that fails --fail-on-incomplete) and parses multi-document YAML without error; coverage_gap checks the custom-rule parsed set only for custom-sourced findings and matches Terraform findings against Checkov's per-directory unprocessed list, so a Terraform finding that disappears is marked remediated when Checkov processed its module. This PR description is rendered from findings.json and the workbook by a generator, so its counts cannot drift from the committed artifacts.

Review follow-up (fifth commit): configuration blocks count as a control only when they carry the required sub-keys in every Fn::If branch — s3_sse_defaults for CSA-ENC-001/CSA-ENC-002 (an empty BucketEncryption, a rule list without ServerSideEncryptionByDefault or a default without SSEAlgorithm is reported as no encryption) and block_gap for S3 LoggingConfiguration, Kinesis StreamEncryption, Firehose DeliveryStreamEncryptionConfigurationInput and Lambda VpcConfig; JSON prop_line searches stop at the next resource so a later resource's property is never cited as evidence for an earlier one; external scanner hits are deduplicated by (source, check, template, resource, discriminator) — cfn-lint uses the property path below the resource, cfn_nag/Checkov use line:message only when one check fires more than once on one resource — so distinct violations stay distinct and a lone violation keeps its plain ID; baseline carry-forward rebuilds rows field by field (finding_from_dict: unknown keys ignored, defaulted fields filled, rows missing a required field skipped and reported) and metadata.schema_version is written and checked; report systemic shares count unique (template, resource) pairs and derive denominators from each rule's applies_to types, failing if a numerator ever exceeds its denominator. scripts/tests/test_cloud_security_assessment.py (pytest, 21 tests) covers parsing and evidence bounds, the S3 and block-gap rules, scanner normalization and merging, stable IDs, baseline schema tolerance, report shares and the committed artifact invariants.

Workbook check (openpyxl, data rows): Findings 1192, Summary 68, Control-Coverage 30, POAM-Draft 380, Method 23.

Top findings

Rank	Severity	Finding	Evidence	NIST / CIS	Rule
1	CAT I	CSA-NET-001-DB6129-0C0E: Security group allows unrestricted ingress to administrative ports	Solutions/CloudFormationEndpointSignals/cfn-endpoint-creationpolicy.yaml:242 BastionSG	SC-7, SC-7(5), AC-17 / CIS 5.2	CSA-NET-001
2	CAT I	CSA-NET-001-96174D-0C0E: Security group allows unrestricted ingress to administrative ports	Solutions/CloudFormationEndpointSignals/cfn-endpoint-waitcondition.yaml:264 BastionSG	SC-7, SC-7(5), AC-17 / CIS 5.2	CSA-NET-001
3	CAT I	CSA-NET-001-EAB626-8C16: Security group allows unrestricted ingress to administrative ports	Solutions/EC2DomainJoin/EC2-Domain-Join.yaml:170 InstanceSecurityGroup	SC-7, SC-7(5), AC-17 / CIS 5.2	CSA-NET-001
4	CAT I	CSA-NET-001-9E1C81-0C0E: Security group allows unrestricted ingress to administrative ports	EFS/efs_with_automount_to_ec2.yaml:313 InstanceSecurityGroup	SC-7, SC-7(5), AC-17 / CIS 5.2	CSA-NET-001
5	CAT I	CSA-NET-003-F362F5-016A: Security group allows unrestricted ingress on all ports and protocols	ECS/EC2LaunchType/clusters/private-vpc.yaml:455 PublicLoadBalancerSG	SC-7, SC-7(5), CM-7 / CIS n/a	CSA-NET-003
6	CAT I	CSA-NET-003-BF62E5-016A: Security group allows unrestricted ingress on all ports and protocols	ECS/EC2LaunchType/clusters/public-vpc.yaml:323 PublicLoadBalancerSG	SC-7, SC-7(5), CM-7 / CIS n/a	CSA-NET-003
7	CAT I	CSA-NET-003-048FBF-016A: Security group allows unrestricted ingress on all ports and protocols	ECS/FargateLaunchType/clusters/private-vpc.yaml:293 PublicLoadBalancerSG	SC-7, SC-7(5), CM-7 / CIS n/a	CSA-NET-003
8	CAT I	CSA-NET-003-3A5EFB-016A: Security group allows unrestricted ingress on all ports and protocols	ECS/FargateLaunchType/clusters/public-vpc.yaml:159 PublicLoadBalancerSG	SC-7, SC-7(5), CM-7 / CIS n/a	CSA-NET-003
9	CAT I	CSA-ENC-003-C5F9F7: Database storage is not encrypted at rest	NeptuneDB/Neptune.yaml:311 NeptuneDBCluster	SC-28, SC-28(1) / CIS n/a	CSA-ENC-003
10	CAT II	CSA-NET-007-5B39EA: S3 bucket does not block public access	CloudFormation/MacrosExamples/Count/test.yaml:16 BucketToCopyA	AC-3, SC-7, CM-6 / CIS 2.1.4	CSA-NET-007

Only 9 CAT I findings exist at baseline; rank 10 is the highest-ranked CAT II. The full top 20 with evidence is in the report; all 1,192 rows are in the tracker.

Traceability

• Each custom rule declares its NIST SP 800-53 Rev. 5 control set, default severity with a one-line justification, DoD Cloud Computing SRG topic area and (where one exists) the CIS AWS Foundations Benchmark v3.0 recommendation ID. Only IDs confirmed against the published benchmark are used (e.g. 2.1.1 S3 TLS, 2.1.4 S3 Block Public Access, 2.2.1 EBS encryption, 2.3.1 RDS encryption, 2.3.3 RDS public access, 2.4.1 EFS encryption, 3.1/3.2/3.4/3.5 CloudTrail, 3.7 VPC Flow Logs, 5.2/5.3 unrestricted admin ingress, 5.6 IMDSv2); rules with no CIS equivalent carry n/a.
• Checkov CKV_AWS_*, cfn_nag W*/F* and cfn-lint E* IDs are mapped to the same control sets through a rule-ID map in the runner; unmapped tool checks are recorded with their tool ID and a generic CM-6 mapping, at CAT II or CAT III.
• SRG areas are the topic areas of the DoD Cloud Computing SRG (Architecture, Security Requirements, Continuous Monitoring, Data Recovery and Destruction, POA&M) with the Mission Owner SRG requirement IDs listed in Appendix A.2 of the report. References cited are limited to NIST CSRC, CIS, AWS Security Hub, DoD Cyber Exchange and the tool repositories (Appendix C).
• Severity policy: CAT I = direct, immediate loss of confidentiality/integrity/availability (e.g. unrestricted admin ingress, unencrypted database storage); CAT II = potential loss; CAT III = degraded protection. Tool-only findings never exceed CAT II.

How to regenerate

pip install checkov cfn-lint pyyaml openpyxl      # optional: gem install cfn-nag; install rain
python3 scripts/cloud_security_assessment.py       # writes security-assessment/findings.json + XLSX, prints summary
python3 scripts/cloud_security_report.py           # writes security-assessment/Cloud-Security-Assessment-Report.md
python3 -c "import openpyxl; wb=openpyxl.load_workbook('security-assessment/Cloud-Security-Findings-Tracker.xlsx'); [print(ws.title, ws.max_row) for ws in wb]"

See security-assessment/README.md for options (--baseline, --dispositions, --skip-*, --include-generated-json, --fail-on-incomplete).

Link to Devin session: https://app.devin.ai/sessions/8d1eec8831044481a04ed15d422c6da2
Open in Devin Desktop: https://app.devin.ai/desktop/session/8d1eec8831044481a04ed15d422c6da2?variant=devin
Requested by: @erikrozi

---

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR that start with 'DevinAI' or '@devin'.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment, CI, and merge conflict monitoring

---

Original prompt from Devin Bot

Repository: COG-GTM/aws-cloudformation-templates (a fork of the AWS CloudFormation sample templates — ~300 CloudFormation YAML/JSON templates organized by service directory: APIGateway/, EC2/, RDS/, S3/, VPC/, EKS/, Lambda/, Solutions/, etc., plus a small Terraform module under EC2/terraform/). Lint command: cfn-lint <template.yaml>. Terraform check: terraform -chdir=EC2/terraform init -backend=false && terraform -chdir=EC2/terraform validate.

You are acting as the Cloud Computing Specialist / Enterprise Architect on a Department of the Air Force cloud-security engineering task. Treat this repository as the infrastructure-as-code baseline for a set of Government cloud workloads and produce a Cloud Security Assessment Report in the form the Government expects from CDRL A008 ("Cloud Security Assessment Report — MS Word/PDF report and MS Excel findings tracker"), then remediate the highest-severity findings in a second PR.

#``# Performance standard you must meet (from the tasking)
"Findings shall be risk-ranked, evidence-based, traceable to the applicable baseline or requirement, and include recommended remediation or disposition."

#``# PR 1 — Assessment (no template changes)

Create security-assessment/ at the repo root containing:

1. scripts/cloud_security_assessment.py (repo-root scripts/) — a reusable Python 3 runner that:
   • Runs checkov (install with pip) across every CloudFormation template and the Terraform module, and cfn_nag if it is installable (Ruby gem; skip gracefully if not) and cfn-lint.
   • Adds its own rule pack for items those tools miss or that the Government baseline emphasizes: encryption at rest for every storage/database/queue/topic/log resource (S3, EBS, RDS, DynamoDB, EFS, SQS, SNS, CloudWatch Logs, Kinesis/Firehose), KMS customer-managed keys vs AWS-managed, TLS-in-transit enforcement (S3 bucket policies with aws:SecureTransport, ELB/ALB listener protocols and TLS security policies, RDS `rds.force_ss... (5985 chars truncated...)

[devin-ai-integration]

Devin Review found 4 new potential issues.