Skip to content

Optimize and Validate DNSTwist Scan#1469

Merged
cduhn17 merged 11 commits intodevelopfrom
jb-crasm-3651
Jan 26, 2026
Merged

Optimize and Validate DNSTwist Scan#1469
cduhn17 merged 11 commits intodevelopfrom
jb-crasm-3651

Conversation

@jayjaybunce
Copy link
Contributor

@jayjaybunce jayjaybunce commented Jan 22, 2026

🗣 Description

  1. Updated Blocklist API to accept bulk ips
  2. Updated "update_blocklist" scan to perform bulk inserts/updates and query blocklist.de concurrently to decrease runtime
  3. Updated DNSTwist to reduce runtime and improve data accuracy, using the updated /blocklist/check endpoint and performing concurrent operations when available and using bulk update/create operations when possible
  4. Fixed issue with DNSTwist sync that was returning 422 status code for org sync pushes.

💭 Motivation and context

CRASM-3651

🧪 Testing

✅ Pre-approval checklist

  • This PR has an informative and human-readable title.
  • Changes are limited to a single goal - eschew scope creep!
  • All future TODOs are captured in issues, which are referenced in code comments.
  • All relevant type-of-change labels have been added.
  • I have read the CONTRIBUTING document.
  • These code changes follow cisagov code standards.
  • All relevant repo and/or project documentation has been updated to reflect the changes in this PR.
  • Tests have been added and/or modified to cover the changes in this PR.
  • All new and existing tests pass.
  • Bump major, minor, patch, pre-release, and/or build versions as appropriate via the bump_version script if this repository is versioned and the changes in this PR warrant a version bump.
  • Create a pre-release (necessary if and only if the pre-release version was bumped).

✅ Pre-merge checklist

  • Revert dependencies to default branches.
  • Finalize version.

✅ Post-merge checklist

  • Create a release (necessary if and only if the version was bumped).

Copy link
Collaborator

@cduhn17 cduhn17 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The move to bulk checking is a great feature, but the current implementation introduces a significant performance risk for our database.

Performance (Blocker): The code performs Blocklist.objects.get(ip=ip) inside the for loop. This creates an N+1 query problem. Please refactor this to use Blocklist.objects.filter(ip__in=ip_addresses) to retrieve all data in a single database query.

Security: Please add a check to limit the length of ip_addresses (e.g., max 100) to prevent resource exhaustion attacks on the Lambda.

Typos: Please fix "Determing" -> "Determining" in the docstring.

@jyang2139
Copy link
Contributor

Don't see any issues other than the suggestions brought up by Craig.

Copy link
Collaborator

@cduhn17 cduhn17 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the updates.

@cduhn17 cduhn17 merged commit 036caac into develop Jan 26, 2026
18 of 19 checks passed
@cduhn17 cduhn17 deleted the jb-crasm-3651 branch January 26, 2026 19:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants