Skip to content

Convert vs sync scan to save to the mini data lake (CRASM-1064)#787

Merged
rapidray12 merged 23 commits intodevelopfrom
crasm-1064
Feb 24, 2025
Merged

Convert vs sync scan to save to the mini data lake (CRASM-1064)#787
rapidray12 merged 23 commits intodevelopfrom
crasm-1064

Conversation

@jayjaybunce
Copy link
Contributor

@jayjaybunce jayjaybunce commented Feb 4, 2025

🗣 Description

  • Mirrored all functionality from TS vsSyncScan.
  • Created /sync endpoint to support the LZ => DMZ data ingestiong
  • Added several utility functions for saving resources to the datalakes using Django models
  • Implemented several utility functions for chunking CSV data and other misc operations

💭 Motivation and context

Resolves Crasm-1064

🧪 Testing

Due to the complex configuration required to develop and test this PR, I have a separate branch that contains all the necessary testing config changes: crasm-1064-test

✅ Pre-approval checklist

  • This PR has an informative and human-readable title.
  • Changes are limited to a single goal - eschew scope creep!
  • All future TODOs are captured in issues, which are referenced
    in code comments.
  • All relevant type-of-change labels have been added.
  • I have read the CONTRIBUTING document.
  • These code changes follow cisagov code standards.
  • All relevant repo and/or project documentation has been updated
    to reflect the changes in this PR.
  • Tests have been added and/or modified to cover the changes in this PR.
  • All new and existing tests pass.

✅ Pre-merge checklist

  • Revert dependencies to default branches.
  • Finalize version.

✅ Post-merge checklist

  • Create a release.

Copy link
Contributor

@chrtorres chrtorres left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Contributor

@hawkishpolicy hawkishpolicy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@rapidray12 rapidray12 changed the title Convert vs sync scan to save to the mini data lake Convert vs sync scan to save to the mini data lake (CRASM-1064) Feb 24, 2025
@rapidray12 rapidray12 merged commit cb6ed38 into develop Feb 24, 2025
17 of 18 checks passed
@rapidray12 rapidray12 deleted the crasm-1064 branch February 24, 2025 20:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants