Skip to content

Increase e2e coverage for NewPICredit and Prepayment processors #308

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
marcoagonzales007 wants to merge 3 commits into
base: main
Choose a base branch
from
+20 −7
Open

Increase e2e coverage for NewPICredit and Prepayment processors #308

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
71112d4
Increase e2e coverage for NewPICredit and Prepayment processors
marcoagonzales007 Jun 29, 2026
bf5c42f
Copy mutable test files into workspace to avoid modifying committed data
marcoagonzales007 Jul 3, 2026
3d6bcc8
Copy mutable test files into workspace subdirectory to avoid modifyin…
marcoagonzales007 Jul 3, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions process_report/tests/e2e/test_data/test_PI.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,5 +2,6 @@ PI,First Invoice Month,Initial Credits,1st Month Used,2nd Month Used
pi5@harvard.edu,2025-06,0.00,0.00,0.00
pi6@mit.edu,2025-06,0.00,0.00,0.00
pi4@example.edu,2025-06,0.00,0.00,0.00
pi1@bu.edu,2025-04,1000.00,300.00,0.00
pi2@harvard.edu,2025-04,1000.00,300.00,0.00
pi1@bu.edu,2025-06,1000.00,320.00,0.00

@QuanMPhm QuanMPhm Jun 30, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Based on your PR description and the changes you made, it seems that since the e2e test will now actually apply the New-PI credit and prepayments to some invoices, mutable test files (test_PI.csv and test_prepay_debits.py) will now be changed after each run of the e2e test. Could you edit the code in test_e2e_pipeline.py so that the aforementioned files are not changed after each run? My suggestion comes from the idea that what we commit into the repo should only be the test cases, not their results or side effects. Investigate the ways you can accomplish this, and let me know what solution you'll like to implement.

Also, I'm curious why I don't see changes to test_prepay_debits.py in this PR diff. Was it not changed when you ran the e2e test on your machine?

@marcoagonzales007 marcoagonzales007 Jun 30, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for your feedback, and I think It was changed by the pipeline itself during the run. I reverted it thinking it was a stray change, but I now realize that's the side effect of the problem you have just described. I'll work on test_e2e_pipeline.py so that the files are not changed after each run. my plan for this is to copy the mutable files into the temp workspace and point the pipeline at these copies, so the test data stays untouched. Let me know if that sounds good. Thank you.

@QuanMPhm QuanMPhm Jul 1, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll work on test_e2e_pipeline.py so that the files are not changed after each run. my plan for this is to copy the mutable files into the temp workspace and point the pipeline at these copies,

Nice! That was the solution I had in mind too. Let me know when you've amended this PR.

pi2@harvard.edu,2025-05,1000.00,300.00,35.00
pi7@mit.edu,2025-04,1000.00,1000.00,0.00
6 changes: 6 additions & 0 deletions process_report/tests/e2e/test_data/test_coldfront_api_data.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,5 +37,11 @@
"Allocated Project Name": "P3",
"Allocated Project ID": "P3ID"
}
},
{
"id" : 1,
"project" : {"pi" : "pi7@mit.edu"},
"resource": { "name": "shift" },
"attributes": { "Allocated Project Name": "P7", "Allocated Project ID": "P7ID" }
}
]
1 change: 1 addition & 0 deletions process_report/tests/e2e/test_data/test_invoices/test_NERC OpenShift 2025-04.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,3 +4,4 @@ Invoice Month,Project - Allocation,Project - Allocation ID,Manager (PI),Cluster
2024-01,P2ID,P2ID,,shift,,,,,280,OpenShift CPU,0.013,200
2024-01,P3ID,P3ID,,shift,,,,,280,Free CPU,0.013,300
2024-01,P9ID,P9ID,,shift,,,,,280,OpenShift CPU,0.013,3000
2024-01,P7ID,P7ID,,shift,,,,,280,OpenShift CPU,0.013,500
2 changes: 1 addition & 1 deletion process_report/tests/e2e/test_data/test_prepay_projects.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
Group Name,Project,Start Date,End Date
TestGroup1,TestProject1,2024-01,2024-12
TestGroup1,P7,2024-01,2025-12
TestGroup2,TestProject2,2023-12,2024-06
13 changes: 9 additions & 4 deletions process_report/tests/e2e/test_e2e_pipeline.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
import os
import shutil
from pathlib import Path
import pandas as pd
import pytest
Expand Down Expand Up @@ -116,11 +117,15 @@ def _prepare_pipeline_execution(
env["FETCH_FROM_S3"] = "false"
env["UPLOAD_TO_S3"] = "false"
env["invoice_path_template"] = str(test_files["test_invoice_dir"])

env["PI_REMOTE_FILEPATH"] = str(test_files["test_PI.csv"])
input_dir = workspace / "input_data"
input_dir.mkdir(exist_ok=True)
pi_file_copy = input_dir / "test_PI.csv"
shutil.copy(test_files["test_PI.csv"], pi_file_copy)
env["PI_REMOTE_FILEPATH"] = str(pi_file_copy)
env["ALIAS_REMOTE_FILEPATH"] = str(test_files["test_alias.csv"])
env["PREPAY_DEBITS_REMOTE_FILEPATH"] = str(test_files["test_prepay_debits.csv"])

prepay_debits_copy = input_dir / "test_prepay_debits.csv"
shutil.copy(test_files["test_prepay_debits.csv"], prepay_debits_copy)
env["PREPAY_DEBITS_REMOTE_FILEPATH"] = str(prepay_debits_copy)
env["PREPAY_CREDITS_FILEPATH"] = str(test_files["test_prepay_credits.csv"])
env["PREPAY_PROJECTS_FILEPATH"] = str(test_files["test_prepay_projects.csv"])
env["PREPAY_CONTACTS_FILEPATH"] = str(test_files["test_prepay_contacts.csv"])
Expand Down
Loading