Invoicing Automation - EPIC

[jimmysway]

Motivation

There are still many steps that Kim does to the invoicing outputs that we wish to automate.

This EPIC details the issues related to Invoicing Automation. We wanted to create a centralized source of truth for invoicing information that we can run analytical queries in the scope of entire projects rather than being restricted to monthly csvs.

The issues in this epic will cover everything that needs to be done implementation wise to make the monthly Invoicing run automatically upload accurate, holistic that encompasses all aspects of billing for the MOC.

Completion Criteria

Part 1: Invoicing Automation

• Padding Credit Codes with '000' - Issue Invoicing Automation - Padding Credit Codes and Handling Special Cases #268
• Manually adding Credit code 0005 to emre_keskin@harvard.edu's Storage so that the balance is 0 - - Issue Invoicing Automation - Padding Credit Codes and Handling Special Cases #268
• Moving griot-grits-aa488b rows from Non-Billable to Combined - Issue Invoicing Automation - Padding Credit Codes and Handling Special Cases #268
• Adding Bare Metal into invoicing - Issue Implement the Bare Metal Usage invoice and processor #152
• Move any new courses (currently CS210, CS599, DS551) from MGHPCC & Combined to Non-Billable - Issue Projects that belong to courses are now non-billable #240
• Manually calculate new NFS-CSI rows for NAIRR projects - Update Storage types in ColdFront nerc-project/operations#1391

Part 2: Iceberg Automation

• Adding a combined billable + non-billable sheet for Iceberg Upload - Issue Add combined invoice (billable + nonbillable) for Iceberg export #261
• Uploading to Iceberg from memory during monthly invoicing pipeline run - Issue Feature: Append dataframe to Iceberg during invoicing pipeline run #259

Description

Everything under "Part 2: Iceberg Automation" will be blocked until everything in "Part 1: Invoicing Automation" is complete

Completion dates

Desired - 2026-03-14
Required - 2026-03-21

[jimmysway]

Everything under "Part 2: Iceberg Automation" will be blocked until everything in "Part 1: Invoicing Automation" is complete

[jimmysway]

@QuanMPhm @knikolla @naved001

For the Bare Metal invoicing, I am not sure the PR I linked is the correct one or if there are more moving parts to the Bare Metal Invoicing effort.

I also don't know if there are plans to pull in prometheus metrics for some of the Bare Metal items. If not we should have a discussion about that @joachimweyl

[jimmysway]

Epic added to Zenhub Epics instead