Address EIA-930 irregularities

[zschira]

Impacted table(s)

core_eia930__hourly_operations, out_eia930__hourly_operations

How severe is this data bug?

Medium

Identify impacted data

Many issues are pervasive and not limited to specific filers. Issues with new fuel categories creates an inconsistency between recent new years of data and historic data.

Describe the bug

In our ecosystem survey, @grgmiller brought to our attention a number of irregularities with EIA 930 data including:

• Bad timestamps where BAs have reported in the incorrect time zone
• Issues with conservation of energy (ideally demand should be equal to net generation + interchange)
• Inconsistencies between neighboring BAs

[zaneselvans]

If we want a version of the EIA-930 data that respects conservation of energy while minimizing the perturbations to the original data, we might want to look at Grid Emissions project from @jdechalendar for one example of managing irregularities in the reported data, and the related paper A Physics-informed data reconciliation framework for real-time electricity and emissions tracking for a method of reconciling the reported data with physical constraints. But IIRC there's a trade-off between making all the interchange physically consistent, and reflecting the originally reported data. @grgmiller are you already familiar with that paper / data product?

[grgmiller]

Yes, we currently in OGE perform a lot of this reconciliation and cleaning from this paper, although we are currently using it incorrectly (it should not be used for bulk data reconciliation, only for minor tweaks on the edges)

Some resources:

• Our write-up of 930 data cleaning: https://docs.singularity.energy/docs/open-grid-emissions/cleaning_eia930-eia-930-data
• Manual adjustments to 930 data: https://github.com/singularity-energy/open-grid-emissions/blob/f387a4cc52af493644857677d03c6f8e7497fbec/src/oge/eia930.py#L559
• Our general EIA-930 handling code: https://github.com/singularity-energy/open-grid-emissions/blob/main/src/oge/eia930.py

A collection of open issues related to 930 data cleaning:

• Handling missing data in EIA-930 singularity-energy/open-grid-emissions#113
• Fix / clean up EIA-930 timestamp corrections singularity-energy/open-grid-emissions#301
• Re-assess cleaning of EIA-930 data singularity-energy/open-grid-emissions#220
• Adjust physics-based reconciliation of EIA-930 data singularity-energy/open-grid-emissions#240

The data quality issues may have also gotten better over time, but there is also a transition going on in the data itself where new fuel categories got introduced, so another helpful standardization effort may be some sort of mapping/consistency between the old fuel types and the new fuel types. Here's our effort at mapping the old to new fuel types:

[aesharpe]

@grgmiller we put this in the icebox for now because we're in the middle of running quarterlies, but we'll circle back when we're done with that to see if we have time to address this this quarter.

From your perspective (and in the scheme of things) how high a priority fix is this? It seems like you have some workarounds in place, but would be helpful to know if this is top of mind. Thanks for including all this detail!