Citrine build csms GH action is broken

[Abby-Wheelis]

After updating to the October release -- integration with citrine is broken again!

The rest of the actions complete successfully, but the action for citrine fails. I think this is the best capture of the problem:

#13 44.64 10:30:11 PM - Project '00_Base/tsconfig.json' is out of date because output file '00_Base/tsconfig.tsbuildinfo' does not exist
#13 44.64 
#13 44.65 10:30:11 PM - Building project '/usr/local/apps/citrineos/00_Base/tsconfig.json'...
#13 44.65 
#13 68.34 00_Base/src/interfaces/api/AbstractModuleApi.ts(398,41): error TS2589: Type instantiation is excessively deep and possibly infinite.
#13 68.34 00_Base/src/interfaces/api/AbstractModuleApi.ts(398,57): error TS2345: Argument of type 'ZodEffects<ZodObject<{ env: ZodEnum<["development", "production"]>; centralSystem: ZodObject<{ host: ZodString; port: ZodNumber; }, "strip", ZodTypeAny, { host: string; port: number; }, { ...; }>; ... 8 more ...; rbacRulesDir: ZodOptional<...>; }, "strip", ZodTypeAny, { ...; }, { ...; }>, { ...; }, { ...; }>' is not assignable to parameter of type 'ZodType<any, ZodTypeDef, any>'.
#13 68.34   The types returned by '_refinement(...)' are incompatible between these types.
#13 68.34     Type 'ZodEffects<ZodEffects<ZodObject<{ env: ZodEnum<["development", "production"]>; centralSystem: ZodObject<{ host: ZodString; port: ZodNumber; }, "strip", ZodTypeAny, { host: string; port: number; }, { ...; }>; ... 8 more ...; rbacRulesDir: ZodOptional<...>; }, "strip", ZodTypeAny, { ...; }, { ...; }>, { ...; }, { ...;...' is missing the following properties from type 'ZodEffects<ZodType<any, ZodTypeDef, any>, any, any>': _type, _output, _input, _def, and 33 more.
#13 68.35 10:30:34 PM - Project '01_Data/tsconfig.json' is out of date because output file '01_Data/tsconfig.tsbuildinfo' does not exist
#13 68.35 
#13 68.35 10:30:34 PM - Building project '/usr/local/apps/citrineos/01_Data/tsconfig.json'...
#13 68.35 
#13 71.04 10:30:37 PM - Project '02_Util/tsconfig.json' is out of date because output file '02_Util/tsconfig.tsbuildinfo' does not exist
#13 71.04 
#13 71.04 10:30:37 PM - Building project '/usr/local/apps/citrineos/02_Util/tsconfig.json'...
#13 71.04 
#13 74.45 10:30:40 PM - Project '03_Modules/Certificates/tsconfig.json' is out of date because output file '03_Modules/Certificates/tsconfig.tsbuildinfo' does not exist
#13 74.45 
#13 74.45 10:30:40 PM - Building project '/usr/local/apps/citrineos/03_Modules/Certificates/tsconfig.json'...
#13 74.45 
#13 74.91 10:30:41 PM - Project '03_Modules/Configuration/tsconfig.json' is out of date because output file '03_Modules/Configuration/tsconfig.tsbuildinfo' does not exist
#13 74.91 
#13 74.91 10:30:41 PM - Building project '/usr/local/apps/citrineos/03_Modules/Configuration/tsconfig.json'...
#13 74.91 
#13 75.50 10:30:41 PM - Project '03_Modules/EVDriver/tsconfig.json' is out of date because output file '03_Modules/EVDriver/tsconfig.tsbuildinfo' does not exist
#13 75.50 
#13 75.50 10:30:41 PM - Building project '/usr/local/apps/citrineos/03_Modules/EVDriver/tsconfig.json'...
#13 75.50 
#13 75.87 10:30:42 PM - Project '03_Modules/Monitoring/tsconfig.json' is out of date because output file '03_Modules/Monitoring/tsconfig.tsbuildinfo' does not exist
#13 75.87 
#13 75.87 10:30:42 PM - Building project '/usr/local/apps/citrineos/03_Modules/Monitoring/tsconfig.json'...
#13 75.87 
#13 76.17 10:30:42 PM - Project '03_Modules/OcppRouter/tsconfig.json' is out of date because output file '03_Modules/OcppRouter/tsconfig.tsbuildinfo' does not exist
#13 76.17 
#13 76.17 10:30:42 PM - Building project '/usr/local/apps/citrineos/03_Modules/OcppRouter/tsconfig.json'...
#13 76.17 
#13 76.59 10:30:42 PM - Project '03_Modules/Reporting/tsconfig.json' is out of date because output file '03_Modules/Reporting/tsconfig.tsbuildinfo' does not exist
#13 76.59 
#13 76.59 10:30:42 PM - Building project '/usr/local/apps/citrineos/03_Modules/Reporting/tsconfig.json'...
#13 76.59 
#13 76.85 10:30:43 PM - Project '03_Modules/SmartCharging/tsconfig.json' is out of date because output file '03_Modules/SmartCharging/tsconfig.tsbuildinfo' does not exist
#13 76.85 
#13 76.85 10:30:43 PM - Building project '/usr/local/apps/citrineos/03_Modules/SmartCharging/tsconfig.json'...
#13 76.85 
#13 77.16 10:30:43 PM - Project '03_Modules/Tenant/tsconfig.json' is out of date because output file '03_Modules/Tenant/tsconfig.tsbuildinfo' does not exist
#13 77.16 
#13 77.16 10:30:43 PM - Building project '/usr/local/apps/citrineos/03_Modules/Tenant/tsconfig.json'...
#13 77.16 
#13 77.27 10:30:43 PM - Project '03_Modules/Transactions/tsconfig.json' is out of date because output file '03_Modules/Transactions/tsconfig.tsbuildinfo' does not exist
#13 77.27 
#13 77.27 10:30:43 PM - Building project '/usr/local/apps/citrineos/03_Modules/Transactions/tsconfig.json'...
#13 77.27 
#13 77.71 10:30:44 PM - Project 'Server/tsconfig.json' is out of date because output file 'Server/tsconfig.tsbuildinfo' does not exist
#13 77.71 
#13 77.71 10:30:44 PM - Building project '/usr/local/apps/citrineos/Server/tsconfig.json'...
#13 77.71 
#13 78.39 10:30:44 PM - Project 'tsconfig.json' is out of date because output file 'dist/tsconfig.tsbuildinfo' does not exist
#13 78.39 
#13 78.39 10:30:44 PM - Building project '/usr/local/apps/citrineos/tsconfig.json'...
#13 78.39 
#13 79.75 10:30:46 PM - Updating unchanged output timestamps of project '/usr/local/apps/citrineos/tsconfig.json'...
#13 79.75 
#13 ERROR: process "/bin/sh -c npm run install-all && npm run build" did not complete successfully: exit code: 2
------
 > [linux/amd64 build 4/7] RUN npm run install-all && npm run build:
77.71 10:30:44 PM - Project 'Server/tsconfig.json' is out of date because output file 'Server/tsconfig.tsbuildinfo' does not exist
77.71 
77.71 10:30:44 PM - Building project '/usr/local/apps/citrineos/Server/tsconfig.json'...
77.71 
78.39 10:30:44 PM - Project 'tsconfig.json' is out of date because output file 'dist/tsconfig.tsbuildinfo' does not exist
78.39 
78.39 10:30:44 PM - Building project '/usr/local/apps/citrineos/tsconfig.json'...
78.39 
79.75 10:30:46 PM - Updating unchanged output timestamps of project '/usr/local/apps/citrineos/tsconfig.json'...
79.75 
------
deploy.Dockerfile:7
--------------------
   5 |     
   6 |     COPY . .
   7 | >>> RUN npm run install-all && npm run build
   8 |     
   9 |     RUN echo "Copying data and hasura metadata folders..."
--------------------
ERROR: failed to build: failed to solve: process "/bin/sh -c npm run install-all && npm run build" did not complete successfully: exit code: 2

It looks like a missing config? I'll run locally and see if I can get a better grasp of the issue.

[Abby-Wheelis]

#13 68.34 00_Base/src/interfaces/api/AbstractModuleApi.ts(398,41): error TS2589: Type instantiation is excessively deep and possibly infinite.
#13 68.34 00_Base/src/interfaces/api/AbstractModuleApi.ts(398,57): error TS2345: Argument of type 'ZodEffects<ZodObject<{ env: ZodEnum<["development", "production"]>; centralSystem: ZodObject<{ host: ZodString; port: ZodNumber; }, "strip", ZodTypeAny, { host: string; port: number; }, { ...; }>; ... 8 more ...; rbacRulesDir: ZodOptional<...>; }, "strip", ZodTypeAny, { ...; }, { ...; }>, { ...; }, { ...; }>' is not assignable to parameter of type 'ZodType<any, ZodTypeDef, any>'.
#13 68.34   The types returned by '_refinement(...)' are incompatible between these types.
#13 68.34     Type 'ZodEffects<ZodEffects<ZodObject<{ env: ZodEnum<["development", "production"]>; centralSystem: ZodObject<{ host: ZodString; port: ZodNumber; }, "strip", ZodTypeAny, { host: string; port: number; }, { ...; }>; ... 8 more ...; rbacRulesDir: ZodOptional<...>; }, "strip", ZodTypeAny, { ...; }, { ...; }>, { ...; }, { ...;...' is missing the following properties from type 'ZodEffects<ZodType<any, ZodTypeDef, any>, any, any>': _type, _output, _input, _def, and 33 more.
#13 68.35 10:30:34 PM - Project '01_Data/tsconfig.json' is out of date because output file '01_Data/tsconfig.tsbuildinfo' does not exist
#13 68.35 

Ah, no, it's a type thing again, let me see if I can find a workaround.

[Abby-Wheelis]

This issue has been reported: citrineos/citrineos#165

The fix suggested worked for me - I'll stick it in as a patch so we don't have to wait for an update

[Abby-Wheelis]

Merging that update did fix the issues that I was having with the build action, but the core tests are still failing! One looks like an ipv6 issue, so I'll ignore it for now, but the other looks like a broken test!

I wonder why the tests aren't run against PRs to this repo? The other actions are.

[Abby-Wheelis]

Here's the error:

DEBUG    root:everest_core.py:209   2025-12-15 23:14:30.378173 [DEBG] iso15118_car    pybind11_init_everestpy(pybind11::module_&)::<lambda(const std::string&)> :: Reason: Offered energy transfer modes [<EnergyTransferModeEnum.AC_SINGLE_PHASE_CORE: 'AC_single_phase_core'>] not compatible with EnergyTransferModeEnum.AC_THREE_PHASE_CORE

[Abby-Wheelis]

The switch_to_single_phase.patch is the issue. When I reverse it the tests pass.

The issue is that while the tests pass, the charging doesn't start in the demo ui. So we'll either need to 1) eliminate the need to switch the phase in the UI-based demo, or 2) edit the test to accommodate the new phase setting. I think 1) makes more sense.

[Abby-Wheelis]

@catarial mentioned the patch was originally implemented because there was a bug where the phase setting was hard coded and didn't respond to updates in the config -- likely.

We don't see the fixme anymore, but aren't sure if setting it in the config works.

[Abby-Wheelis]

Having both sets of pushes does let both the tests and the UI demo work as expected, so I'll keep that update to the patch as a backup, but I think a better way would be to remove the patch entirely and set the phase with a config? I'll spend some time tomorrow figuring out if that's an option.

[Abby-Wheelis]

It looks like just reverting back to three phase and getting rid of the patch would also be an option, I wonder what motivated our switch to single phase in the first place?

Since we will use this in the US Testival, where we expect to have cars that
are set up to accept single phase.

Ok, so this is a US-specific choice, and I think we should keep it in that case.

[Abby-Wheelis]

I found updates to this section of the code base in the main branch of everest-core. With this update, both the tests and the demo UI run as expected. I'll update the patch!

        // Set up energy transfer modes for HLC. For now we only support either DC or AC, not both at the same time.
        std::vector<types::iso15118::EnergyTransferMode> transfer_modes;
        if (config.charge_mode == "AC") {
            types::iso15118::SetupPhysicalValues setup_physical_values;
            setup_physical_values.ac_nominal_voltage = config.ac_nominal_voltage;
            r_hlc[0]->call_set_charging_parameters(setup_physical_values);

            switch (hw_capabilities.max_phase_count_import) {
            case 3:
                transfer_modes.push_back(types::iso15118::EnergyTransferMode::AC_three_phase_core);
                [[fallthrough]];
            case 2:
                transfer_modes.push_back(types::iso15118::EnergyTransferMode::AC_two_phase);
                [[fallthrough]];
            case 1:
                transfer_modes.push_back(types::iso15118::EnergyTransferMode::AC_single_phase_core);
                break;
            default:
                break;
            }

This update is only 2 weeks old!

[Abby-Wheelis]

That did not seem to work once I merged it, but that might be because I did not update the tags. The tests passed when I rand them locally with my updated patches.

[Abby-Wheelis]

Ok, now there's some sort of a certificate issue -- when I run in the dev containers with the patches, it works. When I run from the image I pushed that SHOULD be the same as what I have in the dev container, it breaks:

Working

2025-12-18 22:14:50.866115 [INFO] evse_security:E  :: Verifying leaf certificate
2025-12-18 22:14:50.866208 [INFO] evse_security:E  :: Including trust anchor for leaf certificate: MO
2025-12-18 22:14:50.866240 [INFO] evse_security:E  :: Including trust anchor for leaf certificate: V2G
2025-12-18 22:14:50.867393 [INFO] evse_security:E  :: SHANKARI: Considering chain ""
2025-12-18 22:14:50.867462 [INFO] evse_security:E  :: SHANKARI: Considering cert UKSWI123456789G
2025-12-18 22:14:50.867669 [INFO] evse_security:E  :: SHANKARI: Considering cert PKI-Ext_CRT_MO_SUB2_VALID
2025-12-18 22:14:50.868002 [INFO] evse_security:E  :: SHANKARI: Considering cert PKI-Ext_CRT_MO_SUB1_VALID
2025-12-18 22:14:50.868240 [INFO] evse_security:E  :: Building new certificate hierarchy!
2025-12-18 22:14:50.868269 [INFO] evse_security:E  :: SHANKARI: Considering chain ""
2025-12-18 22:14:50.868278 [INFO] evse_security:E  :: SHANKARI: Considering cert UKSWI123456789G
2025-12-18 22:14:50.868492 [INFO] evse_security:E  :: SHANKARI: Considering cert PKI-Ext_CRT_MO_SUB2_VALID
2025-12-18 22:14:50.868946 [INFO] evse_security:E  :: SHANKARI: Considering cert PKI-Ext_CRT_MO_SUB1_VALID
2025-12-18 22:14:50.873012 [INFO] evse_security:E  :: Building new certificate hierarchy!
2025-12-18 22:14:50.873055 [INFO] evse_security:E  :: SHANKARI: Considering chain "/ext/dist/etc/everest/certs/ca/v2g/V2G_ROOT_CA.pem"
2025-12-18 22:14:50.873080 [INFO] evse_security:E  :: SHANKARI: Considering cert V2GRootCA
2025-12-18 22:14:50.874204 [INFO] evse_security:E  :: SHANKARI: Considering chain "/ext/dist/etc/everest/certs/ca/v2g/V2G_ROOT_CA.pem"
2025-12-18 22:14:50.874248 [INFO] evse_security:E  :: SHANKARI: Considering cert V2GRootCA
2025-12-18 22:14:50.875217 [INFO] evse_security:E  :: Building new certificate hierarchy!
2025-12-18 22:14:50.875239 [INFO] evse_security:E  :: SHANKARI: Considering chain "/ext/dist/etc/everest/certs/ca/mo/MO_ROOT_CA.pem"
2025-12-18 22:14:50.875250 [INFO] evse_security:E  :: SHANKARI: Considering cert MORootCA
2025-12-18 22:14:50.876422 [INFO] evse_security:E  :: SHANKARI: Considering chain "/ext/dist/etc/everest/certs/ca/mo/MO_ROOT_CA.pem"
2025-12-18 22:14:50.876455 [INFO] evse_security:E  :: SHANKARI: Considering cert MORootCA
2025-12-18 22:14:50.878128 [INFO] ocpp:OCPP201     :: Local contract validation result: Valid
2025-12-18 22:14:50.879678 [INFO] ocpp:OCPP201     :: Offline: contract Valid
2025-12-18 22:14:50.882663 [INFO] ocpp:OCPP201     :: Offline authorization due to OfflineTxForUnknownIdEnabled being enabled
2025-12-18 22:14:50.883801 [INFO] auth:Auth        :: Providing authorization to evse#1
2025-12-18 22:14:50.944501 [INFO] auth:Auth        :: Result for token: [redacted] hash: ACA498D730285149: USED_TO_START_TRANSACTION

Breaking

2025-12-18 22:15:32.426089 [INFO] evse_security:E  :: Verifying leaf certificate
2025-12-18 22:15:32.426187 [INFO] evse_security:E  :: Including trust anchor for leaf certificate: MO
2025-12-18 22:15:32.426207 [INFO] evse_security:E  :: Including trust anchor for leaf certificate: V2G
2025-12-18 22:15:32.427315 [INFO] evse_security:E  :: SHANKARI: Considering chain ""
2025-12-18 22:15:32.427365 [INFO] evse_security:E  :: SHANKARI: Considering cert UKSWI123456789G
2025-12-18 22:15:32.427606 [INFO] evse_security:E  :: SHANKARI: Considering cert PKI-Ext_CRT_MO_SUB2_VALID
2025-12-18 22:15:32.427819 [INFO] evse_security:E  :: SHANKARI: Considering cert PKI-Ext_CRT_MO_SUB1_VALID
2025-12-18 22:15:32.428035 [INFO] evse_security:E  :: Building new certificate hierarchy!
2025-12-18 22:15:32.428056 [INFO] evse_security:E  :: SHANKARI: Considering chain ""
2025-12-18 22:15:32.428067 [INFO] evse_security:E  :: SHANKARI: Considering cert UKSWI123456789G
2025-12-18 22:15:32.428287 [INFO] evse_security:E  :: SHANKARI: Considering cert PKI-Ext_CRT_MO_SUB2_VALID
2025-12-18 22:15:32.428504 [INFO] evse_security:E  :: SHANKARI: Considering cert PKI-Ext_CRT_MO_SUB1_VALID
2025-12-18 22:15:32.432907 [INFO] evse_security:E  :: Building new certificate hierarchy!
2025-12-18 22:15:32.432978 [INFO] evse_security:E  :: SHANKARI: Considering chain "/ext/dist/etc/everest/certs/ca/v2g/V2G_ROOT_CA.pem"
2025-12-18 22:15:32.432995 [INFO] evse_security:E  :: SHANKARI: Considering cert V2GRootCA
2025-12-18 22:15:32.433898 [INFO] evse_security:E  :: SHANKARI: Considering chain "/ext/dist/etc/everest/certs/ca/v2g/V2G_ROOT_CA.pem"
2025-12-18 22:15:32.433931 [INFO] evse_security:E  :: SHANKARI: Considering cert V2GRootCA
2025-12-18 22:15:32.435092 [INFO] evse_security:E  :: Building new certificate hierarchy!
2025-12-18 22:15:32.435162 [INFO] evse_security:E  :: SHANKARI: Considering chain "/ext/dist/etc/everest/certs/ca/mo/MO_ROOT_CA.pem"
2025-12-18 22:15:32.435200 [INFO] evse_security:E  :: SHANKARI: Considering cert MORootCA
2025-12-18 22:15:32.436124 [INFO] evse_security:E  :: SHANKARI: Considering chain "/ext/dist/etc/everest/certs/ca/mo/MO_ROOT_CA.pem"
2025-12-18 22:15:32.436147 [INFO] evse_security:E  :: SHANKARI: Considering cert MORootCA
2025-12-18 22:15:32.437810 [INFO] ocpp:OCPP201     :: Local contract validation result: Valid
2025-12-18 22:15:32.440678 [INFO] evse_security:E  :: SHANKARI: Considering chain ""
2025-12-18 22:15:32.440725 [INFO] evse_security:E  :: SHANKARI: Considering cert UKSWI123456789G
2025-12-18 22:15:32.440891 [INFO] evse_security:E  :: SHANKARI: Considering cert PKI-Ext_CRT_MO_SUB2_VALID
2025-12-18 22:15:32.441052 [INFO] evse_security:E  :: SHANKARI: Considering cert PKI-Ext_CRT_MO_SUB1_VALID
2025-12-18 22:15:32.441566 [INFO] evse_security:E  :: SHANKARI: Considering chain "/ext/dist/etc/everest/certs/ca/v2g/V2G_ROOT_CA.pem"
2025-12-18 22:15:32.441603 [INFO] evse_security:E  :: SHANKARI: Considering cert V2GRootCA
2025-12-18 22:15:32.441986 [INFO] evse_security:E  :: SHANKARI: Considering chain "/ext/dist/etc/everest/certs/ca/mo/MO_ROOT_CA.pem"
2025-12-18 22:15:32.442012 [INFO] evse_security:E  :: SHANKARI: Considering cert MORootCA
2025-12-18 22:15:32.447533 [INFO] ocpp:OCPP201     :: Online: Pass generated OCSP data to CSMS
2025-12-18 22:15:32.456431 [INFO] evse_manager_1:  ::                                     CAR ISO V2G AuthorizationReq2025-12-18 22:15:32.456443 [INFO] ocpp:OCPP201     :: CSMS idToken status: Unknown

2025-12-18 22:15:32.456483 [INFO] ocpp:OCPP201     :: CSMS certificate status: NoCertificateAvailable
2025-12-18 22:15:32.502082 [INFO] auth:Auth        :: Result for token: [redacted] hash: ACA498D730285149: REJECTED

It looks like the difference is that the working version isn't connected to the CSMS, so it works, but when the CSMS is connected, it fails to verify.