[SPIKE] Explore CBA results when using different solvers and tolerances

[measrainsey]

Issue to track exploration of how using different solvers (HiGHS vs Gurobi) and different solver tolerances affect the results as well as performance (memory, solve time) of the CBA

Runs

Will be focusing on 4 different runs:

• tol1e6-gurobi
  • MSV solver: gurobi-default, BarConvTol: 1.0e-06
  • RH solver: gurobi-simplex, FeasibilityTol: 1.0e-06, OptimalityTol: 1.0e-06
• tol1e6-highs
  • MSV solver: gurobi-default, BarConvTol: 1.0e-06
  • RH solver: highs-simplex, primal_feasibility_tolerance: 1.0e-06, dual_feasibility_tolerance: 1.0e-06
• tol1e8-gurobi
  • MSV solver: gurobi-default, BarConvTol: 1.0e-08
  • RH solver: gurobi-simplex, FeasibilityTol: 1.0e-08, OptimalityTol: 1.0e-08
• tol1e8-highs
  • MSV solver: gurobi-default, BarConvTol: 1.0e-08
  • RH solver: highs-simplex, primal_feasibility_tolerance: 1.0e-08, dual_feasibility_tolerance: 1.0e-08

All runs are with 1 hour resolution, for the NT scenario

Indicator Results

Changing the solver and the tolerance seems to have a really big impact on our results (and note that we only change the solver for the rolling horizon solve, as the MSV still uses Gurobi, just with different tolerances).

[measrainsey]

I've re-done my analysis for checking the accuracy of select projects, across different solvers and tolerances.

Runs

The runs are for the NT scenario, with 1H resolution. See the config below for full config.

The runs are all done with the latest master, as well as the feat/bio-dispatch (from #719) and fix/t339-itsi (from #751) branches merged onto there.

Run using the command snakemake --profile profiles/slurm -call cba --configfile config/config.tyndp.yaml config/config.zib.yaml.

The runs included are:

• gurobi-msv1e6-rh1e6
• highs-msv1e6-rh1e6
• gurobi-msv1e8-rh1e6
• gurobi-msv1e8-rh1e8
• highs-msv1e8-rh1e6
• highs-msv1e8-rh1e8

The naming convention is: [solver]-[msv-tolerance]-[rolling-horizon-tolerance].

• [solver]: the rolling horizon solver ONLY (gurobi-simplex or highs-simplex). The MSV uses gurobi-default no matter what.
• [msv-tolerance]: the tolerance for the MSV solve (either 1e-6 or 1e-8)
• [rolling-horizon-tolerance]: the tolerance for the rolling horizon solves (reference networks, project networks) (either 1e-6 or 1e-8)

Results

With our latest model implementations, it can be seen that the negative B1 indicators (at least for this selection of projects) are looking generally a lot better -- there are fewer projects that are negative, and if negative, the scale of the negativity is a lot smaller.

However, it can still be seen that both solver choice (Gurobi vs Highs for just the rolling horizon) and the solver tolerance can play a big impact on the results. Sometimes the sign of the B1 indicator would flip depending on the solver and tolerance used. I can't see a clear pattern of accuracy across the solvers and tolerances, though (e.g., one solver-tolerance combination giving closer results to the TYNDP 2024).

2030:

2040:

Config

The config file used for the ZIB runs is (with changing solver settings for MSV and rolling horizon):

run:
  name: "NT"

clustering:
  temporal:
    resolution_sector: 1H

solving:
  options:
    assign_all_duals: true
    include_objective_constant: false
  solver:
    name: gurobi
    options: gurobi-default
  partition: big
  mem_mb: 60000
  runtime: 12h

cba:
  projects: [t4, t227, t350, t342, t1167, t1109, t127, t341, t1112, t328, t1235, t150, t1207, t339]

  msv_extraction:
    solving:
      options:
        include_objective_constant: false
      solver:
        name: gurobi
        options: gurobi-default
      solver_options:
        gurobi-default:
          BarConvTol: 1.0e-06
      partition: big
      mem_mb: 60000
      runtime: 12h

  # Rolling horizon solving settings
  solving:
    options:
      include_objective_constant: false
    partition: big
    mem_mb: 60000
    runtime: 12h

    solver:
      name: gurobi
      options: gurobi-simplex

    solver_options:
      gurobi-simplex:
        FeasibilityTol: 1.0e-06
        OptimalityTol: 1.0e-06

[lisazeyen]

thanks @measrainsey for preparing the spike and constantly updating with new results!

Can we just check for at least one project (e.g. t350 in 2040), how opex is differing between the different runs and document the findings here? is there a lot of load shedding? Then we can close the issue. I think we should tighten the solver tolerance for the rolling horizon to 1e-8.

[lisazeyen]

closing the issue, we will experiment with lower load shedding costs in a follow up issue #734 to see if that reduces the numerical noise + tighten solver tolerance.