FUNCTIONALITY_TEST_SCOPFLOW_HIOP_TESTSUITE not converging on Frontier

[nkoukpaizan]

I disabled FUNCTIONALITY_TEST_SCOPFLOW_HIOP_TESTSUITE in #183, because HiOp PriDec did not converge in finite time on Frontier. It would be good to investigate why this test is not converging.

[nkoukpaizan]

Default log with a 300 sec timeout

With MA57, decreasing the Ipopt verbosity and increasing the PriDec verbosity, we can see how the solution bounces:

[ExaGO] Creating SCOPFlow Functionality Test
[ExaGO] Creating SCOPFLOW Functionality Test
Test Description: case9mod.m with HIOP solver/9 contingencies
[ExaGO] SCOPFLOW: Application created
[ExaGO] SCOPFLOW running with 10 subproblems (base case + 9 contingencies)
[ExaGO] SCOPFLOW: Setup completed
===============
Hiop Primal Decomposition SOLVER
===============
total number of recourse problems  9

******************************************************************************
This program contains Ipopt, a library for large-scale nonlinear optimization.
 Ipopt is released as open source code under the Eclipse Public License (EPL).
         For more information visit https://github.com/coin-or/Ipopt
******************************************************************************

real rval -3.590978747128e-01
alpd 1.000000000000e-05
solving full problem starts, iteration 0
real rval -3.590978747128e-01
previous base  4.144101531615e+03, current base 4.144101531607e+03, previous val  -3.590978747128e-01, real val -3.590978747128e-01, predicted val -3.590978747136e-01, rho_k 9.998906386702e-01
decreasing alpha ratio or decreasing minimum for quadratic coefficient
alpd 1.000000000000e-05
convergence measure 1.020370528818e-07
function val convergence measure 8.316341366866e-09
iteration           objective                   residual                   step_size                   convg
1              4.143742433732e+03            8.316341366866e-09           2.085976308132e-08         1.020370528818e-07
solving full problem starts, iteration 1
reaching error tolerance, successfully found solution
[ExaGO] -- FAIL


Test Description: case_ACTIVSg200.m with HIOP solver/20 contingencies
[ExaGO] SCOPFLOW: Application created
[ExaGO] SCOPFLOW running with 21 subproblems (base case + 20 contingencies)
[ExaGO] SCOPFLOW: Setup completed
===============
Hiop Primal Decomposition SOLVER
===============
total number of recourse problems  20
real rval -7.979952771520e+00
alpd 1.000000000000e-05
solving full problem starts, iteration 0
real rval 3.025661179515e+03
previous base  2.754959175033e+04, current base 4.180320737650e+04, previous val  -7.979952771520e+00, real val 3.025661179515e+03, predicted val -2.005871221624e+04, rho_k -2.982043978080e+00
increasing alpha ratio or increasing minimum for quadratic coefficient
This step needs to be rejected.
alpd 1.000000000000e+06
convergence measure 1.238538702132e+00
function val convergence measure 9.559764478999e+06
iteration           objective                   residual                   step_size                   convg
1              4.482886855602e+04            9.559764478999e+06           4.373913944222e+00         1.238538702132e+00
solving full problem starts, iteration 1
real rval -7.979952761702e+00
previous base  4.180320737650e+04, current base 3.469669835165e+04, previous val  3.025661179515e+03, real val -7.979952761702e+00, predicted val -1.255099322741e+06, rho_k 8.014462344376e-03
increasing alpha ratio or increasing minimum for quadratic coefficient
This step needs to be rejected.
alpd 1.000000000000e-05
convergence measure 1.517420479196e+00
function val convergence measure 2.371480804668e+06
iteration           objective                   residual                   step_size                   convg
2              3.468871839889e+04            2.371480804668e+06           1.487447015348e+00         1.517420479196e+00
solving full problem starts, iteration 2
real rval -7.979952708171e+00
previous base  3.469669835165e+04, current base 3.437695074838e+04, previous val  -7.979952761702e+00, real val -7.979952708171e+00, predicted val -6.331888539604e+06, rho_k 5.049549725366e-05
increasing alpha ratio or increasing minimum for quadratic coefficient
This step needs to be rejected.
alpd 1.000000000000e-05
convergence measure 1.000000038589e+00
function val convergence measure 6.332200307254e+06
iteration           objective                   residual                   step_size                   convg
3              3.436897079567e+04            6.332200307254e+06           2.849836328140e+00         1.000000038589e+00
solving full problem starts, iteration 3
real rval 9.149566770786e+05
previous base  3.437695074838e+04, current base 4.641765031250e+06, previous val  -7.979952708171e+00, real val 9.149566770786e+05, predicted val -8.237633312502e+13, rho_k -6.703810229114e-08
increasing alpha ratio or increasing minimum for quadratic coefficient
This step needs to be rejected.
alpd 1.000000000000e+06
convergence measure 1.527576900198e+00
function val convergence measure 8.237630883154e+13
iteration           objective                   residual                   step_size                   convg
4              5.556721708329e+06            8.237630883154e+13           6.274723993230e+00         1.527576900198e+00
solving full problem starts, iteration 4
real rval -7.980494309649e+00
previous base  4.641765031250e+06, current base 3.437693750000e+04, previous val  9.149566770786e+05, real val -7.980494309649e+00, predicted val -8.395505293986e+13, rho_k 6.577748681825e-08
increasing alpha ratio or increasing minimum for quadratic coefficient
This step needs to be rejected.
alpd 1.000000000000e-05
convergence measure 1.948976587343e+00
function val convergence measure 8.395507814829e+13
iteration           objective                   residual                   step_size                   convg
5              3.436895700569e+04            8.395507814829e+13           6.274724053782e+00         1.948976587343e+00
solving full problem starts, iteration 5
real rval 9.149566770786e+05
previous base  3.437693750000e+04, current base 4.641765015625e+06, previous val  -7.980494309649e+00, real val 9.149566770786e+05, predicted val -9.782668451756e+13, rho_k -5.645037469093e-08
increasing alpha ratio or increasing minimum for quadratic coefficient
This step needs to be rejected.
alpd 1.000000000000e+06
convergence measure 2.053752313598e+00
function val convergence measure 9.782666022408e+13
iteration           objective                   residual                   step_size                   convg
6              5.556721692704e+06            9.782666022408e+13           6.274724053782e+00         2.053752313598e+00
solving full problem starts, iteration 6
real rval -7.980494309649e+00
previous base  4.641765015625e+06, current base 3.437693750000e+04, previous val  9.149566770786e+05, real val -7.980494309649e+00, predicted val -8.395505576262e+13, rho_k 6.577748442055e-08
increasing alpha ratio or increasing minimum for quadratic coefficient
This step needs to be rejected.
alpd 1.000000000000e-05
convergence measure 1.948976578885e+00
function val convergence measure 8.395508097105e+13
iteration           objective                   residual                   step_size                   convg
7              3.436895700569e+04            8.395508097105e+13           6.274724053782e+00         1.948976578885e+00
solving full problem starts, iteration 7
real rval 9.149566770786e+05
previous base  3.437693750000e+04, current base 4.641765015625e+06, previous val  -7.980494309649e+00, real val 9.149566770786e+05, predicted val -9.782668451756e+13, rho_k -5.645037469093e-08
increasing alpha ratio or increasing minimum for quadratic coefficient
This step needs to be rejected.
alpd 1.000000000000e+06
convergence measure 2.053752313598e+00
function val convergence measure 9.782666022408e+13
iteration           objective                   residual                   step_size                   convg
8              5.556721692704e+06            9.782666022408e+13           6.274724053782e+00         2.053752313598e+00
solving full problem starts, iteration 8
real rval -7.980494309649e+00
previous base  4.641765015625e+06, current base 3.437693750000e+04, previous val  9.149566770786e+05, real val -7.980494309649e+00, predicted val -8.395505576262e+13, rho_k 6.577748442055e-08
increasing alpha ratio or increasing minimum for quadratic coefficient
This step needs to be rejected.
alpd 1.000000000000e-05
convergence measure 1.948976578885e+00
function val convergence measure 8.395508097105e+13
iteration           objective                   residual                   step_size                   convg
9              3.436895700569e+04            8.395508097105e+13           6.274724053782e+00         1.948976578885e+00
solving full problem starts, iteration 9
real rval 9.149566770786e+05
previous base  3.437693750000e+04, current base 4.641765015625e+06, previous val  -7.980494309649e+00, real val 9.149566770786e+05, predicted val -9.782668451756e+13, rho_k -5.645037469093e-08
increasing alpha ratio or increasing minimum for quadratic coefficient
This step needs to be rejected.
alpd 1.000000000000e+06
convergence measure 2.053752313598e+00
function val convergence measure 9.782666022408e+13
iteration           objective                   residual                   step_size                   convg
10              5.556721692704e+06            9.782666022408e+13           6.274724053782e+00         2.053752313598e+00
solving full problem starts, iteration 10
real rval -7.980494309649e+00
previous base  4.641765015625e+06, current base 3.437693750000e+04, previous val  9.149566770786e+05, real val -7.980494309649e+00, predicted val -8.395505576262e+13, rho_k 6.577748442055e-08
increasing alpha ratio or increasing minimum for quadratic coefficient
This step needs to be rejected.
alpd 1.000000000000e-05
convergence measure 1.948976578885e+00
function val convergence measure 8.395508097105e+13
iteration           objective                   residual                   step_size                   convg
11              3.436895700569e+04            8.395508097105e+13           6.274724053782e+00         1.948976578885e+00
solving full problem starts, iteration 11
real rval 9.149566770786e+05
previous base  3.437693750000e+04, current base 4.641765015625e+06, previous val  -7.980494309649e+00, real val 9.149566770786e+05, predicted val -9.782668451756e+13, rho_k -5.645037469093e-08
increasing alpha ratio or increasing minimum for quadratic coefficient
This step needs to be rejected.
alpd 1.000000000000e+06
convergence measure 2.053752313598e+00
function val convergence measure 9.782666022408e+13
iteration           objective                   residual                   step_size                   convg
12              5.556721692704e+06            9.782666022408e+13           6.274724053782e+00         2.053752313598e+00
solving full problem starts, iteration 12
real rval -7.980494309649e+00
previous base  4.641765015625e+06, current base 3.437693750000e+04, previous val  9.149566770786e+05, real val -7.980494309649e+00, predicted val -8.395505576262e+13, rho_k 6.577748442055e-08
increasing alpha ratio or increasing minimum for quadratic coefficient
This step needs to be rejected.
alpd 1.000000000000e-05
convergence measure 1.948976578885e+00
function val convergence measure 8.395508097105e+13
iteration           objective                   residual                   step_size                   convg
13              3.436895700569e+04            8.395508097105e+13           6.274724053782e+00         1.948976578885e+00
solving full problem starts, iteration 13
real rval 9.149566770786e+05
previous base  3.437693750000e+04, current base 4.641765015625e+06, previous val  -7.980494309649e+00, real val 9.149566770786e+05, predicted val -9.782668451756e+13, rho_k -5.645037469093e-08
increasing alpha ratio or increasing minimum for quadratic coefficient
This step needs to be rejected.
alpd 1.000000000000e+06
convergence measure 2.053752313598e+00
function val convergence measure 9.782666022408e+13
iteration           objective                   residual                   step_size                   convg
14              5.556721692704e+06            9.782666022408e+13           6.274724053782e+00         2.053752313598e+00
solving full problem starts, iteration 14
real rval -7.980494309649e+00
previous base  4.641765015625e+06, current base 3.437693750000e+04, previous val  9.149566770786e+05, real val -7.980494309649e+00, predicted val -8.395505576262e+13, rho_k 6.577748442055e-08
increasing alpha ratio or increasing minimum for quadratic coefficient
This step needs to be rejected.
alpd 1.000000000000e-05
convergence measure 1.948976578885e+00
function val convergence measure 8.395508097105e+13
iteration           objective                   residual                   step_size                   convg
15              3.436895700569e+04            8.395508097105e+13           6.274724053782e+00         1.948976578885e+00
solving full problem starts, iteration 15
real rval 9.149566770786e+05
previous base  3.437693750000e+04, current base 4.641765015625e+06, previous val  -7.980494309649e+00, real val 9.149566770786e+05, predicted val -9.782668451756e+13, rho_k -5.645037469093e-08
increasing alpha ratio or increasing minimum for quadratic coefficient
This step needs to be rejected.
alpd 1.000000000000e+06
convergence measure 2.053752313598e+00
function val convergence measure 9.782666022408e+13
iteration           objective                   residual                   step_size                   convg
16              5.556721692704e+06            9.782666022408e+13           6.274724053782e+00         2.053752313598e+00
solving full problem starts, iteration 16
real rval -7.980494309649e+00
previous base  4.641765015625e+06, current base 3.437693750000e+04, previous val  9.149566770786e+05, real val -7.980494309649e+00, predicted val -8.395505576262e+13, rho_k 6.577748442055e-08
increasing alpha ratio or increasing minimum for quadratic coefficient
This step needs to be rejected.
alpd 1.000000000000e-05
convergence measure 1.948976578885e+00
function val convergence measure 8.395508097105e+13
iteration           objective                   residual                   step_size                   convg
17              3.436895700569e+04            8.395508097105e+13           6.274724053782e+00         1.948976578885e+00
solving full problem starts, iteration 17
real rval 9.149566770786e+05
previous base  3.437693750000e+04, current base 4.641765015625e+06, previous val  -7.980494309649e+00, real val 9.149566770786e+05, predicted val -9.782668451756e+13, rho_k -5.645037469093e-08
increasing alpha ratio or increasing minimum for quadratic coefficient
This step needs to be rejected.
alpd 1.000000000000e+06
convergence measure 2.053752313598e+00
function val convergence measure 9.782666022408e+13
iteration           objective                   residual                   step_size                   convg
18              5.556721692704e+06            9.782666022408e+13           6.274724053782e+00         2.053752313598e+00
solving full problem starts, iteration 18
real rval -7.980494309649e+00
previous base  4.641765015625e+06, current base 3.437693750000e+04, previous val  9.149566770786e+05, real val -7.980494309649e+00, predicted val -8.395505576262e+13, rho_k 6.577748442055e-08
increasing alpha ratio or increasing minimum for quadratic coefficient
This step needs to be rejected.
alpd 1.000000000000e-05
convergence measure 1.948976578885e+00
function val convergence measure 8.395508097105e+13
iteration           objective                   residual                   step_size                   convg
19              3.436895700569e+04            8.395508097105e+13           6.274724053782e+00         1.948976578885e+00
solving full problem starts, iteration 19
real rval 9.149566770786e+05
previous base  3.437693750000e+04, current base 4.641765015625e+06, previous val  -7.980494309649e+00, real val 9.149566770786e+05, predicted val -9.782668451756e+13, rho_k -5.645037469093e-08
increasing alpha ratio or increasing minimum for quadratic coefficient
This step needs to be rejected.
alpd 1.000000000000e+06
convergence measure 2.053752313598e+00
function val convergence measure 9.782666022408e+13
iteration           objective                   residual                   step_size                   convg
20              5.556721692704e+06            9.782666022408e+13           6.274724053782e+00         2.053752313598e+00
solving full problem starts, iteration 20
real rval -7.980494309649e+00
previous base  4.641765015625e+06, current base 3.437693750000e+04, previous val  9.149566770786e+05, real val -7.980494309649e+00, predicted val -8.395505576262e+13, rho_k 6.577748442055e-08
increasing alpha ratio or increasing minimum for quadratic coefficient
This step needs to be rejected.

It will certainly be worth trying different Ipopt and HiOp PriDec options (and upgrading to a more recent version of HiOp).

[pelesh]

@nychiang @cnpetra: This seems to be similar to an earlier issue we had with HiOp PriDec convergence. Could you please advise how to debug this?