[PEP] Redesign of Pyomo Solvers

[michaelbynum]

I would like to start redesigning the solver interfaces in Pyomo.

Motivation

1. The current solver interfaces are more complex than necessary.
2. The current solver interfaces have many inconsistencies.
3. The current solver interfaces do not allow the solution of a model/block with constraints that use variables which live elsewhere.

Preliminary Design Decisions (to be updated with time)

• When solving a model/block, the model is defined by the set of active components and any components used by those active components. This means that variables are handed to the solver if and only if they are used in an active constraint or objective.

Implementation

• I intend to develop new solver interfaces alongside the existing ones. I do not intend to remove existing solver interfaces. Once the new solver interfaces are sufficiently mature, we can rename/archive the old interfaces.
• I intend to start with a couple simple solver interfaces and get feedback before proceeding further.

Thoughts? Concerns?

Related Issues

• with cplex, solver_io='python' is much slower than executable=(solver_path)  #1169
• Unused variables are added to Direct solvers #1490
• Fall back to direct interface if shell is not available #1776 🇺🇸
• BUG: stopping the default log of appsi_highs #2845
• ["scip.exe", "--version"] timed out after 1 second #3064
• SCIPAMPL solver does not assign values to all variables #3110
• Sometimes constraints are not satisfied with NL Writer V2 with export_defined_variables=True #3243
• Defer starting the SAS session until needed #3448
• Use initial solution for MIP (and LP?) with APPSI HiGHS interface #3450

[ghackebeil]

I think you should reduce the scope of this PEP. The third bullet point is an orthogonal and controversial issue. There is no need to have it possibly get in the way the other (much needed) changes.

[jwatsonnm]

Regarding the third bullet Gabe references, can you clarify? Is said variable at least below the current block in the model component hierarchy? If it's not in the hierarchy, then I agree with Gabe that this is highly controversial.

[carldlaird]

I updated the motivation in the pep to have numbers.

It would be good to document the cases where "Bullet 3" is required. I also admit that the first time I heard that we needed to support solves where variables live outside the hierarchy, I thought this was very strange because it breaks encapsulation of sub-models. Having said that, I would like to see a use case where this is required to be convinced one way or another.

[michaelbynum]

@ghackebeil @jwatsonnm I will open a separate PEP for (3).

[michaelbynum]

@ghackebeil @jwatsonnm @carldlaird Let's move the discussion of (3) to #1032.

[DLWoodruff]

Now that the controversy has moved to #1032, I predict that everyone will think the new PEP with only items one and two is non-controversial great idea and that you should proceed with your plan to create a couple of examples.

[jsiirola]

I am adding a performance label to this PEP. I believe that a redesign of the base solver API is a prerequisite before we can (effectively) tackle things like reworking the writers / readers / persistent APIs.

[jsiirola]

Archived on the master Performance Proposals Issue (#1430). Not closing as this is also marked as a PEP.