[PEP] Solving Blocks with variables that do not live on the Block

[michaelbynum]

I would like to be able to solve Pyomo Blocks with variables that do not live on the Block. I will describe my use case, but there are others.

I have a structured MINLP. In fact, it looks like a binary tree of Blocks. I have a transformation to relax this MINLP. The the root block has a constraint with a bilinear term (for example), x*y, then I will replace that bilinear term with an auxiliary variable, aux, and add constraints for the McCormick envelopes relating aux to x and y. If this same bilinear term, x*y appears in other constraints, I want to replace it with the same auxiliary variable aux. I do not want to create a new variable for the same term. Now, suppose I encounter the same bilinear term in a constraint on a child Block. I already have the auxiliary variable on the root Block. It is not easy to move the auxiliary variable so that it has a new parent. I also have a need to solve individual child Blocks.

Another example I am aware of arises in GDP transformations. Maybe @jsiirola can elaborate.

[carldlaird]

I see the use case you describe. Two comments:

1. Not that I think this is better, but would it make sense to use a transformation to flatten the model first? Then this kind of issue would go away? (I suspect this would bring a host of other issues, not the least of which is the required name mangling when flattening.)

2. Is this (and other use cases like this) implying that variables are not really tied to their parent?

[emma58]

I think that besides the naming issues, flattening the model also obscures the structure which was likely why you were using Blocks in the first place. I don't think there is any reason to assume that you are going to stop messing with a model after just one solve. If you have to flatten to do that solve, you've just made it much harder to make further changes. And I think if individual Blocks on the original model were solvable before the transformation, it seems reasonable to expect them to be after it as well. Granted, transformations could be designed so that this was still true, but in the GDP transformations at least, we are using Blocks so that the transformed model contains a completely intact version of the original model (possibly with some components deactivated). Everything the transformation adds is on a new Block it creates. This helps map the original model to what the transformation did and makes the names of components created by transformations predictable.

[qtothec]

I'll add that flattening also obscures the structure that the solvers I am involved with rely upon.

[michaelbynum]

The only reason I can think of to require that variables live on the block bening solved is for model validation (which certainly depends on how you conceptually define a model). However, model validation is not really what is currently happening. Consider the following:

>>> b = pe.Block(concrete=True)
>>> b.x = pe.Var()
>>> b.b = pe.Block(concrete=True)
>>> b.b.obj = pe.Objective(expr=b.x**2)
>>> b.b.c = pe.Constraint(expr=b.x >= 1)
>>> opt = pe.SolverFactory('ipopt')
>>> res = opt.solve(b)
>>> res = opt.solve(b.b)
Traceback (most recent call last):
...
KeyError: 90652546328

Here, b can be solved, but b.b cannot. However, in both cases, constraints are defined which use variables that do not live on the same block.

[michaelbynum]

@carldlaird

1. Flattening the model would solve this problem. However, the block structure was actually induced on purpose and is necessary.

2. I’m not sure. There might be two questions here. First, how is an optimization model defined? Second, what are the software design issues? For design reasons, it may be necessary to “tie” a variable to it’s parent - I’m not sure.

[michaelbynum]

Here is another use case. Suppose I want to model a stochastic programming problem, but I want to use the same Var objects (for the first stage variables) in all of the scenarios rather than using nonanticipativity constraints. I can’t think of a good reason for this right now, but it is certainly a valid way to model stochastic programs. I could put the first stage variables on the root block, and use a block for each of the scenarios. I could then use the first stage variables in each of the scenario sub-blocks, but I would not be able to solve the scenario sub-blocks independently.

[jwatsonnm]

Regarding the above case, an alternative would be to: (1) define a block containing all first stage variables, (2) reference that block in both the scenario blocks and the master model block, (3) define the master optimization objective by referencing cost terms in the individual scenario blocks. Then you at least have encapsulation. But you have other issues with respect to non-tree block reference structures. I bring up this alternative because I really don't think violating encapsulation is a good idea (e.g., are we going to allow variable names of the type "child_block."..".first_stage_var?). If violating encapsulation is a good idea, I assert that we should first ensure that supporting a block DAG in model specification isn't a cleaner or at least equally viable alternative.

[carldlaird]

I like the way you are leaning here JP. Basically, if violating encapsulation is a good idea, then this is an indicator that we do not have the correct structure for grouping constraints (e.g., you said a DAG would be better than a tree).

[michaelbynum]

My primary question (which I don't know the answer to) is whether or not variables should be involved in encapsulation. Why do we add variables to models/blocks? Is it simply for convenient access so that they may be used in other components (e.g., constraints, objectives, disjuncts, etc.), or is it actually for model definition?

If we want to enforce encapsulation, then should we throw an error when a constraint uses a variable that lives elsewhere in the model? We do not currently do this (as illustrated by the example above).

A block DAG is an interesting idea. This would require blocks to have more than one parent though, right?

[ghackebeil]

I like this suggestion as well.

I would argue, however, that there is a simple way to handle Michael's SP example without any changes to the code. Assuming the master variables are on the top-level model, if you want to solve the scenarios individually, simply do the following:

# deactivate all scenarios
for scenario in model.scenarios:
    scenario.deactivate()

# solve each scenario
for scenario in model.scenarios:
    scenario.activate()
    solve(model)
    scenario.deactivate()

[carldlaird]

@ghackebeil Can you be more specific? Which suggestion do you like?

[michaelbynum]

@ghackebeil I agree that your solution works for the two stage problem. It would be more complex in the multi-stage case (which is more similar to my original use case). Additionally, what if I don't want to include the constraints on the root block? I'm thinking of a bender's type algorithm...