MPS (#497)

* created an mps readers based on pulp

* created an mps_module to allow for mps based scenarios

* [WIP] added to the format for json files to accompany lp and mps files

* can now read mps using coin mip; here's a little demo

* major corrections by dlw

* mps_coin_mip.py executes, but the solutions don't match

* the mps reader is ready for testing

* added a test for the mps_reader

* [WIP] getting start on testing for the mps module option in generic cylinders

* [WIP] working on testin generic tester and fixing bugs (still need to work on cfg.num_scens

* working on generic tester to get cfg.num_scens taken care of

* there is now a test for mps reader and mps module in generic tester, but it needs to be easier for xpress

* touch up tests

* added a demonstration and some documentation.

* tests are ready to run?

* cleanup

* trying to debug the admm_wrapper test

* Update test_admmWrapper.py

* make the top level of the json file schema symmetric

* Update doc/src/agnostic.rst

Co-authored-by: bknueven <[email protected]>

* Update mpisppy/utils/mps_reader.py

Co-authored-by: bknueven <[email protected]>

* change the name of the mps write opion in generic_cylinders

---------

Co-authored-by: bknueven <[email protected]>

Author: DLWoodruff

.github/workflows/test_pr_and_main.yml

@@ -310,6 +310,34 @@ jobs:
           python test_pickle_bundle.py
 
 
+  mps:
+    name: MPS tests
+    runs-on: ubuntu-latest
+    needs: [ruff]
+
+    steps:
+      - uses: actions/checkout@v3
+      - uses: conda-incubator/setup-miniconda@v2
+        with:
+          activate-environment: test_env
+          python-version: 3.11
+          auto-activate-base: false
+      - name: Install dependencies
+        run: |
+          conda install mpi4py pandas setuptools
+          pip install pyomo xpress cplex mip
+
+      - name: setup the program
+        run: |
+          pip install -e .
+
+      - name: run MPS tests
+        timeout-minutes: 2
+        run: |
+          cd mpisppy/tests
+          python test_mps.py
+
+
   confidence-intervals:
     name: confidence intervals tests
     runs-on: ubuntu-latest
@@ -494,7 +522,7 @@ jobs:
         timeout-minutes: 10
         run: |
           cd mpisppy/tests
-          mpiexec -np 2 python -m mpi4py test_with_cylinders.py          
+          mpiexec -np 2 python -m mpi4py test_with_cylinders.py
 
   test-agnostic:
     name: tests on agnostic

doc/src/agnostic.rst

@@ -4,7 +4,34 @@ AML Agnosticism
 The mpi-sppy package provides callouts so that algebraic modeling languages
 (AMLs) other than Pyomo can be used. A growing number of AMLs are supported
 as `guest` languages (we refer to mpi-sppy as the `host`). This code is
-in an alpha-release state; use with extreme caution.
+in an alpha-release state; use with extreme caution.  This is referred to
+as `tight` integration with the guest. It is also possible to simply read
+scenario data from an mps file and the mps file (and the associated json
+nonant file) that can be created however you like. 
+
+Loose integration
+^^^^^^^^^^^^^^^^^
+
+Code for creating a
+Pyomo model from an mps file is in ``mpisppy.utils.mps_reader.py``,
+but you can also just use ``generic_cylinders.py`` and give
+it the module ``mpisppy.utils.mps_module`` (you will need to specify
+that path to this module) and the ``--mps-files-directory``
+option.  Note
+that at the time of this writing, the number of scenarios is obtained
+by counting the mps files in the directory given.
+
+The file ``examples.sizes.mps_demo.bash`` has two commands. The second illustrates
+how to instruction ``MPI-SPPY`` to read mps/json file pairs for each scenario from a
+directory. The first command illustrates how to use ``MPI-SPPY`` to write
+them in the first place (but if ``MPI-SPPY`` can get your scenarios, there
+is probably no reason to write them and then read them again!). This
+functionality is intended to be used by users of other AMLs or other
+scenario-based stochastic programming applications.
+
+
+Tight integration
+^^^^^^^^^^^^^^^^^
 
 From the end-user's perspective
 -------------------------------

doc/src/extensions.rst

@@ -274,11 +274,17 @@ If some variables have zero probability in all scenarios, then you will need to
 all variable probabilities! So you might want to set this to False to verify that the probabilities sum to one
 only for the Vars you expect before setting it to True.
 
-Scenario_lpwriter
------------------
-
-This extension writes an lp file with the model and json file with (a) list(s) of
-scenario tree node names and nonanticaptive variables for each scenario before
-the iteration zero solve of PH or APH. Note that for two-stage problems, all
-json files will be the same. See ``mpisppy.generic_cylinders.py``
-for an example of use.
+Scenario_lp_mps_writer
+----------------------
+
+This extension writes an lp file and an mps file with the model as well as a
+json file with (a) list(s) of scenario tree node names and
+nonanticaptive variables for each scenario before the iteration zero
+solve of PH or APH. Note that for two-stage problems, all json files
+will be the same. See ``mpisppy.generic_cylinders.py`` for an example
+of use. In that program it is activated with the
+``--scenario-lp-mps-writer`` option. Note that it
+writes the files to the current working directory and for each scenario
+the base name of the three files written is the scenario name.
+
+Unless you know exactly why you need this, you probably don't.

examples/generic_tester.py

@@ -177,12 +177,19 @@ def do_one(dirname, modname, np, argstring, xhat_baseline_dir=None, tol=1e-6):
 #rebaseline_xhat("farmer", "farmer", 3, farmer_rd, "test_data/farmer_rd_baseline")
 do_one("farmer", "farmer", 3, farmer_rd, xhat_baseline_dir="test_data/farmer_rd_baseline")
 
-# Just a smoke test to make sure sizes_expression still exists and
-# that lpfiles still executes.
+### combined runs to test mps files ####
+# Make sure sizes_expression still exists and lpfiles still executes.
 sizese = ("--module-name sizes_expression --num-scens 3 --default-rho 1"
           f" --solver-name {solver_name} --max-iterations 0"
-          " --scenario-lpfiles")
-do_one("sizes", "sizes", 3, sizese, xhat_baseline_dir=None)   
+          " --write-scenario-lp-mps-files")
+do_one("sizes", "sizes_expression", 3, sizese, xhat_baseline_dir=None)
+# just smoke for now
+sizesMPS = ("--module-name ../../mpisppy/utils/mps_module --default-rho 1"
+          f" --solver-name {solver_name} --max-iterations 0"
+          " --mps-files-directory=.")   # we will be in the sizes dir
+do_one("sizes", "../../mpisppy/utils/mps_module", 1, sizesMPS, xhat_baseline_dir=None)
+
+### end combined mps file runs ###
 
 quit()
 

examples/sizes/mps_demo.bash

@@ -0,0 +1,23 @@
+#!/bin/bash
+# This is mainly to demonstrate what loose agnostic files look like and
+#   how to use them with agnostic_cylinders.py.
+# To do that, we write the files based on a Pyomo model, then
+# read them in.
+# Note: if you actually have a Pyomo model, you probably don't want to do
+#  it this way since you would have had to have written most of the
+# functions (e.g. scenario_creator) anyway.
+# If you are using some other AML, then you migth want to use the second
+#   command line to read the files you wrote with your AML and
+#   you can use the first command to write files as an example of the format
+#   for the json files.
+
+set -e
+
+SOLVER=cplex
+
+# assumes we are in the sizes directory and don't mind polluting it with 6 files
+python ../../mpisppy/generic_cylinders.py --module-name sizes_expression --num-scens 3 --default-rho 1 --solver-name ${SOLVER} --max-iterations 0 --scenario-lp-mps-files
+
+# By specifying the module to be mps_module we will read files for the problem
+#  from the specified mps-files-directory.
+mpiexec -np 3 python -m mpi4py ../../mpisppy/generic_cylinders.py --module-name ../../mpisppy/utils/mps_module --xhatshuffle --lagrangian --default-rho 1 --solver-name ${SOLVER} --max-iterations 10 --mps-files-directory=.

mpisppy/extensions/scenario_lpfiles.py renamed to mpisppy/extensions/scenario_lp_mps_files.py

@@ -17,22 +17,28 @@
 
 
 def lpize(varname):
-    # convert varname to the string that will appear in the lp file
-    # return varname.replace("[", "(").replace("]", ")").replace(",", "_").replace(".","_")
+    # convert varname to the string that will appear in the lp and mps files
     return pyomo_label.cpxlp_label_from_name(varname)
 
 
-class Scenario_lpfiles(mpisppy.extensions.extension.Extension):
+class Scenario_lp_mps_files(mpisppy.extensions.extension.Extension):
 
     def __init__(self, ph):
         self.ph = ph
 
     def pre_iter0(self):
         for k, s in self.ph.local_subproblems.items():
             s.write(f"{k}.lp", io_options={'symbolic_solver_labels': True})
-            nonants_by_node = {nd.name: [lpize(var.name) for var in nd.nonant_vardata_list] for nd in s._mpisppy_node_list}
+            s.write(f"{k}.mps", io_options={'symbolic_solver_labels': True})
+            scenData = {"name": s.name, "scenProb": s._mpisppy_probability} 
+            scenDict = {"scenarioData": scenData}
+            treeData = dict()
+            for nd in s._mpisppy_node_list:
+                treeData[nd.name] = {"condProb": nd.cond_prob}
+                treeData[nd.name].update({"nonAnts": [lpize(var.name) for var in nd.nonant_vardata_list]})
+            scenDict["treeData"] = treeData
             with open(f"{k}_nonants.json", "w") as jfile:
-                json.dump(nonants_by_node, jfile)
+                json.dump(scenDict, jfile, indent=2)
 
     def post_iter0(self):
         return

mpisppy/generic_cylinders.py

@@ -29,7 +29,7 @@
 from mpisppy.extensions.fixer import Fixer
 from mpisppy.extensions.mipgapper import Gapper
 from mpisppy.extensions.gradient_extension import Gradient_extension
-from mpisppy.extensions.scenario_lpfiles import Scenario_lpfiles
+from mpisppy.extensions.scenario_lp_mps_files import Scenario_lp_mps_files
 
 from mpisppy.utils.wxbarwriter import WXBarWriter
 from mpisppy.utils.wxbarreader import WXBarReader
@@ -55,15 +55,17 @@ def _parse_args(m):
                       description="The string used for a directory of ouput along with a csv and an npv file (default None, which means no soltion output)",
                       domain=str,
                       default=None)
-    cfg.add_to_config(name="scenario_lpfiles",
-                      description="Invokes an extension that writes an model lp file and a nonants json file for each scenario before iteration 0",
+    cfg.add_to_config(name="write_scenario_lp_mps_files",
+                      description="Invokes an extension that writes an model lp file, mps file and a nonants json file for each scenario before iteration 0",
                       domain=bool,
                       default=False)
 
     m.inparser_adder(cfg)
     # many models, e.g., farmer, need num_scens_required
     #  in which case, it should go in the inparser_adder function
     # cfg.num_scens_required()
+    # On the other hand, this program really wants cfg.num_scens somehow so
+    # maybe it should just require it.
 
     cfg.EF_base()  # If EF is slected, most other options will be moot
     # There are some arguments here that will not make sense for all models
@@ -120,7 +122,7 @@ def _name_lists(module, cfg, bundle_wrapper=None):
             "For now, stage2EFsolvern is required for multistage xhat"
     else:
         all_nodenames = None
-        num_scens = cfg.num_scens
+        num_scens = cfg.get("num_scens")  # maybe None is OK
 
     # proper bundles should be almost magic
     if cfg.unpickle_bundles_dir or cfg.scenarios_per_bundle is not None:
@@ -230,8 +232,8 @@ def _do_decomp(module, cfg, scenario_creator, scenario_creator_kwargs, scenario_
         ext_classes.append(Gradient_extension)
         hub_dict['opt_kwargs']['options']['gradient_extension_options'] = {'cfg': cfg}        
 
-    if cfg.scenario_lpfiles:
-        ext_classes.append(Scenario_lpfiles)
+    if cfg.write_scenario_lp_mps_files:
+        ext_classes.append(Scenario_lp_mps_files)
 
     if cfg.W_and_xbar_reader:
         ext_classes.append(WXBarReader)

mpisppy/spbase.py

@@ -105,7 +105,8 @@ def __init__(
             global_toc("Initializing SPBase")
 
         if self.n_proc > len(self.all_scenario_names):
-            raise RuntimeError("More ranks than scenarios")
+            raise RuntimeError(f"More ranks ({self.n_proc}) than scenarios"
+                               f" ({len(self.all_scenario_names)})")
 
         self._calculate_scenario_ranks()
         # Put the deprecation message in the init so they should only see it once per rank