pyomo Overhaul PyROS Preprocessor Subroutine and Subproblem Objects

Fixes #2964.

Summary/Motivation:

Driven by recent algorithmic developments of the PyROS preprocessor, this PR implements sweeping changes to the PyROS preprocessor subroutine and subproblem formulations/object structures.

Changes proposed in this PR:

Update PyROS as follows:

Overhaul the preprocessor
- Leverage decision rules and equality constraints to identify second-stage variables and state variables that are mathematically nonadjustable
- More structured methods for reformulating variable domains/bounds to constraints
- More structured methods for rearranging inequality constraints
- Rewrite and extend coefficient matching: carefully account for state variable-independent constraints that cannot be reformulated by casting to inequality constraint pairs
Update the subproblem formulations and modeling objects
- Modify block structures of the working model, master-like models, and separation model: partition the constraints by stage
- Change the objective of the decision rule polishing problem to the 1-norm of the nonstatic terms of the decision rule expressions
- Perform more careful initialization of the auxiliary uncertain parameter variables of the separation problem
Update the UncertaintySet class and pre-implemented subclasses to facilitate changes to the subproblems
- Modify return protocol of UncertaintySet.set_as_constraint to make auxiliary uncertain parameters easier to track
- Add new efficient methods for calculation of auxiliary uncertain parameter values for the CardinalitySet and FactorModelSet classes
- Restrict range of valid FactorModelSet instances to those for which psi_mat is full column rank
- Bug fix(es) for the IntersectionSet
Logging updates
- Make log of the model component statistics slightly more informative
- Add DEBUG-level messages for logging of the individual preprocessor steps
Documentation updates
- Update "Methodology Overview" section
- Add section on PyROS installation instructions
Make tests more rigorous and extensive
- Address #2964
- Use expression comparison methods (core.expr.compare.assertExpressionsEqual) for more precise model testing
- Separate testing modules for the:
  - preprocessor
  - subproblem formulations
  - uncertainty set classes
    - more rigorous testing for point_in_set, set_as_constraint
  - PyROS solver as a whole

TODO

[x] (After PR #3324 merged) use ParameterizedQuadraticRepnVisitor in lieu of generate_standard_repn
[x] Update PyROS version number and changelog

Legal Acknowledgement

By contributing to this software project, I have read the contribution guide and agree to the following terms and conditions for my contribution:

I agree my contributions are submitted under the BSD license.
I represent I am authorized to make the contributions and grant the license. If my employer has rights to intellectual property that includes these contributions, I represent that I have received permission to make contributions and grant the required license on behalf of that employer.

Aug 12 '24 14:08 shermanjasonaf

@shermanjasonaf - One failed test on Jenkins across all Python versions:

pyomo.contrib.pyros.tests.test_grcs.RegressionTest.test_coefficient_matching_nonlinear_expr (from pytest) 
self = <pyomo.contrib.pyros.tests.test_grcs.RegressionTest testMethod=test_coefficient_matching_nonlinear_expr>

    @unittest.skipUnless(baron_license_is_valid, "BARON solver license is invalid.")
    def test_coefficient_matching_nonlinear_expr(self):
        """
        Test behavior of PyROS solver for model with
        equality constraint that cannot be reformulated via
        coefficient matching due to nonlinearity.
        """
        # Write the deterministic Pyomo model
        m = ConcreteModel()
        m.x1 = Var(initialize=0, bounds=(0, None))
        m.x2 = Var(initialize=0, bounds=(0, None))
        m.u = Param(initialize=1.125, mutable=True)
    
        m.con = Constraint(expr=m.u ** (0.5) * m.x1 - m.u * m.x2 <= 2)
        m.eq_con = Constraint(
            expr=m.u**2 * (m.x2 - 1)
            + m.u * (m.x1**3 + 0.5)
            - 5 * m.u * m.x1 * m.x2
            + m.u * (m.x1 + 2)
            == 0
        )
        m.obj = Objective(expr=(m.x1 - 4) ** 2 + (m.x2 - 1) ** 2)
    
        interval = BoxSet(bounds=[(0.25, 2)])
    
        # Instantiate the PyROS solver
        pyros_solver = SolverFactory("pyros")
    
        # Define subsolvers utilized in the algorithm
        local_subsolver = SolverFactory('baron')
        global_subsolver = SolverFactory("baron")
    
        # Call the PyROS solver
        with LoggingIntercept(module="pyomo.contrib.pyros", level=logging.DEBUG) as LOG:
            results = pyros_solver.solve(
                model=m,
                first_stage_variables=[m.x1],
                second_stage_variables=[m.x2],
                uncertain_params=[m.u],
                uncertainty_set=interval,
                local_solver=local_subsolver,
                global_solver=global_subsolver,
                options={
                    "objective_focus": ObjectiveType.worst_case,
                    "solve_master_globally": True,
                    "decision_rule_order": 1,
                },
            )
    
        pyros_log = LOG.getvalue()
        self.assertRegex(
            pyros_log, r".*Equality constraint '.*eq_con.*'.*cannot be written.*"
        )
    
        # should still solve in spite of coefficient matching
        # failure
>       self.assertEqual(
            results.pyros_termination_condition,
            pyrosTerminationCondition.robust_optimal,
        )
E       AssertionError: <pyrosTerminationCondition.subsolver_error: 4> != <pyrosTerminationCondition.robust_optimal: 1>

pyomo/pyomo/contrib/pyros/tests/test_grcs.py:1806: AssertionError

Aug 20 '24 14:08 mrmundt

@jsiirola On my machine, the test test_grcs::RegressionTest::test_discrete_separation_invalid_value_error fails; this appears to be a consequence of #3386.

Oct 21 '24 04:10 shermanjasonaf

That's it for updates post #3382.

Oct 31 '24 18:10 shermanjasonaf

Codecov Report

Attention: Patch coverage is 97.27685% with 18 lines in your changes missing coverage. Please review.

Project coverage is 88.66%. Comparing base (ad90ee2) to head (58c6526). Report is 615 commits behind head on main.

Files with missing lines	Patch %	Lines
pyomo/contrib/pyros/solve_data.py	82.75%	5 Missing :warning:
pyomo/contrib/pyros/uncertainty_sets.py	97.50%	5 Missing :warning:
pyomo/contrib/pyros/master_problem_methods.py	98.26%	3 Missing :warning:
pyomo/contrib/pyros/pyros_algorithm_methods.py	95.77%	3 Missing :warning:
pyomo/contrib/pyros/separation_problem_methods.py	98.78%	2 Missing :warning:

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3341      +/-   ##
==========================================
+ Coverage   88.57%   88.66%   +0.08%     
==========================================
  Files         883      883              
  Lines      100343   100156     -187     
==========================================
- Hits        88882    88800      -82     
+ Misses      11461    11356     -105

Flag	Coverage Δ
linux	`86.25% <93.93%> (+0.20%)`	:arrow_up:
osx	`76.22% <91.81%> (+0.17%)`	:arrow_up:
other	`86.75% <93.93%> (+0.20%)`	:arrow_up:
win	`84.60% <93.78%> (+0.21%)`	:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Nov 06 '24 04:11 codecov[bot]