Appsi HiGHS interface : TeeStream and capture_output causes HiGHS execution to freeze

[lpierezan]

Summary

Hello, I run hundreds of optimizations a day using Pyomo with the HiGHS Appsi interface. Certain optimizations, non-deterministically, cause the thread to freeze without any text output to stdout or stderr.

As far as I could trace, the thread hangs when the HiGHS model (from highspy) executes: https://github.com/Pyomo/pyomo/blob/3b0edaff61435028c394c91f3044a991bbff825f/pyomo/contrib/appsi/solvers/highs.py#L231

When I set the highs specific option 'log_to_console' to false this behavior no longer happens.

I'm not an expert but I believe that the context threads created by TeeStream and capture_output to handle output streams are causing the problem.

Steps to reproduce the issue

Unfortunately, I don't know exactly how to share my scenario because creating the model is quite complex and the problem occurs non-deterministically. However, I managed to generate a simple script that mimics part of the code implemented within the appsi_highs interface. The behavior observed in the script below is similar to what I observe in the real-life scenario. If 'log_to_console' is set to True, the thread freezes without error messages. Otherwise, it runs without any problems.

import highspy
import logging
from pyomo.common.log import LogStream
from pyomo.common.tee import TeeStream, capture_output

logger = logging.getLogger(__name__)
log_level = logging.DEBUG

h = highspy.Highs()
filename = 'model_trava.mps'
h.readModel(filename)

ostreams = [
    LogStream(
        level = log_level, logger = logger
    )
]
           
with TeeStream(*ostreams) as t:
    with capture_output(output=t.STDOUT, capture_fd=True):
        h.setOptionValue('log_to_console', True)
        # h.setOptionValue('log_to_console', False)
        h.run()
        print('Model ', filename, ' has status ', h.getModelStatus())

print('the end')

Information on your system

Pyomo version: 6.7.0 Python version: 3.9.16 Operating system: Win 11 , also debian based container image How Pyomo was installed (PyPI, conda, source): pip Solver (if applicable): highspy 1.5.3

[jsiirola]

This sounds like a bad interaction between the Python GIL and HiGHS, and is almost certainly related to #3031.

TeeStream / capture_output work by spawning threads to monitor / process / route the stream data. Unfortunately, the implementation of highspy.run() never releases the Python GIL, so the stream processing threads never get a chance to run. Evenutally the intermediate stream buffer eventually fills up and blocks a write() in HiGHS, causing deadlock.

@michaelbynum, I think we need to refactor the solver interface / output processing in appsi_highs to not rely on TeeStream in output/log processing. This probably also applies to other direct interfaces as well?

[michaelbynum]

Could we just make sure highspy releases the GIL? That should not be hard. I can create a PR to HiGHS.