Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ensure capture_output does not output to captured file descriptors #3537

Merged
merged 30 commits into from
Mar 27, 2025

Conversation

jsiirola
Copy link
Member

Fixes #3520

Summary/Motivation:

This further improves capture_output when capture_fd=True. The core issue was that we were creating output loops when capturing the file descriptor if capture_output was capturing a file descriptor shared (directly or indirectly through a logger) with the stream that it was sending output to. While this solution is not bulletproof (I am not sure that we can ever be bullet proof against things like user-defined classes), this is significantly more robust than the previous implementations.

This primarily resolves issues with solvers where tee is implemented with capture_fd=True (cyipopt, highs).

Thanks to @jlinderoth for finding the issue and providing the MRE.

Changes proposed in this PR:

  • Redirect output streams (IOBase and LogStreams) passed to capture_output so that capture_output doesn't send output to a stream / file descriptor that it was capturing
  • support passing either a logger or the module name to LoggingIntercept
  • Provide utilities for redirecting logging handlers that output to captured file descriptors
  • Improve context management within capture_output to better guarantee all contexts are unwound correctly
  • Improve testing

Legal Acknowledgement

By contributing to this software project, I have read the contribution guide and agree to the following terms and conditions for my contribution:

  1. I agree my contributions are submitted under the BSD license.
  2. I represent I am authorized to make the contributions and grant the license. If my employer has rights to intellectual property that includes these contributions, I represent that I have received permission to make contributions and grant the required license on behalf of that employer.

jsiirola added 28 commits March 13, 2025 07:27
@jsiirola jsiirola requested a review from mrmundt March 27, 2025 16:27
Copy link
Member

@blnicho blnicho left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One minor comment/question.

Comment on lines 334 to 335
# if not hasattr(stream, 'write'):
# raise
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this debugging code that should be removed or history that could use context? If it can be removed then you could combine this except with the one below for OSError.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it can be removed. At one point I was using that to better ensure that the stream was an actual stream (I think I was accidentally getting stream to be a tuple of streams, but that is handled elsewhere now.

@jsiirola jsiirola merged commit fc9559f into Pyomo:main Mar 27, 2025
35 checks passed
@jsiirola jsiirola deleted the tee-capture-fd branch March 27, 2025 19:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Python logging and appsi_highs, deadlock or infinite looping
3 participants