[Elastic] Add the log collection and diagnosis function

[wanglei19991004]

• Record training status changes in file: outputs/logs/monitor/status.log
• incrementally collect log content in file: outputs/logs/monitor/host_*_temp_*.log
• automatically analyze training errors and generate report in file: outputs/logs/monitor/host_*_diagnostic_*.txt
  diagnostic report examples:

Diagnostic Report for localhost (node 0)
Generated at 2025-09-11 01:53:17
Analysis:
- OutOfMemoryError: The training process ran out of GPU memory. Check host_0_localhost_temp_20250918_163815.log line:32-128
- RendezvousConnectionError: Connection to rendezvous backend failed. Check host_0_localhost_temp_20250918_163815.log line:33-97
- CodeError: Python exception occurred during execution. Check host_0_localhost_temp_20250918_163815.log line:29-125

• Add hang detection. If there is no update in the log within 30 minutes, it is determined to be hung
• The log monitoring and diagnosis module is disabled by default. Set the switch control:

python run.py action=run +experiment.runner.enable_monitoring=true

or modify train.yaml

experiment:
    runner:
      enable_monitoring: true

[gemini-code-assist]

Summary of Changes

Hello @wanglei19991004, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly enhances the observability and debugging capabilities for elastic training jobs. It establishes a new framework for systematically gathering training logs from various nodes and automatically generating diagnostic reports, providing immediate insights into potential issues. This system is complemented by a fault injection utility, enabling developers to proactively test the robustness of the logging and diagnostic pipeline.

Highlights

• Enhanced Log Collection: Introduced a robust system for incrementally collecting logs from both local and remote (multi-machine) training environments, ensuring efficient and continuous log gathering.
• Automated Log Diagnosis: Implemented an automated log diagnosis module that analyzes collected logs against predefined error patterns to identify and categorize common training failures, such as OOM errors, worker errors, and timeouts.
• Fault Simulation Utility: Added a new utility to simulate various error conditions by writing specific error keys into log files, enabling comprehensive testing and validation of the log collection and diagnosis system.
• Integrated Monitoring: Integrated continuous, asynchronous log monitoring and reporting directly into the training runner, allowing for real-time insights into job status and potential issues, with automatic cleanup upon job completion.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in issue comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces log collection and diagnostic capabilities for elastic training. It adds new modules for log collection, diagnostics, and fault simulation. The core training runner is updated to use these new features in a background monitoring thread.

The overall approach is good, but there are some critical issues that need to be addressed. The incremental log collection for remote nodes is not implemented correctly and currently downloads the full log file on each check. Additionally, shell commands are constructed without escaping file paths, which poses a security and reliability risk. I've also pointed out some smaller issues like unused variables, non-idiomatic naming, and overly broad exception handling.

I've provided specific suggestions to address these points. After these fixes, the new functionality will be much more robust and efficient.

[wanglei19991004]

@gemini-code-assist please review this pull request in details.

[gemini-code-assist]

Code Review

This pull request introduces a new log collection and diagnostic system, which is a great addition for monitoring and debugging training jobs. The implementation includes a monitoring service, log collector, and diagnostic report generator. However, I've found several critical issues that need to be addressed before merging. These include git conflict markers, a method redefinition bug in runner_train.py, and broken remote log collection logic. There are also some high-severity design issues, such as the use of global state and improper signal handling, which could lead to instability. I've provided detailed comments and suggestions for each issue.

[wanglei19991004]

@gemini-code-assist please review this pull request in details.

[gemini-code-assist]

Code Review

This pull request introduces a comprehensive log collection and diagnosis system, which is a valuable addition for monitoring training jobs. The implementation includes services for incremental log collection, error diagnosis, hang detection, and manual kill detection. The code is well-structured into separate modules for different functionalities.

My review focuses on several critical issues related to remote execution and file system assumptions, which could cause the monitoring to fail in a distributed, non-shared filesystem environment. I've also identified potential race conditions, inefficiencies, and bugs like method overloading. Additionally, some of the new unit tests contain incorrect assertions that need to be addressed.

I've provided detailed comments and suggestions to resolve these issues. Once these are fixed, this will be a very robust and useful feature.