Conversation
Change-Id: I6297bf30f923444bf914d9509cbd2de959703f99
…ion logic 1. Added step analysis to the NeMo summarization script. 2. Improved timestamp detection accuracy: - Introduced a dynamic threshold determined by `train_step_time` to validate step intervals. - Implemented lookahead logic: if a timestamp appears too close to the previous step (gap < threshold), the system now looks ahead for the next timestamp to ensure it correctly represents the step's completion time. Nemo Log Summary: https://paste.googleplex.com/6312345980764160 MLF Log Summary: https://paste.googleplex.com/6706872717541376 Change-Id: I742e56b1d41f5a42c9c50110b825ee3fb3a4c505
Change-Id: I3065f6b3f2ab2d5bda7ef80bc7e75e6c700abbb5
Summary of ChangesHello @kkkapu, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request significantly enhances the log analysis capabilities for NeMo training runs. It introduces a detailed step-to-step time gap analysis, which helps in identifying and quantifying overhead between training steps. The core improvement lies in a more robust timestamp selection mechanism that intelligently handles asynchronous log entries, ensuring more accurate calculations of step completion times and overall training efficiency. Highlights
🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces a valuable step-gap analysis for NeMo logs and significantly refactors the timestamp selection logic to be more robust against asynchronous log flushing. The new event-stream approach with lookahead correction is a clever solution to handle incorrect gap calculations. My review identifies a potential edge case in the new dynamic threshold calculation that could undermine the correction logic, and I have provided a suggestion to make it more robust.
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Change-Id: Ibe681628d3ab27a2200cbc26f4d18d0fbbe44657
Change-Id: I04fb9ad0a7145958d4071630e03a1ccf1b17332b
| try: | ||
| last_seen_timestamp = datetime.strptime(ts_match.group(1), "%Y-%m-%d %H:%M:%S") | ||
| dt = datetime.strptime(ts_match.group(1), "%Y-%m-%d %H:%M:%S") | ||
| events.append({"type": "ts", "val": dt}) |
| THRESHOLD_BUFFER_SECONDS = 2.0 | ||
| DEFAULT_THRESHOLD_SECONDS = 20.0 | ||
| train_times = [e["train_time"] for e in events if e["type"] == "step"] | ||
| if train_times: |
There was a problem hiding this comment.
how do we know this is correct - the threshold subtraction approach? can you explain why we do this and how we know it's correct
2 participants
analyze_step_time_breakdownto use an event-stream approach with lookahead correction. If a timestamp is too close to the previous step (gap < dynamic threshold), the logic now selects the next available timestamp to correctly mark the step completion.