Skip to content

feat: add logger to the testing infrastructure#1261

Open
fvranicTT wants to merge 1 commit intomainfrom
filip/add-loguru
Open

feat: add logger to the testing infrastructure#1261
fvranicTT wants to merge 1 commit intomainfrom
filip/add-loguru

Conversation

@fvranicTT
Copy link
Contributor

@fvranicTT fvranicTT commented Feb 9, 2026

Ticket

None

Problem description

Added logger to test infrastructure. Replaces print() statements with leveled logging using loguru. tt-metal uses the same logger, so it should provide the same level of user experience we're used to.

from helpers.logger import logger

logger.debug("Detailed info")
logger.info("Progress updates")
logger.warning("Unexpected but non-fatal")
logger.error("Test failures with details")

What's changed

  • New logger module (tests/python_tests/helpers/logger.py) with configurable log levels
  • CLI control: pytest --loguru-level=INFO to see logs in terminal (default: file-only)
  • Easy to adjust verbosity: --loguru-level=DEBUG for local debugging
  • Log files:
    • test_run.log - full session log (overwritten each run)
    • test_errors.log - persistent error log across runs
    • CI uploads both as artifacts on failure
  • Replaced prints in helpers (utils.py, test_config.py, fused_golden.py, etc.) with appropriate log levels
  • CI configuration: Default log level WARNING (via LOGURU_LEVEL env var)

Type of change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation update
  • Refactoring

Checklist

@github-actions
Copy link
Contributor

github-actions bot commented Feb 9, 2026

Thank you for your contribution! 🚀
If you want to run metal post-commit tests, you can add the metal-post-commit-tests label to this pull request.
📖 For more information, please refer to our CONTRIBUTING guide.

@github-actions github-actions bot added ci test-infra This label is used for issues, pull requests, or tasks related to the LLK testing framework labels Feb 9, 2026
@fvranicTT fvranicTT marked this pull request as ready for review February 9, 2026 19:03
Copilot AI review requested due to automatic review settings February 9, 2026 19:03
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 5 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines +95 to +102
# Session log - full log for this run (overwritten each session)
logger.add(
"test_run.log",
format=_file_format,
level=level,
mode="w",
colorize=False,
)
Copy link

Copilot AI Feb 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

configure_logger() writes to test_run.log with mode="w", but CI runs pytest with xdist (-n 10). Each worker process will reconfigure loguru and truncate the same file, so logs will be lost/corrupted and the uploaded artifact may be incomplete. Consider making log filenames worker-specific (e.g., include PYTEST_XDIST_WORKER/PID), or only let the controller process create/truncate the session log and have workers append (plus consider enqueue=True if sharing a sink).

Copilot uses AI. Check for mistakes.
Comment on lines +89 to +90
level = os.environ.get("LOGURU_LEVEL", "INFO").upper()

Copy link

Copilot AI Feb 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The CLI-provided level is not normalized, while the env var path is uppercased. If a user passes --loguru-level=info, it may not match loguru's expected level names consistently. Suggest normalizing level = level.upper() (and possibly validating against allowed values) when level is provided.

Suggested change
level = os.environ.get("LOGURU_LEVEL", "INFO").upper()
level = os.environ.get("LOGURU_LEVEL", "INFO")
# Normalize log level to match loguru's expected uppercase names
level = level.upper()

Copilot uses AI. Check for mistakes.
# Enable pytest's live logging when --loguru-level is set.
# Loguru propagates to stdlib logging, and pytest's log_cli displays
# those messages in the terminal - integrating cleanly with pytest-sugar.
if log_level is not None:
Copy link

Copilot AI Feb 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Setting config.option.log_cli_level alone may not enable pytest live logging; without log_cli=True (or equivalent), users may still not see terminal logs even when --loguru-level is provided, contradicting the intended behavior described in this file and PR description. Consider also setting config.option.log_cli = True when log_level is not None.

Suggested change
if log_level is not None:
if log_level is not None:
# Ensure live logging is enabled when a log level is provided.
config.option.log_cli = True

Copilot uses AI. Check for mistakes.

if print_pcc:
print("PCC:", pcc)
logger.info("PCC: {:.6f} | format={}", pcc, output_data_format.name)
Copy link

Copilot AI Feb 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

print_pcc is now ignored: passed_test(..., print_pcc=False) will still emit an INFO log for PCC on every call, which can get very noisy (especially under xdist) and changes the function’s existing behavior. Suggest gating this log behind if print_pcc: (and/or using logger.debug for the default path).

Suggested change
logger.info("PCC: {:.6f} | format={}", pcc, output_data_format.name)
if print_pcc:
logger.info("PCC: {:.6f} | format={}", pcc, output_data_format.name)
else:
logger.debug("PCC: {:.6f} | format={}", pcc, output_data_format.name)

Copilot uses AI. Check for mistakes.
Comment on lines +190 to +197
- name: Upload test logs on failure
uses: actions/upload-artifact@v4
if: failure()
with:
name: test-logs-${{ env.CHIP_ARCH }}-${{ matrix.test_group }}
path: |
tests/python_tests/test_run.log
tests/python_tests/test_errors.log
Copy link

Copilot AI Feb 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CI runs pytest with xdist (-n 10), but the logger currently writes fixed filenames (test_run.log/test_errors.log) and truncates test_run.log each session. Under parallel workers this can lead to missing/overwritten logs, so these artifacts may be unreliable. Once logger output is made xdist-safe (e.g., per-worker filenames), update this upload step to include all worker log files (glob).

Copilot uses AI. Check for mistakes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci test-infra This label is used for issues, pull requests, or tasks related to the LLK testing framework

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant