Skip to content

Commit c664184

Browse files
ayerofieiev-ttArtem Yerofieiev
andauthored
[tt-train] Make training profiler respect TT_METAL_DEVICE_PROFILER (#37142)
### Ticket None ### Problem description <img width="754" height="530" alt="image" src="https://github.com/user-attachments/assets/e62f493f-e3ac-4dc5-bb82-3cbcdeb22085" /> See cache entires increasing even in release build. This is due to a change in debug build config when tracy is now included by default. What matters is a runtime env flag TT_METAL_DEVICE_PROFILER. If it is not set, it does not make sense to add `profiler_no_op` calls. ### What's changed Initially wanted to use `tt::tt_metal::MetalContext::instance().rtoptions().get_profiler_enabled()` but it is not accessible - `MetalContext` is a part of metal's private api. Raised this with Audrey. Keeping it raw for now. ### Checklist - [ ] [![All post-commit tests](https://github.com/tenstorrent/tt-metal/actions/workflows/all-post-commit-workflows.yaml/badge.svg?branch=ay/train_profiling_tweak)](https://github.com/tenstorrent/tt-metal/actions/workflows/all-post-commit-workflows.yaml?query=branch:ay/train_profiling_tweak) - [ ] [![Blackhole Post commit](https://github.com/tenstorrent/tt-metal/actions/workflows/blackhole-post-commit.yaml/badge.svg?branch=ay/train_profiling_tweak)](https://github.com/tenstorrent/tt-metal/actions/workflows/blackhole-post-commit.yaml?query=branch:ay/train_profiling_tweak) - [ ] [![cpp-unit-tests](https://github.com/tenstorrent/tt-metal/actions/workflows/tt-metal-l2-nightly.yaml/badge.svg?branch=ay/train_profiling_tweak)](https://github.com/tenstorrent/tt-metal/actions/workflows/tt-metal-l2-nightly.yaml?query=branch:ay/train_profiling_tweak) - [ ] New/Existing tests provide coverage for changes Co-authored-by: Artem Yerofieiev <ayerofieiev@tenstorrent.com>
1 parent 356605d commit c664184

File tree

1 file changed

+6
-2
lines changed

1 file changed

+6
-2
lines changed

tt-train/sources/ttml/core/tt_profiler.cpp

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,8 @@
44

55
#include "tt_profiler.hpp"
66

7+
#include <cstdlib>
8+
79
#include "core/tt_tensor_utils.hpp"
810
#include "metal/operations.hpp"
911

@@ -55,15 +57,17 @@ void TTProfiler::disable() {
5557
}
5658

5759
TTProfiler::TTProfiler() : m_enabled(false) {
58-
if (is_tracy_enabled) {
60+
const char* env_value = std::getenv("TT_METAL_DEVICE_PROFILER");
61+
bool runtime_profiler_enabled = env_value != nullptr && std::string(env_value) == "1";
62+
if (is_tracy_enabled && runtime_profiler_enabled) {
5963
enable();
6064

6165
tt::tt_metal::detail::ProfilerSync(tt::tt_metal::ProfilerSyncState::INIT);
6266
}
6367
}
6468

6569
TTProfiler::~TTProfiler() {
66-
if (is_tracy_enabled) {
70+
if (is_enabled()) {
6771
tt::tt_metal::detail::ProfilerSync(tt::tt_metal::ProfilerSyncState::CLOSE_DEVICE);
6872
}
6973
}

0 commit comments

Comments
 (0)