Add --profile flag for LLM benchmarks

[odjuricicTT]

Problem description

When profiling LLM benchmarks with tools like Tracy, running the full benchmark with all layers and many iterations is slow and generates excessive data. A streamlined profiling mode is needed for faster iteration during performance analysis.

What's changed

Added a --profile pytest flag to enable profiling mode for LLM benchmarks:

• conftest.py: Added --profile command-line option and fixture
• test_llms.py: All test functions now accept and pass the profile fixture. When enabled, models are loaded with num_layers=1
• llm_benchmark.py:
  • Added tracy signposts for token generation and warmup phases
  • When --profile is passed, limits max_tokens_to_generate to 2 (affects both warmup and benchmark)
  • Skips the 10-iteration validation in profile mode

Usage:

pytest -svv benchmark/tt-xla/test_llms.py::test_llama_3_2_1b --profile

Checklist

• New/Existing tests provide coverage for changes

---

Note: This PR depends on the num_layers PR in tt_forge_models and should be merged after that PR lands.

[odjuricicTT]

Performance benchmark run (llama_3_2_1b only): https://github.com/tenstorrent/tt-forge/actions/runs/21752281045

[odjuricicTT]

Tracy is not on XLA main yet, will not merge this

[vkovacevicTT]

Tracy is not on XLA main yet, will not merge this

Let's do this in tt-xla then when we migrate the benchmark: tenstorrent/tt-xla#2971