Skip to content

Telemetry thread exceptions are silently swallowed #168

@joelsmithTT

Description

@joelsmithTT

Issue

When telemetry extraction fails (e.g. during chip reset), the telemetry update thread dies silently without showing any error message or traceback. This makes debugging very difficult.

Current behavior: Heartbeat stops, telemetry freezes, no error shown
Expected behavior: Exception and traceback displayed when telemetry fails

Reproduction

  1. Start tt-smi and navigate to the Telemetry tab
  2. In another terminal, reset a chip: tt-smi -r 0
  3. Observe: heartbeat stops updating, but no error message appears

Root Cause

The telemetry thread uses threading.Thread(daemon=True) which silently terminates on exceptions.

Current code (tt_smi/tt_smi.py, line ~574):

thread = threading.Thread(target=update_telem, name="telem_thread")
thread.setDaemon(True)  # Daemon threads die silently on exception
thread.start()

Suggested Fix

Use Textual's Worker API, which properly displays exceptions:

# Simple fix: add thread=True parameter
worker = self.run_worker(
    update_telem, 
    thread=True,
    exit_on_error=True,  # Show traceback and exit (helpful for debugging)
    name="telem_worker"
)
TELEM_THREADS.append(worker)

For production use, set exit_on_error=False to continue on transient errors (though you'll want error handling inside update_telem() in that case).

Reference

From Textual docs:

A Worker that encounters an exception will, by default, display a traceback and exit the app. This behavior is designed to help you quickly identify issues during development.

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions