Reset via luwen backend leaves stale device context and affects subsequent tests/processes

[wholewhite]

Description

We are observing unstable behavior when performing device reset using the luwen backend in tt-smi (v5.x), especially in CI hardware tests.

After a reset operation, subsequent tests or processes interacting with the device may fail, suggesting that the reset might leave stale device context in user-space.

This might be related to issue #172, where reset conflicts with active user-space applications.Duplicate of #

Steps to Reproduce

A simplified reproduction pattern:

1. Start a process that initializes and uses a device (e.g. via tt-smi or other tooling)
2. Perform a reset using:tt-smi -r
3. Continue using the device in the same process or subsequent tests

---

Observed Behavior

• Subsequent operations may fail with:
• device not found
• initialization errors
• telemetry not updating / hanging
• In CI scenarios, this can cause cascading failures across test cases
• Background telemetry or monitoring threads may stop working after reset

---

Expected Behavior

After reset:

• Device should be fully re-initializable
• Previous device contexts should be invalidated or safely handled
• Subsequent processes/tests should not be affected by prior state

---

Hypothesis

It appears that:

• Reset invalidates the hardware state
• However, existing user-space contexts (possibly via luwen) remain alive
• These stale contexts may continue to interact with the device, leading to undefined behavior

This could result in cross-test contamination in CI environments.

---

Environment

• tt-smi version: 5.0
• Backend: luwen
• Hardware: (e.g. Wormhole)
• OS: (e.g. Ubuntu 22.04)

---

Additional Notes

This issue is especially visible in CI pipelines where multiple tests are executed in the same process or worker.

A potential workaround is to fully restart the process after reset, suggesting that the issue may be related to lifecycle management of device contexts.

---

Question

Is this expected behavior for the luwen backend reset?

Are there recommended best practices for safely handling reset in multi-test or multi-process environments?