Improve timeout error reporting with signal-based error propagation#94
Merged
lambdalisue merged 3 commits intomainfrom Jan 8, 2026
Merged
Improve timeout error reporting with signal-based error propagation#94lambdalisue merged 3 commits intomainfrom
lambdalisue merged 3 commits intomainfrom
Conversation
Previously, the CLI subprocess combined timeout and manual abort signals using AbortSignal.any(). This approach conflated two concerns: - Timeout enforcement (scenario-level timeout) - Manual cancellation (Ctrl-C, --fail-fast) Now, the subprocess passes timeout as a separate option to Runner.run(), allowing Runner to distinguish between scenario timeouts and external cancellations. This enables accurate timeout error reporting with proper ScenarioTimeoutError vs StepTimeoutError distinction. The Runner internally creates timeout signals and properly propagates timeout reasons through the signal chain, resulting in clear error messages that indicate whether a timeout was caused by: - Step's own timeout setting (StepTimeoutError) - Scenario-wide timeout (ScenarioTimeoutError with step context)
Updates: - @probitas/runner: ^0.5.1 → ^0.5.6 - @probitas/builder: ^0.5.1 → ^0.5.2 - @probitas/core: ^0.3.1 → ^0.3.2 The runner update includes improved timeout error handling with signal-based error propagation, enabling proper distinction between StepTimeoutError and ScenarioTimeoutError.
Adds comprehensive test scenarios to verify timeout error handling: - Step timeout: Individual step exceeds its configured timeout - Step timeout with retry: All retry attempts timeout - Scenario timeout: Scenario-level timeout triggers during step execution These scenarios validate that: - StepTimeoutError is reported when step's own timeout is exceeded - ScenarioTimeoutError is reported with step context when scenario timeout triggers - Error messages clearly distinguish the timeout source
There was a problem hiding this comment.
Pull request overview
This PR refactors timeout error handling to provide clearer, more actionable error messages by separating timeout enforcement from manual cancellation. The CLI subprocess now delegates timeout handling to the Runner package, which uses signal-based error propagation to distinguish between step-level and scenario-level timeouts.
Key changes:
- Refactored CLI subprocess to pass timeout as a dedicated Runner option instead of combining it with abort signals
- Updated @probitas/runner to v0.5.6, @probitas/builder to v0.5.2, and @probitas/core to v0.3.2 to support signal-based timeout error propagation
- Added comprehensive timeout failure test scenarios covering step timeouts, retry behavior, and scenario-level timeouts
Reviewed changes
Copilot reviewed 3 out of 4 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| src/cli/_templates/run.ts | Removes AbortSignal.any() combining timeout and manual abort; passes timeout as a dedicated option to Runner |
| probitas/100-failure-timeout.probitas.ts | New test file demonstrating step timeout, step timeout with retry, and scenario timeout scenarios |
| deno.json | Updates dependency versions for @probitas/runner (0.5.6), @probitas/builder (0.5.2), and @probitas/core (0.3.2) |
| deno.lock | Synchronizes lockfile with updated dependency versions and transitive dependencies |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Why
Previously, timeout errors were indistinguishable - whether a step failed due to its own timeout or the scenario's overall timeout, both appeared as generic timeout errors. This made debugging difficult.
The CLI subprocess previously combined timeout and manual abort signals using
AbortSignal.any(), which conflated two concerns:This refactor separates these concerns by:
Runner.run()AbortSignal.reasonThe result is clear, actionable error messages:
StepTimeoutError: Step exceeded its own timeout settingScenarioTimeoutError: Scenario timeout triggered, with context about which step was executingTest Plan