Skip to content

Commit aaccd27

Browse files
committed
add Debugging from the UI section: parallel-worker URLs, Bundle Upload ID, RSpec/Go caveats
1 parent 97856c2 commit aaccd27

3 files changed

Lines changed: 62 additions & 2 deletions

File tree

flaky-tests/dashboard.mdx

Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -196,3 +196,55 @@ Click any row in the runs table to open a detail panel on the right side of the
196196
* **Run header**: Timestamp, a result badge (Pass, Fail, Error, or Quarantined), and run duration.
197197
* **Source control**: A CI job link (with the provider's icon, the job name, and the CI duration), the linked pull request, branch, and commit. Merge queue runs also include a **View in Merge Queue** link.
198198
* **Error details**: For failed, errored, or quarantined runs, an optional AI summary of the failure followed by the raw error text or stack trace.
199+
200+
### Debugging a flaky test from the UI
201+
202+
The Summary tab, failure details, and Test History tab give you most of what you need to investigate a flaky test. A few gaps come up often enough to call out, along with the workarounds that exist today.
203+
204+
#### Drilling into the right parallel worker
205+
206+
The CI job link in the run detail panel points to the parent build, not the specific worker that produced the failure. If your CI runs a single job, this is fine. If you fan out across many parallel workers (some customers run 40+), you'll have to click through workers in the CI provider to find the one whose log contains the failure.
207+
208+
To shortcut this, capture the per-worker URL at test run time and include it in your JUnit output so it surfaces in the failure detail. Most CI providers expose an environment variable for the running job's URL:
209+
210+
| CI provider | Environment variable |
211+
|---|---|
212+
| CircleCI | `CIRCLE_BUILD_URL` |
213+
| GitHub Actions | `${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID}` |
214+
| Buildkite | `BUILDKITE_BUILD_URL` |
215+
| GitLab CI | `CI_JOB_URL` |
216+
217+
Read the value at the start of the test job and append it to a test property, log line, or system-out block in your JUnit XML. The link then appears alongside the failure in Trunk instead of routing to the parent build.
218+
219+
#### Bundle Upload ID lookups
220+
221+
When the `trunk-analytics-cli` uploads a bundle, it prints a Bundle Upload ID to the job log. This ID does not currently map to a URL in the web app — there's no search field for it in the dashboard. If you need to trace a specific upload back to its run data, contact support with the Bundle Upload ID.
222+
223+
<Info>
224+
Uploads can take up to 10 minutes to populate run data in the dashboard. If a run you just uploaded isn't visible yet, wait and refresh before assuming the upload failed.
225+
</Info>
226+
227+
#### CI artifact retention
228+
229+
CI providers typically retain build artifacts (screenshots, videos, traces) for one to two days. Flaky test tickets often take longer than that to investigate and resolve, which means the artifacts that would have helped explain the failure may already be gone by the time you open the ticket.
230+
231+
If artifacts matter for your debugging flow, store them outside the CI provider's retention window:
232+
233+
* Upload screenshots, videos, and traces to S3 (or another long-lived store) as a CI step, and include the object URL in the JUnit output alongside the per-worker URL described above.
234+
* For especially noisy tests, attach the artifact URL to the Jira/Linear ticket Trunk opens via the [ticketing integration](./management/ticketing/jira-integration).
235+
236+
#### Known limitations
237+
238+
A few framework-specific quirks are worth knowing about up front.
239+
240+
<Info>
241+
**RSpec `MultipleExceptionError`**
242+
243+
When an example raises multiple exceptions (for example, an error in the test body plus a separate error in an `after` hook), RSpec wraps them in `RSpec::Core::MultipleExceptionError`. The failure detail view currently surfaces only one of the captured exceptions. The `rspec_trunk_flaky_tests` gem is being extended to extract the full set; until that ships, check the CI job logs for the full exception list when you see this error type.
244+
</Info>
245+
246+
<Info>
247+
**Go subtests with 0ms duration**
248+
249+
`go test` reports the top-level test duration but does not always emit per-subtest durations. When the duration arrives as 0ms, the AI failure analysis can mistake a slow subtest for an instantaneous one and miss timing-related context. Post-process your JUnit XML to reflect real subtest durations before uploading, or rely on the raw stack trace rather than the AI summary for these cases.
250+
</Info>

flaky-tests/get-started/frameworks/gotestsum.mdx

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -92,7 +92,11 @@ go test -json ./... 2>&1 | go-junit-report -parser gojson -out report-go-junit.x
9292

9393
The tools will write a JUnit test report to the file specified (e.g., `junit-gotestsum.xml` or `report-go-junit.xml`). You'll need this path when configuring uploads to Trunk.
9494

95-
### Disable Retries
95+
<Info>
96+
If a subtest's top-level duration is reported as 0ms, the AI failure analysis in the dashboard can lose timing context. See [Known limitations](../../dashboard#known-limitations) for the recommended workaround.
97+
</Info>
98+
99+
#### Disable Retries
96100

97101
Regardless of the tool chosen, you need to disable automatic retries if you previously enabled them. Retries compromise the accurate detection of flaky tests.\
98102
\

flaky-tests/get-started/frameworks/rspec/index.mdx

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -56,7 +56,11 @@ You can find the Gem for `rspec_trunk_flaky_tests` [here](https://rubygems.org/g
5656
bundle update rspec_trunk_flaky_tests
5757
```
5858

59-
## Environment Variables
59+
<Info>
60+
If a failing test raises `RSpec::Core::MultipleExceptionError`, only one of the captured exceptions appears in the failure detail view today. See [Known limitations](../../../dashboard#known-limitations) for the current status and workaround.
61+
</Info>
62+
63+
### Environment Variables
6064

6165
These optional environment variables can be set in your project to change the behavior of the RSpec plugin.
6266

0 commit comments

Comments
 (0)