feat: track caught errors and error locations in step traces - #696
Merged
Merged
Conversation
A test that catches an assertion error and goes on (an optional dialog probed in a try/catch, a retried toPass attempt) still records that assertion as an errored step and an errored trace action. The timeline, the trace evidence, the clues and the AI context all took the first errored step or action as "the failure", so they pointed at the caught probe while the headline named the real error. - Reporter: flattened steps keep where each error was thrown, and a step whose error matches none of the test's errors (first line, then location) is marked `recovered`. Blob-report imports do the same. - Core: `failingStepIndex` takes the execution's error text. It skips recovered steps, walks each failing chain down through the steps that carry its error, prefers the chain thrown where the error text points, then one matching the error's first line, else the last failing chain of the body. `stepFailureRoles` tells the failing step, the steps around it, a second failure (soft assertion, failing teardown) and a caught error apart. Headline helpers pass the error through. - Trace: the failing action is the one whose error matches a test-level `error` event, called where its stack points, the innermost of them; else the latest errored runner action. Snapshots, the call stack and the fallback ARIA tree read the page-side call a runner action drove. - Timeline: the failing row, the window and the auto-opened section follow the fatal step; a caught error is greyed out and marked as such, and never counts as a section failure. - AI context: "Failed Steps" lists the failing chain first and labels caught errors; "Steps" and "Actions Leading to Failure" mark them. Fixes #695 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MmFBpnCmRPzkjMedY3A6GZ
The page at the failing step now sits in the step's own block on the timeline table (it was a separate full-width row under it), and it adds the trace's DOM snapshot of the same moment as the screenshot: the failing action's after-phase DOM next to its after-phase screenshot, the failure-time DOM next to the run's own failure screenshot. - `dom-snapshot` and `dom-snapshot-frame` take optional `callId` and `phase` query parameters naming the action snapshot to render first, falling back to the failure-time one; the demo mirrors them. - The scaled, sandboxed page render moves out of the page structure card into `DomSnapshotFrame` and `useDomSnapshot`, shared by both. The stage never sizes its container, so a table cell keeps its width. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MmFBpnCmRPzkjMedY3A6GZ
Contributor
Coverage Report for Reporter (./packages/reporter)
File Coverage
|
||||||||||||||||||||||||||||||||||||||
Contributor
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
This change adds support for tracking errors that tests catch and recover from (e.g., probes in
try/catchblocks, retriedtoPassattempts), as well as precise error locations in step traces. The reporter now marks these recovered errors distinctly from fatal failures, allowing the UI to display them as muted rather than as the primary failure.Key additions:
file:line:col), enabling precise error attributionrecoveredflag marks errors the test caught and continued from, distinguishing them from fatal failuresstepFailureRoles()function identifies which steps are the actual failure vs. caught errors vs. enclosing stepssameCodeLocation()utility compares error locations across different representations (parsed objects or strings)The failing step is now defined as the innermost step of the failing chain that carries the execution's own error, not just any error. This allows proper handling of multi-error scenarios where some errors are caught and others are fatal.
How was it tested
step-tree.test.tscovering:toPassretry failuresplaywright-steps.tswith real Playwright 1.63 trace examplescaught-error-traces.tsfor trace parsing teststimeline-rows.test.tsto verify recovered steps are marked correctlytrace-parser.test.tsto handle new trace action fields (parentId,location)dom-snapshot.test.tsto test snapshot extraction from runner actionsChecklist
feat: ...)https://claude.ai/code/session_01MmFBpnCmRPzkjMedY3A6GZ