EDU-18161/18154/18162: restore signal in tests-summary PR comment (Phase 1)#1249
Open
PedroAntunesCosta wants to merge 45 commits into
Open
EDU-18161/18154/18162: restore signal in tests-summary PR comment (Phase 1)#1249PedroAntunesCosta wants to merge 45 commits into
PedroAntunesCosta wants to merge 45 commits into
Conversation
✅ Deploy Preview for elated-hoover-5c29bf ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Performance ResultsPage /docs/api-reference/headless-cms-api#get-/_v/cms/api/-projectId-/-content-type-
Complete results here Page /updates/release-notes/improvement-price-consistency-checkout-discounts-installments
Complete results here Overall Performance
|
End-to-end tests (seed
|
PedroAntunesCosta
changed the title
Each failed test now emits a JSON line to cypress-failures.jsonl with spec, title, attempt, type, and message fields instead of a plain-text line in cypress.log. All callers updated accordingly. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
summary-report.js now reads cypress-failures.jsonl, groups records by spec and title, and reports the number of unique failing tests regardless of how many times each was retried. A (retried Nx) suffix is shown when a test was attempted more than once. Workflow files updated to preserve, merge, and clean up cypress-failures.jsonl instead of the old cypress.log. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…spec Replace the expect() status assertion with a conditional that sets a failureType flag (http / load_timeout / dom) before the potentially failing command. afterEach reads the flag so writeLog emits the correct type for each failure rather than always emitting 'dom'. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
summary-report.js now renders three sections: - Content regressions (type=dom) — drives the headline count - Preview infrastructure (type=http/load_timeout) — visible but not counted in the headline, each entry shows the status code or timeout - Other — defensive bucket for unrecognised types The headline "N tests failed" now reflects only actionable content failures. Infrastructure noise is surfaced separately so reviewers can distinguish flaky preview from real broken pages. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
load_timeout entries had no explanation in the Preview infrastructure section. Now falls back to the type name (e.g. "load timeout") so reviewers always know why a failure was classified as infrastructure. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
When the Cypress component tests crash (not just test failures), no cypress-failures.jsonl is written. Previously the report ran with an empty input and published "All tests were successful!" which is wrong. Now the report only runs when failure records exist OR the component test step actually completed successfully. A Cypress crash instead publishes a "failed to run" message. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Without a timeout, the integration tests can run for GitHub's full 6-hour job limit (e.g. when the sampled pages include many slow API reference pages), preventing the summary report step from ever executing.
…pages section - Surface the sample seed in the top-level H1 so it's immediately visible - Content regressions / Preview infrastructure / Other sections are always rendered; each falls back to a "none" line when the bucket is empty - Sampled pages list remains at the bottom of the comment - Remove the now-inlined printSampleMetadata helper
Adds src/tests/utils/__tests__/summary-report.test.js covering retry collapsing, multi-spec aggregation, type partitioning, malformed JSONL tolerance, specDisplayName, and empty-input edge cases (node:test, no new dependencies). Adds `test:utils` script to package.json and a pre-component-tests CI step so a regression in report logic blocks merge. Closes the missing acceptance criterion for Task 2 (EDU-18154).
…pi-reference spec Adds a polling helper that waits for rapi-doc's resolvedSpec + scrollToPath to be ready, replacing the fixed 10-second wait. Part of EDU-16758.
Replaces opacity-based visibility checks on .toggleIcon with data-cy="sidebar-section" state checks and force-clicks. Part of EDU-16758.
Replaces raw cy.visit() in visitGuidePage with visitPageAllowingLoadTimeout so a slow preview page produces a visible skip rather than a silent 60-min CI hang. Guards 'try to send feedback' and the TOC test behind sawLoadTimeout alias.
PedroAntunesCosta
force-pushed
the
fix/edu-18161-deterministic-doc-sampling
branch
from
1fe71c0 to
892f966
Compare
…nistic-doc-sampling
Reduce the Cypress step timeout and drop headed mode so CI fails faster and uses less runner time.
A swallowed remote page-load timeout aborts the command queue, so the happy-path cleanup that removed the global fail handler never ran. The leaked handler then bled into later tests, stalling healthy assertions and leaving the guide load-timeout alias unregistered. Remove the handler inside the handler itself so it can never leak, while preserving the slow-load tolerance and the value yielded to callers.
Disambiguate the populated desktop TOC from the mobile shell so anchor-link assertions resolve reliably in CI.
Table of contents entries use full preview URLs with a hash fragment, not hash-only hrefs, so the api-guides spec now queries a[href*="#"].
Pre-register the alias so feedback tests do not fail when a swallowed page load timeout aborts the visit chain before .as() runs.
Collapsed AnimateHeight sections hide nested links; filter to :visible entries before selecting the last anchor to click.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR started as the
tests-summarysignal-restoration work and now also carries the interim Cypress fixes needed on the same branch for#1239and#1242.Linked issues and Jira tasks
documentation-pages-status.cy.jsdeterministic and unify navigation sourcetests-summaryPR comment to count distinct failing testsdocumentation-pages-status.cy.js@vtexdocs/componentshook bump remains tracked separatelyapi-reference.cy.jsresilient to slow loads on/docs/api-reference#1268into#1249Changes
Deterministic page sampling (
select-pages.js,cypress.yml,cypress-extensive.yml)Math.random()with a seeded PRNG (mulberry32 over a SHA hash) inselect-pages.jsSAMPLE_SEEDenv var; fall back to a time-based seed for local runsSAMPLE_SEED: ${{ github.event.pull_request.head.sha || github.sha }}incypress.ymlandSAMPLE_SEED: ${{ github.run_id }}incypress-extensive.ymlFetch navigation.jsonstep -public/navigation.jsonfrom the PR branch is the single navigation source for both workflowscypress-sample.json; render a collapsible "Sampled pages" section at the bottom of the comment with the seed in the H1 headingStructured failure records (
support/functions.js,documentation-pages-status.cy.js)writeLog(title)withwriteLog({ spec, title, attempt, type, message })emitting one JSONL record tocypress-failures.jsonlper failuredocumentation-pages-status.cy.jsclassifies each failure before the failing command: non-2xx responses ->http(status code in message), page-load timeouts ->load_timeout, DOM assertion failures ->domSummary report (
summary-report.js)cypress-failures.jsonlline-by-line; skip malformed lines without crashingspec x title, keeping the highest-attempt entry per unique test(retried Nx)suffix when a test was attempted more than oncedom), Preview infrastructure (http/load_timeout), Other (defensive bucket)N tests failedcounts only content regressions; infrastructure failures are surfaced but not counted in the headlineWorkflow robustness (
cypress.yml)Unit tests (utils)step before component tests so a regression in report logic blocks mergeUnit tests (
src/tests/utils/__tests__/summary-report.test.js)7 tests using
node:test+node:assert/strict(no new dependencies):countDistinct === 1,maxAttempt === 2countDistinct === 3, per-spec map sizes correctdom/http/load_timeout/unknown ->content/infra/infra/otherbucketsnot-json-> returns only the valid record, no throwspecDisplayName: strips.cy.js/.cy.ts/.cy.tsxand directory prefix[];countDistinct(collapseFailures([])) === 0Interim Cypress fixes for
#1239/#1242(api-guides.cy.js,api-reference.cy.js,support/commands.js,support/network.js)waitForRapiDocReady()and replace the fixedcy.wait(10000)boot wait inapi-reference.cy.jsapi-guides.cy.jsandapi-reference.cy.jsso they assert sidebar content visibility and use forced.toggleIconclicks instead of the old visibility paradoxvisitPageAllowingLoadTimeout()in a newsupport/network.js: it swallows remote page-load timeouts and removes its globalfailhandler inside the handler itself, so a swallowed timeout can no longer leak the handler into later tests and stall the job (root cause of the prior 25-minute integration-job timeouts)api-guides.cy.jsby addinggetDesktopTableOfContents()/getTocAnchorId()/getDesktopTocEntryLinks()- they target the populated desktop right-rail TOC (disambiguated from the empty mobile shell via the instance containing[data-cy="feedback-section"]) and matcha[href*="#"], replacing the:visibleselector that resolved to zero anchor linksPlanned follow-up CI issues
The items below are scoped out of this PR and will be tackled separately after this merges:
EDU-16758 scope addition
Adds the EDU-16758 regression suite (5 Cypress cases) carved out of #1268, which shipped the product fix without tests. See plans/api-navitagion-bug/ for context.
Closes
Closes #1240
Closes #1234
Closes #1241
Closes #1239 (interim spec-side patch; the proper @vtexdocs/components hook bump for the toggle and category items is tracked separately)
Closes #1242