-
Notifications
You must be signed in to change notification settings - Fork 1
test(demo): add bisect seed history #99
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
1ad6c45
8de754b
76ba114
5c902c7
0e5e600
c2ee811
44cca91
e71c449
844590c
4f4434f
678abc6
5483ac5
a0af6e6
bc79de2
8012cc1
a5adce5
d25d667
068c9fa
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -63,6 +63,7 @@ export default defineConfig({ | |
| browser: 'chromium', | ||
| args: ['--no-sandbox'], | ||
| }, | ||
| viewports: ['phone'], | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
The viewport planner intersects this global list with each test's viewport list. With only |
||
| maxNumDiffPixels: 50, | ||
| defaultMisMatchThreshold: 0.1, | ||
| }, | ||
|
|
@@ -76,15 +77,29 @@ export default defineConfig({ | |
| // `screenEmulation` are NOT set here — the viewport referenced from | ||
| // shared.viewports owns them; the runner lowers them via | ||
| // lhConfigForViewport. | ||
| viewports: ['phone'], | ||
| lighthouseConfig: LIGHTHOUSE_CONFIG, | ||
| numberOfMeasurements: 10, | ||
| numberOfMeasurements: 8, | ||
| pValueThreshold: 0.01, | ||
| }, | ||
|
|
||
| audit: { | ||
| lighthouseConfig: LIGHTHOUSE_CONFIG, | ||
| }, | ||
|
|
||
| bisect: { | ||
| rebuildCommands: [ | ||
| { | ||
| description: 'Install JavaScript dependencies', | ||
| command: 'yarn install --immutable', | ||
| }, | ||
| { | ||
| description: 'Precompile application assets', | ||
| command: 'rm -rf public/packs tmp/cache && SECRET_KEY_BASE_DUMMY=1 ./bin/rails assets:precompile', | ||
| }, | ||
| ], | ||
| }, | ||
|
|
||
| twinServers: { | ||
| experimentDir: process.cwd(), | ||
| controlDir: process.env.CONTROL_REPO_DIR || '../../shaka-perf-control/demo-ecommerce', | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -17,17 +17,42 @@ import LoadingSpinner from '../shared/LoadingSpinner'; | |
| import LazySection from '../shared/LazySection'; | ||
| import ExperimentA11yRegressions from '../shared/ExperimentA11yRegressions'; | ||
|
|
||
| const BisectAccessibilityProbe: React.FC = () => ( | ||
| <button | ||
| type="button" | ||
| data-cy="bisect-a11y-probe" | ||
| style={{ position: 'absolute', left: -10000, top: 0, width: 1, height: 1 }} | ||
| /> | ||
| ); | ||
|
|
||
| const runMerchandisingWarmup = () => { | ||
| const deadline = performance.now() + 450; | ||
| let checksum = 0; | ||
|
|
||
| while (performance.now() < deadline) { | ||
| checksum += Math.sqrt(checksum + 1); | ||
| } | ||
|
|
||
| if (checksum === Number.POSITIVE_INFINITY) { | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
|
||
| console.info(checksum); | ||
| } | ||
| }; | ||
|
|
||
| const HomePage: React.FC = () => { | ||
| const { products, loading, error } = useProducts(); | ||
| const featuredProducts = products.filter((p) => p.featured).slice(0, 4); | ||
|
|
||
| React.useEffect(() => { | ||
| runMerchandisingWarmup(); | ||
| }, []); | ||
|
|
||
| return ( | ||
| <Box sx={{ bgcolor: '#f5f5f5', minHeight: '100vh' }}> | ||
| {/* Hero Section */} | ||
| <Box | ||
| data-cy="hero-section" | ||
| sx={{ | ||
| background: 'linear-gradient(135deg, #4f46b5 0%, #764ba2 100%)', | ||
| background: 'linear-gradient(135deg, #b45309 0%, #92400e 100%)', | ||
| color: 'white', | ||
| py: { xs: 6, md: 10 }, | ||
| mb: 6, | ||
|
|
@@ -67,6 +92,7 @@ const HomePage: React.FC = () => { | |
| </Box> | ||
|
|
||
| <Container maxWidth="lg"> | ||
| <BisectAccessibilityProbe /> | ||
| <ExperimentA11yRegressions /> | ||
|
|
||
| {/* Features Section */} | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,90 @@ | ||
| # Git Bisect Seed History | ||
|
|
||
| This branch is intentionally shaped for future shaka-perf git bisect testing. | ||
| Commits alternate between harmless demo maintenance and deterministic regression | ||
| fixtures so category-specific bisect flows have clear first-bad targets. | ||
|
|
||
| The regression fixtures are artificial by design, but each one is scoped to the | ||
| demo app and kept easy to inspect. | ||
|
|
||
| Clean commits in this branch should stay runtime-neutral. They can update this | ||
| note or other demo-only documentation without changing measured pages. | ||
|
|
||
| Runtime fixtures should prefer existing measured routes so future bisect tests | ||
| do not need a special-case demo configuration. | ||
|
|
||
| The homepage fixtures are intended for the existing homepage and shop-now | ||
| coverage. The product detail fixture is intended for the existing product detail | ||
| coverage. | ||
|
|
||
| Performance fixtures should use deterministic client-side work instead of | ||
| network delays so results remain easy to reproduce in local containers. | ||
|
|
||
| Accessibility fixtures should avoid layout-visible changes when they are meant | ||
| to be accessibility-only. | ||
|
|
||
| Clean commits should be safe to classify as good for every category that has | ||
| not yet received its first dedicated regression fixture. | ||
|
|
||
| Combined regression commits are useful for checking that category-specific | ||
| bisect still reports the earliest bad commit for each individual category. | ||
|
|
||
| Product detail fixtures should avoid changing product data or cart behavior, so | ||
| they remain independent of Rails seeds and API responses. | ||
|
|
||
| Fixture commits should stay small enough that `git show` explains the category | ||
| signal without requiring a full app run. | ||
|
|
||
| Seed base: `38dae68` (`main` when this branch was created). | ||
|
|
||
| Category map: | ||
|
|
||
| - `623a1ae` - no regression, seed docs only. | ||
| - `fe8900e` - no regression, docs only. | ||
| - `58cc828` - first `visreg` bad commit, homepage hero color change. | ||
| - `a55e7f4` - no regression, docs only. | ||
| - `754fcd9` - no regression, docs only. | ||
| - `9c7cfff` - first `perf` bad commit, homepage CPU warmup. | ||
| - `744fe90` - no regression, docs only. | ||
| - `ac38e53` - no regression, docs only. | ||
| - `463c429` - no regression, docs only. | ||
| - `fcb0e2b` - first `accessibility` bad commit, offscreen unnamed button. | ||
| - `c1e2a62` - no regression, docs only. | ||
| - `ce1f601` - no regression, docs only. | ||
| - `3846371` - no regression, docs only. | ||
| - `5345dff` - visual plus performance regression on product detail. | ||
| - `088afb9` - no regression, docs only. | ||
| - `4406a78` - no regression, fixture-impact docs only. | ||
|
|
||
| Expected affected AB tests and metrics: | ||
|
|
||
| - `Homepage` / `visreg` - screenshot mismatch and diff pixels for | ||
| `[data-cy="hero-section"]` and `document` after the hero gradient changes. | ||
| - `Homepage` / `perf` - primarily worse `TBT` and lower `LH Score` from the | ||
| 450ms homepage CPU warmup. `speed-index`, `FCP`, and `LCP` can also move | ||
| depending on timing. | ||
| - `Homepage` / `accessibility` - a new axe finding, expected to be | ||
| `button-name`, from `button[data-cy="bisect-a11y-probe"]`. | ||
| - `Click Shop Now on the homepage` / `perf` - primarily worse `TBT` and lower | ||
| `LH Score` in the phone viewport because the test starts on `/` and runs the | ||
| homepage CPU warmup. | ||
| - `Product Detail` / `visreg` - screenshot mismatch and diff pixels from the | ||
| product image `boxShadow`. | ||
| - `Product Detail` / `perf` - primarily worse `TBT` and lower `LH Score` from | ||
| the 350ms product-detail CPU warmup. | ||
| - `Product Detail - Show Product Journey Toggle` / `visreg` - likely full-page | ||
| screenshot mismatch from the product image `boxShadow` in the phone viewport. | ||
|
|
||
| Expected unaffected AB tests: | ||
|
|
||
| - `Product Detail - Desktop Actions` captures only | ||
| `[data-cy="product-actions-desktop"]`, so the product image `boxShadow` should | ||
| not appear in its screenshot. | ||
| - `Click Reviews on Product Detail` and | ||
| `Product Details => Click on Reviews => Click on Deals` end on reviews or | ||
| deals pages and run as visual-only tests, so the product-detail image change | ||
| should not be captured. | ||
| - `Products List`, `Cart`, `Carousel`, and `Admin` do not visit touched routes | ||
| or changed fixtures. | ||
| - Product-detail accessibility should stay unchanged because no semantic | ||
| accessibility regression was added there. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This scopes
visregdown toviewports: ['phone'](droppingdesktop/tablet), and belowperfdrops to phone-only too, plusnumberOfMeasurementsgoes 10→8. These are global config changes that apply to every AB run for this demo, not just bisect runs — worth confirming that's intentional. If it's purely to keep bisect (which rebuilds per-commit) fast, consider scoping it to thebisectcategory only if the config surface supports per-category viewport overrides, so regular (non-bisect) visreg/perf runs keep full desktop/tablet coverage and full measurement count. Otherwise this is a real, unstated reduction in demo test coverage/statistical power.