Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 16 additions & 1 deletion demo-ecommerce/abtests.config.ts
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,7 @@ export default defineConfig({
browser: 'chromium',
args: ['--no-sandbox'],
},
viewports: ['phone'],

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This scopes visreg down to viewports: ['phone'] (dropping desktop/tablet), and below perf drops to phone-only too, plus numberOfMeasurements goes 10→8. These are global config changes that apply to every AB run for this demo, not just bisect runs — worth confirming that's intentional. If it's purely to keep bisect (which rebuilds per-commit) fast, consider scoping it to the bisect category only if the config surface supports per-category viewport overrides, so regular (non-bisect) visreg/perf runs keep full desktop/tablet coverage and full measurement count. Otherwise this is a real, unstated reduction in demo test coverage/statistical power.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Desktop Visual Tests Are Skipped

The viewport planner intersects this global list with each test's viewport list. With only phone enabled, desktop-only Admin tests and the tablet/desktop Product Detail Actions test produce no visual work, so regressions in those tests disappear instead of remaining unaffected.

maxNumDiffPixels: 50,
defaultMisMatchThreshold: 0.1,
},
Expand All @@ -76,15 +77,29 @@ export default defineConfig({
// `screenEmulation` are NOT set here — the viewport referenced from
// shared.viewports owns them; the runner lowers them via
// lhConfigForViewport.
viewports: ['phone'],
lighthouseConfig: LIGHTHOUSE_CONFIG,
numberOfMeasurements: 10,
numberOfMeasurements: 8,
pValueThreshold: 0.01,
},

audit: {
lighthouseConfig: LIGHTHOUSE_CONFIG,
},

bisect: {
rebuildCommands: [
{
description: 'Install JavaScript dependencies',
command: 'yarn install --immutable',
},
{
description: 'Precompile application assets',
command: 'rm -rf public/packs tmp/cache && SECRET_KEY_BASE_DUMMY=1 ./bin/rails assets:precompile',
},
],
},

twinServers: {
experimentDir: process.cwd(),
controlDir: process.env.CONTROL_REPO_DIR || '../../shaka-perf-control/demo-ecommerce',
Expand Down
28 changes: 27 additions & 1 deletion demo-ecommerce/app/javascript/components/pages/HomePage.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -17,17 +17,42 @@ import LoadingSpinner from '../shared/LoadingSpinner';
import LazySection from '../shared/LazySection';
import ExperimentA11yRegressions from '../shared/ExperimentA11yRegressions';

const BisectAccessibilityProbe: React.FC = () => (
<button
type="button"
data-cy="bisect-a11y-probe"
style={{ position: 'absolute', left: -10000, top: 0, width: 1, height: 1 }}
/>
);

const runMerchandisingWarmup = () => {
const deadline = performance.now() + 450;
let checksum = 0;

while (performance.now() < deadline) {
checksum += Math.sqrt(checksum + 1);
}

if (checksum === Number.POSITIVE_INFINITY) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

checksum === Number.POSITIVE_INFINITY is effectively unreachable (a ~450ms sum of sqrt increments can't overflow to Infinity), so this is dead code kept only to dodge dead-code elimination of the busy-loop. That's fine as a fixture technique, but as written it reads like a real conditional. A one-line comment explaining "kept to prevent the loop from being optimized away" would save the next reader from treating this as a real check. Same pattern in ProductDetailPage.tsx.

console.info(checksum);
}
};

const HomePage: React.FC = () => {
const { products, loading, error } = useProducts();
const featuredProducts = products.filter((p) => p.featured).slice(0, 4);

React.useEffect(() => {
runMerchandisingWarmup();
}, []);

return (
<Box sx={{ bgcolor: '#f5f5f5', minHeight: '100vh' }}>
{/* Hero Section */}
<Box
data-cy="hero-section"
sx={{
background: 'linear-gradient(135deg, #4f46b5 0%, #764ba2 100%)',
background: 'linear-gradient(135deg, #b45309 0%, #92400e 100%)',
color: 'white',
py: { xs: 6, md: 10 },
mb: 6,
Expand Down Expand Up @@ -67,6 +92,7 @@ const HomePage: React.FC = () => {
</Box>

<Container maxWidth="lg">
<BisectAccessibilityProbe />
<ExperimentA11yRegressions />

{/* Features Section */}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -24,11 +24,28 @@ import { useCart } from '../../hooks/useCart';
import LoadingSpinner from '../shared/LoadingSpinner';
import ProductFeatures from '../shared/ProductFeatures';

const runRecommendationScoring = () => {
const deadline = performance.now() + 350;
let checksum = 0;

while (performance.now() < deadline) {
checksum += Math.sqrt(checksum + 1);
}

if (checksum === Number.POSITIVE_INFINITY) {
console.info(checksum);
}
};

const ProductDetailPage: React.FC = () => {
const { id } = useParams<{ id: string }>();
const { product, loading, error } = useProduct(Number(id));
const { addToCart } = useCart();

React.useEffect(() => {
runRecommendationScoring();
}, []);

if (loading) {
return <LoadingSpinner />;
}
Expand Down Expand Up @@ -78,6 +95,7 @@ const ProductDetailPage: React.FC = () => {
maxHeight: 400,
objectFit: 'cover',
borderRadius: 2,
boxShadow: '0 0 0 8px rgba(180, 83, 9, 0.35)',
}}
/>
</Box>
Expand Down
90 changes: 90 additions & 0 deletions demo-ecommerce/docs/git-bisect-seed-history.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,90 @@
# Git Bisect Seed History

This branch is intentionally shaped for future shaka-perf git bisect testing.
Commits alternate between harmless demo maintenance and deterministic regression
fixtures so category-specific bisect flows have clear first-bad targets.

The regression fixtures are artificial by design, but each one is scoped to the
demo app and kept easy to inspect.

Clean commits in this branch should stay runtime-neutral. They can update this
note or other demo-only documentation without changing measured pages.

Runtime fixtures should prefer existing measured routes so future bisect tests
do not need a special-case demo configuration.

The homepage fixtures are intended for the existing homepage and shop-now
coverage. The product detail fixture is intended for the existing product detail
coverage.

Performance fixtures should use deterministic client-side work instead of
network delays so results remain easy to reproduce in local containers.

Accessibility fixtures should avoid layout-visible changes when they are meant
to be accessibility-only.

Clean commits should be safe to classify as good for every category that has
not yet received its first dedicated regression fixture.

Combined regression commits are useful for checking that category-specific
bisect still reports the earliest bad commit for each individual category.

Product detail fixtures should avoid changing product data or cart behavior, so
they remain independent of Rails seeds and API responses.

Fixture commits should stay small enough that `git show` explains the category
signal without requiring a full app run.

Seed base: `38dae68` (`main` when this branch was created).

Category map:

- `623a1ae` - no regression, seed docs only.
- `fe8900e` - no regression, docs only.
- `58cc828` - first `visreg` bad commit, homepage hero color change.
- `a55e7f4` - no regression, docs only.
- `754fcd9` - no regression, docs only.
- `9c7cfff` - first `perf` bad commit, homepage CPU warmup.
- `744fe90` - no regression, docs only.
- `ac38e53` - no regression, docs only.
- `463c429` - no regression, docs only.
- `fcb0e2b` - first `accessibility` bad commit, offscreen unnamed button.
- `c1e2a62` - no regression, docs only.
- `ce1f601` - no regression, docs only.
- `3846371` - no regression, docs only.
- `5345dff` - visual plus performance regression on product detail.
- `088afb9` - no regression, docs only.
- `4406a78` - no regression, fixture-impact docs only.

Expected affected AB tests and metrics:

- `Homepage` / `visreg` - screenshot mismatch and diff pixels for
`[data-cy="hero-section"]` and `document` after the hero gradient changes.
- `Homepage` / `perf` - primarily worse `TBT` and lower `LH Score` from the
450ms homepage CPU warmup. `speed-index`, `FCP`, and `LCP` can also move
depending on timing.
- `Homepage` / `accessibility` - a new axe finding, expected to be
`button-name`, from `button[data-cy="bisect-a11y-probe"]`.
- `Click Shop Now on the homepage` / `perf` - primarily worse `TBT` and lower
`LH Score` in the phone viewport because the test starts on `/` and runs the
homepage CPU warmup.
- `Product Detail` / `visreg` - screenshot mismatch and diff pixels from the
product image `boxShadow`.
- `Product Detail` / `perf` - primarily worse `TBT` and lower `LH Score` from
the 350ms product-detail CPU warmup.
- `Product Detail - Show Product Journey Toggle` / `visreg` - likely full-page
screenshot mismatch from the product image `boxShadow` in the phone viewport.

Expected unaffected AB tests:

- `Product Detail - Desktop Actions` captures only
`[data-cy="product-actions-desktop"]`, so the product image `boxShadow` should
not appear in its screenshot.
- `Click Reviews on Product Detail` and
`Product Details => Click on Reviews => Click on Deals` end on reviews or
deals pages and run as visual-only tests, so the product-detail image change
should not be captured.
- `Products List`, `Cart`, `Carousel`, and `Admin` do not visit touched routes
or changed fixtures.
- Product-detail accessibility should stay unchanged because no semantic
accessibility regression was added there.
Loading