feat: free Lighthouse audits via PageSpeed Insights when PAGESPEED_API_KEY is set

[mattmacrocket]

What

Site-audit Lighthouse currently runs on the metered DataForSEO OnPage endpoint ($0.00425/page) even though Google's PageSpeed Insights API returns the same Lighthouse report for free (roughly 25k requests/day with a free API key). This PR routes the audit's Lighthouse phase to PSI when a PAGESPEED_API_KEY is set, and falls back to the existing DataForSEO path otherwise, so current deployments keep their exact behavior with zero action.

• New src/server/lib/pagespeedLighthousePayload.ts: fetches PSI and normalizes the report into the exact same StoredLighthousePayload the DataForSEO path produces. R2 payloads, scores, metrics, and the issue-details view are provider-agnostic; the stored-payload source enum accepts both providers and existing stored payloads still parse.
  • Handles a Lighthouse 13.x quirk: details.items comes back as an object (not array) for some audits (e.g. document-latency-insight), which would otherwise fail the whole parse.
  • 90s timeout (PSI runs can be slow), same 3-attempt retry loop as before.
• Plumbs PAGESPEED_API_KEY through .env.example, compose.yaml, env.d.ts, and documents it in docs/SELF_HOSTING_DOCKER.md.

Why opt-in via key rather than always PSI

Keyless PSI requests share a tiny anonymous global quota that is almost always exhausted (HTTP 429 in practice), so PSI is only reliable with a key. Gating on the key means: self-hosters get free audits with a 2-minute free key, and deployments that do nothing (including hosted) are completely unaffected.

Notes for review

• Hosted: setting PAGESPEED_API_KEY on the hosted deployment would make audit Lighthouse free there too (no DataForSEO OnPage spend, no user credit metering on that path). Without the key, hosted behavior is unchanged.
• The old DataForSEO Lighthouse client path stays intact (it is the fallback). If you ever want PSI-only, client.lighthouse.live and dataforseoLighthousePayload.ts could be removed in a follow-up.

Verification

• Live end-to-end audit (10 pages, Lighthouse on, key set): 8/8 Lighthouse runs via PSI (4 sampled pages x mobile/desktop), scores and per-page issue details render unchanged, zero DataForSEO credits spent (account balance verified before/after).
• Fallback path exercised by unsetting the key (identical to current behavior).
• tsc --noEmit, oxlint --type-aware, prettier --check, knip, and the full vitest suite (408 tests) pass.

🤖 Generated with Claude Code

---

Follow-up (review round 1)

• Root cause of the timeouts/500s you hit: the audit's Lighthouse batch fires up to 10 URLs x 2 strategies at PSI simultaneously, and PSI's Lighthouse backend rejects parallel bursts with exactly that HTTP 500 ("Lighthouse returned error: Something went wrong.") and queues the rest past the request timeout. Measured against openseo.so: 20 concurrent runs fail ~10% instantly, 4 concurrent completes 20/20. All PSI calls now go through a small in-module queue (4 at a time; queue wait does not consume the request budget) and the per-request timeout is 120s (successful runs under load measured at 60-90s).
• runLighthouse defaults back to false and the extra checkbox copy is removed, per the inline comments.
• New test script: pnpm pagespeed:check --url=https://openseo.so/,https://openseo.so/docs/skills exercises the exact audit code path (same module, queue, and 3-attempt retry shape). Reads PAGESPEED_API_KEY from .env.local. --attempts=1 shows raw single-shot behavior, --strategy= and --sequential=true also available. Verified 6/6 against the URLs from your review log.
• Rebased on current main.

Greptile Summary

This PR adds an opt-in PageSpeed Insights path for site-audit Lighthouse runs. The main changes are:

• Routes Lighthouse audits to PageSpeed Insights when PAGESPEED_API_KEY is configured.
• Keeps the existing DataForSEO Lighthouse path as the fallback when the key is absent.
• Normalizes PageSpeed Insights responses into the existing stored Lighthouse payload shape.
• Adds PSI concurrency limiting, timeout handling, and response validation.
• Documents the new env var and adds a pagespeed:check script for local verification.

Confidence Score: 5/5

Safe to merge with low risk.

The PSI path is opt-in, validates responses with zod, reuses the existing payload builders, and preserves the DataForSEO fallback when the key is absent.

No files require special attention.

T-Rex Logs

What T-Rex did

• Inspected the proof log to confirm that each validation step records command, working directory, timestamps, outputs, and exit codes.
• Confirmed the CLI load check exited with code 1 and displayed the expected PAGESPEED_API_KEY is not set message and usage text.
• Verified the TypeScript check ran with no emitted output and exited successfully.
• Validated the lint step completed with zero warnings and zero errors.
• Verified the Vitest run finished with 3 test files and 11 tests passing.

_{Ran code and verified through T-Rex}

Important Files Changed

Filename	Overview
src/server/lib/audit/lighthouse.ts	Routes Lighthouse fetches to PageSpeed Insights when the API key is configured, otherwise preserving DataForSEO fallback behavior.
src/server/lib/pagespeedLighthousePayload.ts	Adds PSI response fetching, concurrency limiting, zod normalization, and provider-neutral stored payload construction.
src/server/lib/lighthouseStoredPayload.ts	Extends stored Lighthouse payload parsing to accept both DataForSEO and PageSpeed Insights sources.
scripts/pagespeed-lighthouse-check.ts	Adds a CLI smoke-test that loads local env, runs PSI checks for URLs and strategies, and reports retry outcomes.
compose.yaml	Passes optional PAGESPEED_API_KEY through the Docker Compose service environment.
docs/SELF_HOSTING_DOCKER.md	Adds self-hosting instructions for enabling free PageSpeed Insights Lighthouse audits.

Sequence Diagram

%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
participant Audit as Site audit Lighthouse phase
participant Env as Runtime env
participant PSI as PageSpeed Insights
participant DFS as DataForSEO Lighthouse
participant R2 as R2 payload storage

Audit->>Env: read PAGESPEED_API_KEY
alt key is configured
    Audit->>PSI: fetch Lighthouse report (queued, retryable)
    PSI-->>Audit: lighthouseResult
    Audit->>Audit: normalize to StoredLighthousePayload
else key is absent
    Audit->>DFS: lighthouse.live(url, strategy)
    DFS-->>Audit: StoredLighthousePayload
end
Audit->>R2: store provider-neutral payload JSON
R2-->>Audit: r2Key and payload size

Loading

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
participant Audit as Site audit Lighthouse phase
participant Env as Runtime env
participant PSI as PageSpeed Insights
participant DFS as DataForSEO Lighthouse
participant R2 as R2 payload storage

Audit->>Env: read PAGESPEED_API_KEY
alt key is configured
    Audit->>PSI: fetch Lighthouse report (queued, retryable)
    PSI-->>Audit: lighthouseResult
    Audit->>Audit: normalize to StoredLighthousePayload
else key is absent
    Audit->>DFS: lighthouse.live(url, strategy)
    DFS-->>Audit: StoredLighthousePayload
end
Audit->>R2: store provider-neutral payload JSON
R2-->>Audit: r2Key and payload size

Loading

_{Reviews (1): Last reviewed commit: "review: cap PSI concurrency, keep Lighth..." | Re-trigger Greptile}

[bensenescu]

Could you add a script which exercises this new logic that I can easily test? When I tried to test this, it didn't work and everything timed out:

    "code": 500,
    "message": "Lighthouse returned error: Something went wrong.",
    "errors": [
      {
        "message": "Lighthouse returned error: Something went wrong.",

Lighthouse attempt 1 failed for https://openseo.so/docs/skills: PageSpeed Insights HTTP 500: {
  "error": {
    "code": 500,
    "message": "Lighthouse returned error: Something went wrong.",
    "errors": [
      {
        "message": "Lighthouse returned error: Something went wrong.",

Lighthouse attempt 1 failed for https://openseo.so/: The operation was aborted due to timeout
Lighthouse attempt 1 failed for https://openseo.so/docs/skills: The operation was aborted due to timeout
Lighthouse attempt 1 failed for https://openseo.so/docs/skills/competitor-analysis: The operation was aborted due to timeout
Lighthouse attempt 1 failed for https://openseo.so/docs/skills/competitor-analysis: The operation was aborted due to timeout
Lighthouse attempt 2 failed for https://openseo.so/: PageSpeed Insights HTTP 500: {
  "error": {
    "code": 500,
    "message": "Lighthouse returned error: Something went wrong.",
    "errors": [
      {
        "message": "Lighthouse returned error: Something went wrong.",

Lighthouse attempt 2 failed for https://openseo.so/docs/skills/competitor-analysis: PageSpeed Insights HTTP 500: {
  "error": {
    "code": 500,
    "message": "Lighthouse returned error: Something went wrong.",
    "errors": [
      {
        "message": "Lighthouse returned error: Something went wrong.",

Lighthouse attempt 3 failed for https://openseo.so/docs/skills/competitor-analysis: PageSpeed Insights HTTP 500: {
  "error": {
    "code": 500,
    "message": "Lighthouse returned error: Something went wrong.",
    "errors": [
      {
        "message": "Lighthouse returned error: Something went wrong.",

Lighthouse failed after 3 attempts for https://openseo.so/docs/skills/competitor-analysis: PageSpeed Insights HTTP 500: {
  "error": {
    "code": 500,
    "message": "Lighthouse returned error: Something went wrong.",
    "errors": [
      {
        "message": "Lighthouse returned error: Something went wrong.",

Lighthouse attempt 2 failed for https://openseo.so/docs/skills: The operation was aborted due to timeout
Lighthouse attempt 3 failed for https://openseo.so/docs/skills: PageSpeed Insights HTTP 500: {
  "error": {
    "code": 500,
    "message": "Lighthouse returned error: Something went wrong.",
    "errors": [
      {
        "message": "Lighthouse returned error: Something went wrong.",

Lighthouse failed after 3 attempts for https://openseo.so/docs/skills: PageSpeed Insights HTTP 500: {
  "error": {
    "code": 500,
    "message": "Lighthouse returned error: Something went wrong.",
    "errors": [
      {
        "message": "Lighthouse returned error: Something went wrong.",

[bensenescu]

Let's keep this false as this will still cost people money who haven't set up the key?

[mattmacrocket]

Done, reverted to false. Agreed: without the key this would have pushed everyone onto the metered path by default.

[bensenescu]

Let's remove this copy to keep things simple

[mattmacrocket]

Removed.

[mattmacrocket]

Thanks for the test run, that log was exactly what I needed to reproduce it.

Root cause: the audit's Lighthouse phase fires batches of up to 10 URLs x 2 strategies at PSI simultaneously, and PSI's Lighthouse backend rejects parallel bursts with that exact HTTP 500 ("Lighthouse returned error: Something went wrong.") while queueing the rest past the request timeout. Reproduced against openseo.so: 20 concurrent runs fail ~10% instantly with the 500; 4 concurrent completes 20/20.

Fixed in the new commit:

• All PSI calls now go through a small in-module queue (4 concurrent max; time spent waiting for a slot does not consume the request timeout), and the per-request timeout is 120s (under load, successful runs measured at 60-90s).
• runLighthouse defaults back to false and the extra copy is removed, per your inline comments.

Test script (reads PAGESPEED_API_KEY from .env.local):

pnpm pagespeed:check --url=https://openseo.so/,https://openseo.so/docs/skills,https://openseo.so/docs/skills/competitor-analysis

It exercises the exact audit code path (same module, same queue, same 3-attempt retry shape). Sample run against the URLs from your log:

OK   desktop https://openseo.so/docs/skills perf=100 a11y=95 bp=100 seo=100 (11.1s, attempt 1/3, lighthouse 13.4.0)
OK   mobile  https://openseo.so/docs/skills perf=96 a11y=95 bp=100 seo=100 (13.2s, attempt 1/3, lighthouse 13.4.0)
OK   mobile  https://openseo.so/ perf=91 a11y=95 bp=100 seo=100 (13.4s, attempt 1/3, lighthouse 13.4.0)
OK   desktop https://openseo.so/ perf=100 a11y=95 bp=100 seo=100 (15.0s, attempt 1/3, lighthouse 13.4.0)
OK   mobile  https://openseo.so/docs/skills/competitor-analysis perf=96 a11y=95 bp=100 seo=100 (24.9s, attempt 1/3, lighthouse 13.4.0)
OK   desktop https://openseo.so/docs/skills/competitor-analysis perf=100 a11y=95 bp=100 seo=100 (25.9s, attempt 1/3, lighthouse 13.4.0)

6/6 succeeded in 25.9s.

--attempts=1 shows raw single-shot behavior (PSI does still shed individual runs in bad windows; the audit's per-URL retries absorb that), and --sequential=true / --strategy= are there too.

Also rebased on current main, and the PR description is updated to match (default stays off). pnpm ci:check and the full test suite pass.

🤖 Generated with Claude Code

[bensenescu]

https://github.com/greptileai

[bensenescu]

@greptileai