diff --git a/CHANGELOG.md b/CHANGELOG.md index d726ad97b..6d99dc35e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,7 +8,22 @@ Semver applies from 1.0.0. A breaking change to a documented API needs a major ## [Unreleased] -Nothing yet. +### Added + +Tier 0 — core. Tier 1 — seo. Tier 4 — render. Tier 5 — cli. + +- **`seo.sitemap.sections` — a sitemap index with named, paginated sections, by configuration.** `/sitemap.xml` was one `` until a site passed 50,000 URLs. `seo: { sitemap: { pageSize: 1000, sections: [{ name: 'blog', match: '/blog/**' }, { name: 'ciudades', match: '/juzgados/**' }] } }` makes it a `` of `/sitemaps/pages-1.xml`, `/sitemaps/blog-1.xml`, `/sitemaps/blog-2.xml`, …, each a valid urlset; a page no section matches goes to the reserved `pages`. `match` is a path pattern or a list (`/precios`, `/blog/*`, `/blog/**`), read against the path without its locale prefix, so a page and its locale copies share a file; the first section that matches wins. Inside a section pages are sorted by path — the bytes depend on which pages exist, not on the order a query returned them — files are numbered from 1, a section with nothing in it writes no file, and the index `` of a child is the newest `` inside it. The web role serves every child from the route it already had (`GET /sitemaps/:file`, `application/xml`, `public, max-age=3600`, from the same hourly answer as the index — a refresh that fails keeps the previous index and its children together), the static export writes the same files, and `robots.txt` still names `/sitemap.xml` alone. A `name` that is not a lower-case slug, a duplicate, `pages`, a `match` that is not a path pattern or a `pageSize` outside 1–50,000 is `X_CONFIG_INVALID` at boot. **With no `sections` the output is byte-for-byte what it was** (`sitemap-sections.test.ts` pins the hash). [SEO → Sections](https://github.com/developerz-ai/ultimate/wiki/SEO#sections-an-index-by-configuration). +- **`prerender()` may say when a page changed and which locales list it.** Beside a bare string and a params object, an item may be an entry: `{ params: { slug }, lastmod: row.updatedAt, locales: ['es-co'] }` (`PrerenderEntry`; the three forms mix). `lastmod` — a `Date`, or ISO 8601 with an explicit zone — is that page's sitemap ``, written as W3C Datetime in UTC and winning over `seo.sitemap.lastmod`. A time with no zone or free text is the new `X_SITEMAP_ENTRY_INVALID`, naming the route file, never a date read in the server's zone; a moment in the future is clamped to now (refusing it would fail the whole sitemap for one row from a clock a second ahead; writing it is a date a crawler stops trusting). `locales` lists the page — one `` each, hreflang among those only, `x-default` only when the default locale is one of them — in the named locales; `[]` lists it nowhere; a name the app does not route is `X_SITEMAP_ENTRY_INVALID`. A route with no params may answer one entry (`[{ params: {}, lastmod, locales }]`), which is how a page with a `load` says either. The BUILD is unchanged: every routed locale's document is still rendered, and `enumeratePrerender()` still answers params. An entry key outside the three, a `lastmod` that is neither a string nor a `Date`, or `locales` that is not a list of names is `X_PRERENDER_FAILED`. The static export still lists only the pages it emitted (`pagesFor`) and now asks `prerender()` once more for what each says of itself. +- **The scaffolded `apps/web/prerender.ts` writes the SEO files through `writeSiteSeo(out, seo)`.** Nothing to change in an existing app: a `prerender.ts` that loops over `seo.sitemaps` itself already writes an index and its parts. +- New names: `SitemapSectionConfig`, `SITEMAP_DEFAULT_SECTION`, `SITEMAP_PAGE_SIZE_MAX`, `sectionPatterns`, `sitemapSectionIssues` (`@ultimat3/core`; `SeoSitemapConfig` gains `sections` and `pageSize`); `SitemapPage`, `SitemapSection`, `expandRoutePages`, `sectionOf`, `sitemapDatetime`, `disallowMatcher`, `SITEMAP_MAX_BYTES`, `sitemapEntryInvalid`, `RouteRecord.locales`, `SitemapFile.lastmod`, and `sections` / `maxBytes` / `disallow` / `now` on `buildSitemap` (`@ultimat3/seo`); `PrerenderEntry`, `PrerenderItem` (`@ultimat3/render`) and `enumeratePrerenderEntries` (`@ultimat3/render/server`); `writeSiteSeo`, `SiteSeoSitemap` (`@ultimat3/cli`). `PrerenderFn`'s return type is wider; code that only declares a `prerender` is unaffected, code that reads one's result itself should call `enumeratePrerender`. + +### Fixed + +Tier 1 — seo. Tier 5 — cli. + +- **A page that is `noindex` in one locale is no longer listed — or cross-linked — in that locale.** The sitemap read each route's `meta` once, in no locale, and emitted one `` per routed locale: a page whose `meta` answers `robots: { index: false }` for English only was listed at `/en/…`, and every other copy's hreflang pointed at it ("alternate page with noindex"). `meta` is now read once per routed locale; the page is listed in the locales that may be indexed and its cluster names only those. As before, this reads a route with no params and no `load` — a dynamic page says it with `locales` on its `prerender()` entry. +- **A URL `seo.robots.disallow` blocks is no longer in the sitemap.** `robots.txt` and the sitemap are written from one config, and the sitemap listed URLs the other told crawlers not to fetch ("submitted URL blocked by robots.txt"). A rule is read as a crawler reads it — a prefix, `*`, a trailing `$`; a locale copy it blocks leaves its hreflang cluster. An app whose sitemap shrinks on upgrade was listing a blocked page: narrow the rule to keep it. +- **A sitemap file past 50 MB is split.** Only the 50,000-URL cap was enforced; a site with many locales (every `` carries the whole hreflang cluster) could pass the protocol's 50 MB uncompressed limit below it, which a crawler answers with "sitemap could not be read". Sections and the unsectioned split both cut on bytes now (`SITEMAP_MAX_BYTES`). ## 27.8.1 - 2026-10-11 diff --git a/framework.manifest.json b/framework.manifest.json index b5fe89306..35cacdbd1 100644 --- a/framework.manifest.json +++ b/framework.manifest.json @@ -1,6 +1,6 @@ { "version": 1, - "buildId": "e7c9d5de792d076bf2174805dbeec082841bf128e7692d3778cc31119ef57910", + "buildId": "dd1b939f88b14962605f053fc98e521bf728bdf30d7825e9d8dd651d77fffff6", "tiers": { "0": [ "core", @@ -3879,6 +3879,11 @@ "owner": "scripts", "at": "scripts/side-effects.ts" }, + { + "code": "X_SITEMAP_ENTRY_INVALID", + "owner": "seo", + "at": "packages/seo/src/errors.ts" + }, { "code": "X_SITEMAP_EXTRA_INVALID", "owner": "cli", diff --git a/packages/cli/src/app-config-load.test.ts b/packages/cli/src/app-config-load.test.ts index 4f13c54f2..d6eeb8a6b 100644 --- a/packages/cli/src/app-config-load.test.ts +++ b/packages/cli/src/app-config-load.test.ts @@ -52,7 +52,7 @@ export const config = { drain: { readinessGraceMs: 7000 }, health: { readiness: 'process' }, site: { origin: 'https://every.example' }, - seo: { robots: { disallow: ['/panel'] }, sitemap: { extra: ['/about'], lastmod: 'build' } }, + seo: { robots: { disallow: ['/panel'] }, sitemap: { extra: ['/about'], lastmod: 'build', sections: [{ name: 'blog', match: '/blog/**' }], pageSize: 1000 } }, navigation: { client: ['app'], speculation: { prefetch: 'conservative', exclude: ['/x/*'] } }, islands: { sharedChunks: true }, }; @@ -95,7 +95,12 @@ describe('loadAppConfig', () => { expect(config?.health.readiness).toBe('process'); expect(config?.site.origin).toBe('https://every.example'); expect(config?.seo.robots.disallow).toEqual(['/panel']); - expect(config?.seo.sitemap).toEqual({ extra: ['/about'], lastmod: 'build' }); + expect(config?.seo.sitemap).toEqual({ + extra: ['/about'], + lastmod: 'build', + sections: [{ name: 'blog', match: '/blog/**' }], + pageSize: 1000, + }); expect(config?.navigation.client).toEqual(['app']); expect(config?.navigation.speculation).toEqual({ prefetch: 'conservative', @@ -127,7 +132,12 @@ describe('loadAppConfig', () => { expect(config?.health.readiness).toBe('dependencies'); expect(config?.site.origin).toBeNull(); expect(config?.seo.robots.disallow).toEqual([]); - expect(config?.seo.sitemap).toEqual({ extra: [], lastmod: 'none' }); + expect(config?.seo.sitemap).toEqual({ + extra: [], + lastmod: 'none', + sections: [], + pageSize: 50_000, + }); expect(config?.navigation.client).toEqual([]); expect(config?.navigation.speculation).toEqual(DEFAULT_SPECULATION); expect(config?.islands.sharedChunks).toBe(false); diff --git a/packages/cli/src/cmd-new.test.ts b/packages/cli/src/cmd-new.test.ts index a85619aa2..8af453e0a 100644 --- a/packages/cli/src/cmd-new.test.ts +++ b/packages/cli/src/cmd-new.test.ts @@ -362,7 +362,9 @@ describe('unit · x new · the sitemap and robots.txt it wires', () => { const entry = emitted('apps/web/prerender.ts', true); expect(entry).toContain("from '@ultimat3/cli'"); expect(entry).toContain('siteSeo('); - expect(entry).toContain("'robots.txt'"); + // `writeSiteSeo` puts every file of that answer on disk: the sitemap, each part of an index + // (`/sitemaps/*.xml`) and `robots.txt` — `site-seo-sections.test.ts` holds it to the bytes. + expect(entry).toContain('writeSiteSeo(out, seo)'); expect(emitted('package.json', true)).toContain('"@ultimat3/seo"'); }); }); diff --git a/packages/cli/src/index.ts b/packages/cli/src/index.ts index 7bef34b79..dc206f03d 100644 --- a/packages/cli/src/index.ts +++ b/packages/cli/src/index.ts @@ -157,8 +157,9 @@ export { export { quoteArg } from './shell-quote'; export type { SiteSettings } from './site-config'; export { loadSiteSettings, publicOrigin } from './site-config'; -export type { SiteSeo, SiteSeoOptions } from './site-seo'; +export type { SiteSeo, SiteSeoOptions, SiteSeoSitemap } from './site-seo'; export { ROBOTS_PATH, SITEMAP_PATH, siteSeo } from './site-seo'; +export { writeSiteSeo } from './site-seo-write'; export { eachSourceFile, isGenerated, isTest, SOURCE_GLOBS } from './source-files'; export type { SkippedRoute, StaticReport } from './static-report'; export { parseStaticReport } from './static-report'; diff --git a/packages/cli/src/seo-meta.ts b/packages/cli/src/seo-meta.ts index 2557fe073..0debccdcb 100644 --- a/packages/cli/src/seo-meta.ts +++ b/packages/cli/src/seo-meta.ts @@ -5,6 +5,8 @@ // this is the only tier that can see both: `@ultimat3/seo` is tier 1 and may not import the route // registry, which is tier 4. +import { ctxOf, runWithContext } from '@ultimat3/core'; +import { localizedPath } from '@ultimat3/i18n'; import { metaContextFor, type RouteEntry, routeDataFor, routeEntries } from '@ultimat3/render'; import type { RouteRecord } from '@ultimat3/seo'; import { loadApp } from './app-load'; @@ -66,11 +68,19 @@ export async function scanSiteMeta(root: string): Promise { /** * The same scan over the registry as it stands, without loading anything. * + * `locale` runs every `meta` inside that locale's context; omitted, in none (the gate's read). + * * Split from `scanSiteMeta` so the resolution rules are testable against routes registered by hand: * the half worth pinning is which routes are reachable and what happens when one throws, and * neither of those is a fact about globbing a directory. */ -export async function readSiteMeta(): Promise { +export async function readSiteMeta(locale?: string): Promise { + // In a locale: each `meta` runs as that locale's document would — `ctx.locale`, `t` and the URL + // are its own — which is what lets a page that is `noindex` in one language say so. + const scoped = (run: () => Promise): Promise => + locale === undefined ? run() : runWithContext(ctxOf({ role: 'web', locale }), run); + const pathOf = (path: string): string => + locale === undefined ? path : localizedPath(path, locale); const records: RouteRecord[] = []; const unresolved: UnresolvedRoute[] = []; const findings: Finding[] = []; @@ -82,10 +92,10 @@ export async function readSiteMeta(): Promise { unresolved.push({ path: entry.path, file: entry.file, reason }); continue; } - const ctx = { url: `${PROBE_ORIGIN}${entry.path}`, params: {} }; + const ctx = { url: `${PROBE_ORIGIN}${pathOf(entry.path)}`, params: {} }; try { - const meta = await entry.config.meta( - metaContextFor(ctx, await routeDataFor(entry.config, ctx)), + const meta = await scoped(async () => + entry.config.meta(metaContextFor(ctx, await routeDataFor(entry.config, ctx))), ); records.push({ path: entry.path, diff --git a/packages/cli/src/seo-routes-sections.test.ts b/packages/cli/src/seo-routes-sections.test.ts new file mode 100644 index 000000000..ed86413db --- /dev/null +++ b/packages/cli/src/seo-routes-sections.test.ts @@ -0,0 +1,153 @@ +// A sectioned sitemap from a RUNNING web role: `/sitemap.xml` is the index, every child it names +// answers at `/sitemaps/-.xml` with the same content type and cache policy, and a refresh +// that fails keeps the whole previous answer — index and children together. + +import { afterEach, beforeEach, describe, expect, test } from 'bun:test'; +import { defineHttpConfig, httpServer } from '@ultimat3/http'; +import { clearRoutes, defineRoute, registerRoute } from '@ultimat3/render'; +import { seoRoutes } from './seo-routes'; +import type { SiteSettings } from './site-config'; + +const BUILD_ID = 'seo-sections-under-test'; +const ENV = { ULTIMATE_ENV: 'production', APP_URL: 'https://www.example.com/' }; +const ORIGIN = 'https://www.example.com'; +const HOUR_MS = 3_600_000; +const SITE: SiteSettings = { + origin: null, + disallow: [], + sitemap: { + extra: [], + lastmod: 'none', + pageSize: 2, + sections: [{ name: 'blog', match: '/blog/**' }], + }, +}; + +let slugs: readonly string[] | 'down' = ['a', 'b', 'c']; +let clock = 0; + +const page = (file: string, prerender?: () => readonly string[] | Promise) => + registerRoute({ + file, + component: () => 'page', + config: defineRoute({ + render: 'isr', + revalidate: { ttl: '5m' }, + hydrate: 'never', + offline: 'network-only', + ...(prerender === undefined ? {} : { prerender }), + meta: () => ({ title: 'Page', description: 'a page' }), + }), + }); + +beforeEach(() => { + clearRoutes(); + slugs = ['a', 'b', 'c']; + clock = 0; + page('apps/web/site/precios/page.tsx'); + page('apps/web/site/blog/[slug]/page.tsx', () => + // A foreign failure handed to the code under test: the driver's, not this test's verdict. + slugs === 'down' ? Promise.reject(new Error('the pool refused a connection')) : slugs, + ); +}); + +afterEach(() => { + clearRoutes(); +}); + +const server = (site: SiteSettings = SITE) => + httpServer({ + routes: seoRoutes({ env: ENV, site, now: () => clock }), + role: 'web', + config: defineHttpConfig({ dev: true, buildId: BUILD_ID, rateLimit: { scope: 'process' } }), + }); + +const locs = (xml: string) => [...xml.matchAll(/([^<]+)<\/loc>/g)].map((m) => m[1] ?? ''); +const get = (app: ReturnType, path: string) => + app.fetch(new Request(`http://10.0.0.7:3000${path}`)); + +describe('unit · the web role serves a sectioned sitemap', () => { + test('/sitemap.xml is the index; each child it names answers its own urlset', async () => { + const app = server(); + const index = await get(app, '/sitemap.xml'); + expect(index.status).toBe(200); + expect(index.headers.get('content-type')).toBe('application/xml; charset=utf-8'); + expect(index.headers.get('cache-control')).toBe('public, max-age=3600'); + const body = await index.text(); + expect(body).toContain('> = { + '/sitemaps/pages-1.xml': [`${ORIGIN}/precios`], + '/sitemaps/blog-1.xml': [`${ORIGIN}/blog/a`, `${ORIGIN}/blog/b`], + '/sitemaps/blog-2.xml': [`${ORIGIN}/blog/c`], + }; + for (const [path, urls] of Object.entries(expected)) { + const child = await get(app, path); + expect(child.status).toBe(200); + expect(child.headers.get('content-type')).toBe('application/xml; charset=utf-8'); + expect(child.headers.get('cache-control')).toBe(index.headers.get('cache-control')); + const xml = await child.text(); + expect(xml).toContain(' { + const app = server(); + for (const path of ['/sitemaps/blog-3.xml', '/sitemaps/nada-1.xml', '/sitemaps/1.xml']) { + const missing = await get(app, path); + expect(missing.status).toBe(404); + expect(missing.headers.get('cache-control')).toContain('no-store'); + } + }); + + test('robots.txt names the index and no child', async () => { + const robots = await (await get(server(), '/robots.txt')).text(); + expect(robots.match(/^Sitemap: .*$/gm)).toEqual([`Sitemap: ${ORIGIN}/sitemap.xml`]); + }); + + test('with no sections the same site is one urlset and /sitemaps/* answers nothing', async () => { + const app = server({ + ...SITE, + sitemap: { extra: [], lastmod: 'none', sections: [], pageSize: 50_000 }, + }); + const body = await (await get(app, '/sitemap.xml')).text(); + expect(body).toContain(' { + test('a failed prerender keeps the previous index AND its children, then the retry replaces both', async () => { + const app = server(); + const before = await (await get(app, '/sitemap.xml')).text(); + expect(locs(before)).toHaveLength(3); + + slugs = 'down'; + clock += HOUR_MS + 1; + const during = await get(app, '/sitemap.xml'); + expect(during.status).toBe(200); + expect(await during.text()).toBe(before); + await Bun.sleep(5); + const child = await get(app, '/sitemaps/blog-2.xml'); + expect(child.status).toBe(200); + expect(locs(await child.text())).toEqual([`${ORIGIN}/blog/c`]); + + // The read recovers with one page fewer: the second blog page is gone from index and role alike. + slugs = ['a', 'b']; + clock += 60_001; + await get(app, '/sitemap.xml'); + await Bun.sleep(5); + expect(locs(await (await get(app, '/sitemap.xml')).text())).toEqual([ + `${ORIGIN}/sitemaps/pages-1.xml`, + `${ORIGIN}/sitemaps/blog-1.xml`, + ]); + expect((await get(app, '/sitemaps/blog-2.xml')).status).toBe(404); + }); +}); diff --git a/packages/cli/src/serve-graph.test.ts b/packages/cli/src/serve-graph.test.ts index 00523ead4..ca1847474 100644 --- a/packages/cli/src/serve-graph.test.ts +++ b/packages/cli/src/serve-graph.test.ts @@ -102,8 +102,11 @@ const NEVER_AT_BOOT: readonly (readonly [string, RegExp])[] = [ * raised 641 → 642, measured 642 (2026-10-11, an isr page is one entry however its URL is spelled): * `http/src/route-path.ts` — a pathname's segments, their decoding and its one spelling, moved out * of `router.ts` so the matcher and the `isr` key share them; reached through the http barrel. + * raised 642 → 643, measured 643 (2026-10-11, sitemap sections): `core/src/config-sitemap.ts` — the + * shape of `seo.sitemap.sections` / `pageSize` and the one validator `defineConfig` and + * `@ultimat3/seo` refuse them with; split from `config-site.ts`, reached through the core barrel. */ -const MIGRATE_CEILING = 642; +const MIGRATE_CEILING = 643; /** * measured: 796 — the 558 above plus what `serve-boot.ts` adds: the services and the roles. @@ -209,8 +212,15 @@ const MIGRATE_CEILING = 642; * node when the bus answers, where a container refuses the boot). * raised 950 → 951, measured 951 (2026-10-11, an isr page is one entry however its URL is spelled): * the module named on `MIGRATE_CEILING`. + * raised 951 → 956, measured 956 (2026-10-11, sitemap sections): the module named on + * `MIGRATE_CEILING`; three in seo, which every role reaches through the render barrel — + * `sitemap-sections.ts` (which named section a page files under), `sitemap-datetime.ts` (a page's + * `lastmod` as W3C Datetime, no ambient zone) and `sitemap-xml.ts` (the urlset and index XML and + * the split on both caps, moved out of `sitemap.ts` at the file ceiling); and + * `render/src/prerender-enumerate.ts` — `prerender()`'s three item forms normalised, moved out of + * `render-static.ts` with the entry form (`{ params, lastmod, locales }`). */ -const SERVING_ROLE_CEILING = 951; +const SERVING_ROLE_CEILING = 956; /** * measured: 888 — the 796 above plus the 92 `serve-web.ts` adds (41 CLI, 36 MCP, 15 PWA). @@ -305,8 +315,10 @@ const SERVING_ROLE_CEILING = 951; * raised 1078 → 1080, measured 1080 (2026-10-11, an isr page is one entry however its URL is * spelled): the module named on `MIGRATE_CEILING`, and `cli/src/runtime-address.ts` — the URL a * `static` or `isr` document is rendered at and kept under, the address the router matched. + * raised 1080 → 1085, measured 1085 (2026-10-11, sitemap sections): the five modules named on + * `SERVING_ROLE_CEILING` — the web role reaches every module the serving roles do. */ -const WEB_ROLE_CEILING = 1080; +const WEB_ROLE_CEILING = 1085; interface MetaInput { readonly imports: readonly { readonly path: string; readonly kind: string }[]; diff --git a/packages/cli/src/site-config.test.ts b/packages/cli/src/site-config.test.ts index 15f5bdd93..3a640011c 100644 --- a/packages/cli/src/site-config.test.ts +++ b/packages/cli/src/site-config.test.ts @@ -33,18 +33,20 @@ describe('loadSiteSettings', () => { expect(await loadSiteSettings(join(ROOT, 'app'))).toEqual({ origin: 'https://notificado.co', disallow: ['/panel'], - sitemap: { extra: [], lastmod: 'none' }, + sitemap: { extra: [], lastmod: 'none', sections: [], pageSize: 50_000 }, }); }); test('reads seo.sitemap when the config carries it', async () => { await Bun.write( join(ROOT, 'sitemap', 'app.config.ts'), - "export const config = { name: 'demo', site: { origin: null }, seo: { robots: { disallow: [] }, sitemap: { extra: ['/verificar'], lastmod: 'git' } } };\n", + "export const config = { name: 'demo', site: { origin: null }, seo: { robots: { disallow: [] }, sitemap: { extra: ['/verificar'], lastmod: 'git', pageSize: 1000, sections: [{ name: 'blog', match: '/blog/**' }] } } };\n", ); expect((await loadSiteSettings(join(ROOT, 'sitemap'))).sitemap).toEqual({ extra: ['/verificar'], lastmod: 'git', + pageSize: 1000, + sections: [{ name: 'blog', match: '/blog/**' }], }); }); }); diff --git a/packages/cli/src/site-config.ts b/packages/cli/src/site-config.ts index 720eec7ed..425451c09 100644 --- a/packages/cli/src/site-config.ts +++ b/packages/cli/src/site-config.ts @@ -11,14 +11,14 @@ export interface SiteSettings { readonly origin: string | null; /** `seo.robots.disallow` — the paths a production `robots.txt` keeps crawlers out of. */ readonly disallow: readonly string[]; - /** `seo.sitemap` — public `app/` pages to list, and where `` comes from. */ + /** `seo.sitemap` — public `app/` pages to list, where `` comes from, sections, page size. */ readonly sitemap: SeoSitemapConfig; } export const NO_SITE_SETTINGS: SiteSettings = { origin: null, disallow: [], - sitemap: { extra: [], lastmod: 'none' }, + sitemap: { extra: [], lastmod: 'none', sections: [], pageSize: 50_000 }, }; const trimSlash = (origin: string): string => origin.replace(/\/+$/, ''); diff --git a/packages/cli/src/site-seo-sections.test.ts b/packages/cli/src/site-seo-sections.test.ts new file mode 100644 index 000000000..bb6125809 --- /dev/null +++ b/packages/cli/src/site-seo-sections.test.ts @@ -0,0 +1,257 @@ +// `seo.sitemap.sections` / `pageSize` and a page's own `lastmod` / `locales`, through `siteSeo` — +// the one answer the web role serves and the static export writes — and `writeSiteSeo`, which is +// how the export puts every file of it on disk. + +import { afterEach, beforeEach, describe, expect, test } from 'bun:test'; +import { rm } from 'node:fs/promises'; // why: Bun has no recursive remove, only a per-file delete. +// why: Bun exposes no path-join primitive; the fixture's files are root-relative. +import { join } from 'node:path'; +import { configureLocales, resetLocaleConfig } from '@ultimat3/i18n'; +import type { RouteConfig } from '@ultimat3/render'; +import { clearRoutes, defineRoute, registerRoute } from '@ultimat3/render'; +import { SEO_ERROR_CODES } from '@ultimat3/seo'; +import { processRoot } from './process-root-fixture'; +import { type SiteSeoOptions, siteSeo } from './site-seo'; +import { writeSiteSeo } from './site-seo-write'; +import { resetLastmodCache } from './sitemap-lastmod'; + +const route = (file: string, patch: Partial = {}): void => { + registerRoute({ + file, + config: defineRoute({ + render: 'static', + hydrate: 'never', + offline: 'network-only', + meta: () => ({ title: 'Page', description: 'a page' }), + ...patch, + } as Parameters[0]), + component: () => 'page', + }); +}; + +const BASE = 'https://notificado.co'; +const ROOT = processRoot(join(import.meta.dir, '..', '.site-seo-sections-fixture')); +const SECTIONS = [ + { name: 'blog', match: '/blog/**' }, + { name: 'ciudades', match: '/juzgados/**' }, +]; + +const declareLocales = async (): Promise => { + await Bun.write( + join(ROOT, 'packages/i18n/src/index.ts'), + "import { defineCatalogs } from '@ultimat3/i18n';\n" + + "export const catalogs = defineCatalogs({ default: 'es-co', locales: { 'es-co': {}, en: {} } });\n", + ); + configureLocales({ supported: ['es-co', 'en'], fallback: 'es-co' }); +}; + +const locs = (xml: string | undefined) => + [...(xml ?? '').matchAll(/([^<]+)<\/loc>/g)].map((m) => m[1] ?? ''); + +const seoOf = (patch: Partial = {}) => + siteSeo({ baseUrl: BASE, environment: 'production', root: ROOT, ...patch }); + +const site = (): void => { + route('apps/web/site/page.tsx'); + route('apps/web/site/precios/page.tsx'); + route('apps/web/site/blog/page.tsx'); + route('apps/web/site/blog/[slug]/page.tsx', { + render: 'isr', + revalidate: { ttl: '5m' }, + prerender: () => [ + { params: { slug: 'tres' }, lastmod: new Date('2025-10-03T00:00:00Z') }, + { params: { slug: 'uno' }, lastmod: '2025-10-01T09:00:00-05:00' }, + 'dos', + ], + }); + route('apps/web/site/juzgados/[ciudad]/page.tsx', { prerender: () => ['medellin', 'bogota'] }); +}; + +beforeEach(() => { + resetLocaleConfig(); + clearRoutes(); + resetLastmodCache(); +}); + +afterEach(async () => { + clearRoutes(); + resetLocaleConfig(); + resetLastmodCache(); + await rm(ROOT, { recursive: true, force: true }); +}); + +describe('siteSeo — sections', () => { + test('with no sections the answer is the one urlset it always was', async () => { + site(); + const seo = await seoOf({ sitemap: { extra: [], lastmod: 'none' } }); + expect(seo.sitemaps.map((file) => file.path)).toEqual(['/sitemap.xml']); + expect(seo.sitemaps[0]?.xml).toContain(' { + site(); + const seo = await seoOf({ + sitemap: { extra: [], lastmod: 'none', sections: SECTIONS, pageSize: 2 }, + }); + expect(seo.sitemaps.map((file) => file.path)).toEqual([ + '/sitemap.xml', + '/sitemaps/pages-1.xml', + '/sitemaps/blog-1.xml', + '/sitemaps/blog-2.xml', + '/sitemaps/ciudades-1.xml', + ]); + const [index, pages, blog1, blog2, ciudades] = seo.sitemaps; + expect(index?.xml).toContain(' `${BASE}${file.path}`)); + expect(locs(pages?.xml)).toEqual([BASE, `${BASE}/precios`]); + expect(locs(blog1?.xml)).toEqual([`${BASE}/blog`, `${BASE}/blog/dos`]); + expect(locs(blog2?.xml)).toEqual([`${BASE}/blog/tres`, `${BASE}/blog/uno`]); + expect(locs(ciudades?.xml)).toEqual([`${BASE}/juzgados/bogota`, `${BASE}/juzgados/medellin`]); + // The index says of each child the newest date inside it, and nothing of one with no dates. + expect(index?.xml).toContain( + `${BASE}/sitemaps/blog-2.xml\n 2025-10-03T00:00:00.000Z`, + ); + expect(index?.xml.match(//g)).toHaveLength(1); + // robots.txt names the index alone: a crawler finds the children through it. + expect(seo.robots.match(/^Sitemap: .*$/gm)).toEqual([`Sitemap: ${BASE}/sitemap.xml`]); + }); + + test('a page`s own lastmod is written; a page with none takes the route file`s', async () => { + site(); + const xml = (await seoOf()).sitemaps[0]?.xml ?? ''; + expect(xml).toContain( + `${BASE}/blog/uno\n 2025-10-01T14:00:00.000Z`, + ); + expect(xml).toContain(`${BASE}/blog/dos\n `); + const built = + (await seoOf({ sitemap: { extra: [], lastmod: 'build' } })).sitemaps[0]?.xml ?? ''; + expect(built).toContain(`${BASE}/blog/tres\n 2025-10-03T00:00:00.000Z`); + expect(built).toMatch(new RegExp(`${BASE}/blog/dos\\n `)); + }); + + test('a lastmod that is no date fails the enumeration, typed and naming the route file', async () => { + route('apps/web/site/blog/[slug]/page.tsx', { + prerender: () => [{ params: { slug: 'a' }, lastmod: 'ayer' }], + }); + const error = await seoOf().then( + () => undefined, + (thrown: unknown) => thrown as { code?: string; cause?: string }, + ); + expect(error?.code).toBe(SEO_ERROR_CODES.sitemapEntryInvalid); + expect(error?.cause).toContain('apps/web/site/blog/[slug]/page.tsx'); + }); +}); + +describe('siteSeo — the locales a page is listed in', () => { + test('an entry`s `locales` lists it — and cross-links it — in those locales only', async () => { + await declareLocales(); + route('apps/web/site/blog/[slug]/page.tsx', { + prerender: () => [{ params: { slug: 'solo' }, locales: ['es-co'] }, 'ambos'], + }); + const xml = (await seoOf()).sitemaps[0]?.xml ?? ''; + expect(locs(xml)).toEqual([`${BASE}/blog/solo`, `${BASE}/blog/ambos`, `${BASE}/en/blog/ambos`]); + expect(xml).not.toContain('/en/blog/solo'); + expect(xml).toContain(`hreflang="en" href="${BASE}/en/blog/ambos"`); + }); + + // The page asks crawlers to stay away IN ENGLISH: its Spanish copy is listed, alone, and no + // hreflang points a crawler at the copy that says noindex. + test('a page whose meta is noindex in one locale is listed in the others only', async () => { + await declareLocales(); + route('apps/web/site/precios/page.tsx'); + route('apps/web/site/terminos/page.tsx', { + meta: async (ctx) => ({ + title: 'Términos', + description: 'los términos', + ...(ctx.locale === 'en' ? { robots: { index: false } } : {}), + }), + }); + route('apps/web/site/gracias/page.tsx', { + meta: async () => ({ title: 'Gracias', description: 'gracias', robots: { index: false } }), + }); + const xml = (await seoOf()).sitemaps[0]?.xml ?? ''; + expect(locs(xml)).toEqual([`${BASE}/precios`, `${BASE}/en/precios`, `${BASE}/terminos`]); + const terminos = xml.slice(xml.indexOf('/terminos')); + expect(terminos).not.toContain('xhtml:link'); + }); + + test('the static export`s pages carry what prerender() says of each; the LIST stays the report`s', async () => { + await declareLocales(); + route('apps/web/site/blog/[slug]/page.tsx', { + prerender: () => [ + { params: { slug: 'uno' }, lastmod: '2025-10-01T00:00:00Z', locales: ['es-co'] }, + { params: { slug: 'no-emitida' }, lastmod: '2025-10-02T00:00:00Z' }, + ], + }); + const seo = await seoOf({ pagesFor: () => ['/blog/uno', '/en/blog/uno', '/blog/otra'] }); + const xml = seo.sitemaps[0]?.xml ?? ''; + expect(locs(xml)).toEqual([`${BASE}/blog/uno`, `${BASE}/blog/otra`, `${BASE}/en/blog/otra`]); + expect(xml).toContain(`${BASE}/blog/uno\n 2025-10-01T00:00:00.000Z`); + }); +}); + +describe('siteSeo — a route with no params and a `load`', () => { + // Its `meta` needs a request, so the sitemap cannot read it; its `prerender()` entry can still + // say when the page changed and where it is listed. + const index = (patch: Partial): void => + route('apps/web/site/blog/page.tsx', { + render: 'isr', + revalidate: { ttl: '5m' }, + load: () => Promise.resolve({ posts: 3 }), + ...patch, + } as Partial); + + test('its prerender() entry carries the page`s lastmod and locales', async () => { + await declareLocales(); + index({ + prerender: () => [{ params: {}, lastmod: '2025-10-04T00:00:00Z', locales: ['es-co'] }], + }); + const xml = (await seoOf()).sitemaps[0]?.xml ?? ''; + expect(locs(xml)).toEqual([`${BASE}/blog`]); + expect(xml).toContain('2025-10-04T00:00:00.000Z'); + }); + + test('a static export lists it whether or not the build emitted it: only a DYNAMIC route`s list is the report`s', async () => { + index({ prerender: () => [{ params: {}, lastmod: '2025-10-04T00:00:00Z' }] }); + const seo = await seoOf({ pagesFor: () => [] }); + expect(locs(seo.sitemaps[0]?.xml)).toEqual([`${BASE}/blog`]); + }); +}); + +describe('siteSeo — robots and the sitemap agree', () => { + test('a URL seo.robots.disallow keeps crawlers out of is not listed', async () => { + route('apps/web/site/precios/page.tsx'); + route('apps/web/site/interno/page.tsx'); + route('apps/web/site/interno/[id]/page.tsx', { prerender: () => ['a'] }); + const seo = await seoOf({ disallow: ['/interno'] }); + expect(locs(seo.sitemaps[0]?.xml)).toEqual([`${BASE}/precios`]); + expect(seo.robots).toContain('Disallow: /interno'); + }); +}); + +describe('writeSiteSeo — the static export', () => { + test('writes the index, every child it names and robots.txt — the bytes the web role serves', async () => { + site(); + const seo = await seoOf({ + sitemap: { extra: [], lastmod: 'none', sections: SECTIONS, pageSize: 2 }, + }); + const out = join(ROOT, 'out'); + const written = await writeSiteSeo(out, seo); + expect(written).toEqual([ + '/sitemap.xml', + '/sitemaps/pages-1.xml', + '/sitemaps/blog-1.xml', + '/sitemaps/blog-2.xml', + '/sitemaps/ciudades-1.xml', + '/robots.txt', + ]); + for (const file of seo.sitemaps) { + expect(await Bun.file(join(out, file.path)).text()).toBe(file.xml); + } + expect(await Bun.file(join(out, 'robots.txt')).text()).toBe(seo.robots); + // Every child the index names is a file on disk. + for (const loc of locs(await Bun.file(join(out, 'sitemap.xml')).text())) { + expect(await Bun.file(join(out, new URL(loc).pathname)).exists()).toBe(true); + } + }); +}); diff --git a/packages/cli/src/site-seo-write.ts b/packages/cli/src/site-seo-write.ts new file mode 100644 index 000000000..99432da6b --- /dev/null +++ b/packages/cli/src/site-seo-write.ts @@ -0,0 +1,16 @@ +// Single responsibility: `siteSeo()`'s answer as files in a static export — `/sitemap.xml`, every +// part an index names, and `robots.txt` — so a CDN serves the bytes a running web role answers. + +// why: Bun exposes no path-join primitive, and every file path is relative to the output root. +import { join } from 'node:path'; +import { ROBOTS_PATH, type SiteSeo } from './site-seo'; + +/** + * Writes every file of `seo` under `out` and answers their site-relative paths, the sitemap files + * first. `Bun.write` creates `sitemaps/` on the way, so an index and its parts land together. + */ +export async function writeSiteSeo(out: string, seo: SiteSeo): Promise { + for (const file of seo.sitemaps) await Bun.write(join(out, file.path), file.xml); + await Bun.write(join(out, ROBOTS_PATH), seo.robots); + return [...seo.sitemaps.map((file) => file.path), ROBOTS_PATH]; +} diff --git a/packages/cli/src/site-seo.ts b/packages/cli/src/site-seo.ts index 477e5b0d1..fd0fd2a7e 100644 --- a/packages/cli/src/site-seo.ts +++ b/packages/cli/src/site-seo.ts @@ -8,8 +8,8 @@ import { localizedPath, unlocalizedPath } from '@ultimat3/i18n'; import { appLocaleSet } from '@ultimat3/i18n/app-catalogs'; import type { RouteEntry } from '@ultimat3/render'; import { routeEntries } from '@ultimat3/render'; -import { enumeratePrerender, fillPath } from '@ultimat3/render/server'; -import type { RouteRecord, SitemapFile } from '@ultimat3/seo'; +import { enumeratePrerenderEntries, fillPath } from '@ultimat3/render/server'; +import type { RouteRecord, SitemapFile, SitemapPage } from '@ultimat3/seo'; import { buildRobots, buildSitemap, isDynamic } from '@ultimat3/seo'; import { readSiteMeta } from './seo-meta'; import { lastmodOf } from './sitemap-lastmod'; @@ -25,7 +25,8 @@ export interface SiteSeoOptions { /** * The concrete pages a build EMITTED for a dynamic route. The static export passes its report's, * so the sitemap cannot name a page the artifact lacks; absent, `prerender()` is asked — which is - * the same list, enumerated by the same function the prerenderer calls. + * the same list, enumerated by the same function the prerenderer calls. Given, `prerender()` is + * still asked what each page says of ITSELF (`lastmod`, `locales`); the list stays this one. */ readonly pagesFor?: ((routePath: string) => readonly string[]) | undefined; /** @@ -35,14 +36,22 @@ export interface SiteSeoOptions { readonly disallow?: readonly string[] | undefined; /** * `seo.sitemap` from `app.config.ts` (`loadSiteSettings`): the public `app/` pages to list beside - * the `site/` ones, and where each `` comes from. Omitted: `site/` pages, no ``. + * the `site/` ones, where each `` comes from, and the sections and page size of an + * index. Omitted: `site/` pages, no ``, one urlset. */ - readonly sitemap?: SeoSitemapConfig | undefined; + readonly sitemap?: SiteSeoSitemap | undefined; /** The app root the route files are relative to — read for `lastmod: 'git' | 'mtime'`. */ readonly root?: string | undefined; } -const NO_SITEMAP_SETTINGS: SeoSitemapConfig = { extra: [], lastmod: 'none' }; +/** + * `seo.sitemap` as `siteSeo` takes it: `sections` and `pageSize` optional, so a caller that wrote + * `{ extra, lastmod }` before either existed still compiles and still gets the one urlset. + */ +export type SiteSeoSitemap = Pick & + Partial>; + +const NO_SITEMAP_SETTINGS: SiteSeoSitemap = { extra: [], lastmod: 'none' }; /** A `seo.sitemap.extra` path that names no public page outside `site/`. */ export class SitemapExtraInvalidError extends UltimateError { @@ -67,42 +76,87 @@ const isPublicSite = (entry: RouteEntry): boolean => entry.surface === 'site' && entry.config.policy === undefined; /** - * A dynamic route's pages, UNPREFIXED and once each. A static export's report lists every locale's - * copy (`/blog/a` and `/en/blog/a`), and the sitemap localizes each page itself — reading the - * prefixed copies back as pages of their own listed `/en/en/blog/a`. + * A route's pages, UNPREFIXED and once each, with what `prerender()` said of each one. A + * static export's report lists every locale's copy (`/blog/a` and `/en/blog/a`), and the sitemap + * localizes each page itself — reading the prefixed copies back as pages of their own listed + * `/en/en/blog/a`. */ -const pagesOf = async (entry: RouteEntry, options: SiteSeoOptions): Promise => { - if (options.pagesFor !== undefined) { - return [...new Set(options.pagesFor(entry.path).map((path) => unlocalizedPath(path)))]; - } - const params = await enumeratePrerender(entry); - return params.map((set) => fillPath(entry.pattern.source, set)); +const pagesOf = async ( + entry: RouteEntry, + options: SiteSeoOptions, +): Promise => { + const said = (await enumeratePrerenderEntries(entry)).map( + (page): SitemapPage => ({ + path: fillPath(entry.pattern.source, page.params), + ...(page.lastmod === undefined ? {} : { lastmod: page.lastmod }), + ...(page.locales === undefined ? {} : { locales: page.locales }), + }), + ); + // Only a DYNAMIC route's list is the report's: a page with no params is listed whether or not + // this build emitted it (an `isr` one is rendered by the server), exactly as before. + if (options.pagesFor === undefined || !isDynamic(entry.path)) return said; + const byPath = new Map(said.map((page) => [page.path, page])); + const emitted = new Set(options.pagesFor(entry.path).map((path) => unlocalizedPath(path))); + return [...emitted].map((path) => byPath.get(path) ?? { path }); }; +/** + * What a page's own `meta` says about being indexed, per locale: the record's `meta` when every + * locale agrees, its `locales` when only some may be indexed. `robots: { index: false }` in ONE + * locale used to leave that copy listed and every other copy's hreflang pointing at it — the read + * was made once, in no locale at all. + */ +async function indexability( + locales: readonly string[], +): Promise<(path: string) => Pick> { + if (locales.length < 2) { + const metaByPath = new Map((await readSiteMeta()).records.map((r) => [r.path, r.meta])); + return (path) => { + const meta = metaByPath.get(path); + return meta === undefined ? {} : { meta }; + }; + } + const scans = new Map>(); + // One locale at a time: each scan runs every route's `meta` inside that locale's context. + for (const locale of locales) { + scans.set(locale, new Map((await readSiteMeta(locale)).records.map((r) => [r.path, r.meta]))); + } + return (path) => { + const listed = locales.filter( + (locale) => scans.get(locale)?.get(path)?.robots?.index !== false, + ); + if (listed.length === locales.length) return {}; + // No locale may be indexed: the `noindex` itself is what keeps the route out. + if (listed.length === 0) return { meta: { robots: { index: false } } }; + return { locales: listed }; + }; +} + /** * The public `site/` routes as `@ultimat3/seo` reads them. `meta` is carried where it resolves * without a request (`readSiteMeta`), which is what lets a page's own `robots: { index: false }` * keep it out of the sitemap — a page that asks crawlers to stay away must not be listed for them. + * A route with params or a `load` resolves no `meta` here: its pages say it per entry (`locales`). */ async function publicSiteRoutes( options: SiteSeoOptions, stamp: Stamp, + locales: readonly string[], ): Promise { - const metaByPath = new Map((await readSiteMeta()).records.map((r) => [r.path, r.meta])); + const indexed = await indexability(locales); return routeEntries() .filter(isPublicSite) - .map((entry) => { - const meta = metaByPath.get(entry.path); - return { - path: entry.path, - file: entry.file, - surface: 'site' as const, - render: entry.config.render, - ...(meta === undefined ? {} : { meta }), - ...(isDynamic(entry.path) ? { prerender: () => pagesOf(entry, options) } : {}), - ...stamp(entry.file), - }; - }); + .map((entry) => ({ + path: entry.path, + file: entry.file, + surface: 'site' as const, + render: entry.config.render, + ...indexed(entry.path), + ...(isDynamic(entry.path) || entry.config.prerender !== undefined + ? { prerender: () => pagesOf(entry, options) } + : {}), + ...stamp(entry.file), + })); } type Stamp = (file: string) => { readonly lastmod?: string }; @@ -158,11 +212,15 @@ export async function siteSeo(options: SiteSeoOptions): Promise { return lastmod === undefined ? {} : { lastmod }; }; const routes = [ - ...(await publicSiteRoutes(options, stamp)), + ...(await publicSiteRoutes(options, stamp, locales)), ...extraRoutes(settings.extra, stamp), ]; const sitemap = await buildSitemap(routes, { baseUrl: options.baseUrl, + // A URL `robots.txt` keeps crawlers out of is not handed to them here: one config, two files. + ...(options.disallow === undefined ? {} : { disallow: options.disallow }), + ...(settings.sections === undefined ? {} : { sections: settings.sections }), + ...(settings.pageSize === undefined ? {} : { maxUrls: settings.pageSize }), ...(locales.length < 2 ? {} : { @@ -172,8 +230,8 @@ export async function siteSeo(options: SiteSeoOptions): Promise { localizedPath(path, locale, defaultLocale), }), }); - // Past 50,000 URLs `files` are `/sitemaps/.xml` and the index is `/sitemap.xml`; below it, - // `files` is that one file and there is no index. + // With sections, or past a protocol cap, `files` are `/sitemaps/*.xml` and the index is + // `/sitemap.xml`; otherwise `files` is that one file and there is no index. const sitemaps = sitemap.index === undefined ? sitemap.files : [sitemap.index, ...sitemap.files]; const robots = buildRobots({ baseUrl: options.baseUrl, diff --git a/packages/cli/src/templates/scaffold-entries.ts b/packages/cli/src/templates/scaffold-entries.ts index a711ef20e..287f3cbe7 100644 --- a/packages/cli/src/templates/scaffold-entries.ts +++ b/packages/cli/src/templates/scaffold-entries.ts @@ -66,7 +66,13 @@ const prerender = // landed on disk — which is what \`x build --target static --json\` reads back. import { join } from 'node:path'; -import { loadSiteSettings, type PrerenderReport, prerenderSite, siteSeo } from '@ultimat3/cli'; +import { + loadSiteSettings, + type PrerenderReport, + prerenderSite, + siteSeo, + writeSiteSeo, +} from '@ultimat3/cli'; const root = join(import.meta.dir, '..', '..'); const flag = Bun.argv.indexOf('--out'); @@ -84,9 +90,12 @@ const origin = Bun.env.SITE_ORIGIN; * page whose \`meta\` says \`robots: { index: false }\` left out — so a crawler reads one sitemap * from the CDN and from the container. * - * A DYNAMIC route contributes exactly the URLs this build emitted for it, read back off the report - * rather than by calling \`prerender()\` a second time: the sitemap then cannot name a page the - * artifact does not contain. + * A DYNAMIC route contributes exactly the URLs this build emitted for it, read back off the report: + * the sitemap then cannot name a page the artifact does not contain. \`prerender()\` is asked again + * only for what each page says of itself — its \`lastmod\`, the \`locales\` it is listed in. + * + * With \`seo.sitemap.sections\` the answer is an index and its parts: \`/sitemap.xml\` plus + * \`/sitemaps/-.xml\`, every one written here. * * \`robots.txt\` fails closed — anything that is not \`ULTIMATE_ENV=production\` emits * \`Disallow: /\` and advertises no sitemap — so a preview build cannot outrank the real site. @@ -104,9 +113,7 @@ async function writeSeoFiles(report: PrerenderReport): Promise report.pages.filter((page) => page.route === route).map((page) => page.path), }); - for (const file of seo.sitemaps) await Bun.write(join(out, file.path), file.xml); - await Bun.write(join(out, 'robots.txt'), seo.robots); - return [...seo.sitemaps.map((file) => file.path), '/robots.txt']; + return await writeSiteSeo(out, seo); } if (import.meta.main) { diff --git a/packages/core/src/config-site.test.ts b/packages/core/src/config-site.test.ts index 09e0041de..6d9111114 100644 --- a/packages/core/src/config-site.test.ts +++ b/packages/core/src/config-site.test.ts @@ -51,13 +51,94 @@ describe('seo.sitemap — refusals', () => { }); }); +describe('seo.sitemap.sections and pageSize — refusals', () => { + const sections = (value: unknown, pageSize?: unknown): string => + causeOf(() => + defineConfig({ + name: 'app', + seo: { + sitemap: { + sections: value as [], + ...(pageSize === undefined ? {} : { pageSize: pageSize as number }), + }, + }, + }), + ); + + test('a name that is not a lower-case slug is refused: it becomes a file name', () => { + for (const name of ['Blog', 'blog posts', '../x', 'blog_', '-blog', 'a--b', '', 5]) { + expect(sections([{ name, match: '/blog/**' }])).toContain('lower-case slug'); + } + }); + + test('the default section`s name is reserved, and a name may be declared once', () => { + expect(sections([{ name: 'pages', match: '/a' }])).toContain('reserved'); + expect( + sections([ + { name: 'blog', match: '/blog/**' }, + { name: 'blog', match: '/notas/**' }, + ]), + ).toContain('twice'); + }); + + test('a match that is not a path pattern is refused, naming the section', () => { + for (const match of ['blog/**', '/blog?x=1', '//x.co/a', '/a/**/b', '/blog/p*', [], 7]) { + expect(sections([{ name: 'blog', match }])).toContain('seo.sitemap.sections "blog" match'); + } + expect(sections('blog')).toContain('must be a list'); + expect(sections([null])).toContain('must be { name, match }'); + }); + + test('pageSize is a whole number within the protocol cap', () => { + for (const pageSize of [0, -1, 2.5, 50_001, '1000', Number.NaN]) { + expect(sections([], pageSize)).toContain('seo.sitemap.pageSize'); + } + }); + + test('the refusal is the typed boot error, with a runnable fix', () => { + let thrown: unknown; + try { + defineConfig({ name: 'app', seo: { sitemap: { sections: [{ name: 'B', match: '/b' }] } } }); + } catch (error) { + thrown = error; + } + expect((thrown as { code?: string }).code).toBe('X_CONFIG_INVALID'); + expect((thrown as { fix?: string }).fix).toMatch(/x verify/); + }); + + test('sections and pageSize are accepted in every documented form', () => { + const config = defineConfig({ + name: 'app', + seo: { + sitemap: { + pageSize: 1000, + sections: [ + { name: 'blog', match: '/blog/**' }, + { name: 'ciudades', match: ['/juzgados/**', '/ciudades/*'] }, + ], + }, + }, + }); + expect(config.seo.sitemap.pageSize).toBe(1000); + expect(config.seo.sitemap.sections.map((section) => section.name)).toEqual([ + 'blog', + 'ciudades', + ]); + }); +}); + describe('site and seo', () => { test('seo.sitemap merges beside robots, key by key', () => { const config = defineConfig( { name: 'app', seo: { sitemap: { extra: ['/verificar'] } } }, { seo: { sitemap: { lastmod: 'git' } } }, ); - expect(config.seo.sitemap).toEqual({ extra: ['/verificar'], lastmod: 'git' }); + expect(config.seo.sitemap).toEqual({ + extra: ['/verificar'], + lastmod: 'git', + sections: [], + pageSize: 50_000, + }); expect(config.seo.robots.disallow).toEqual([]); }); @@ -66,7 +147,7 @@ describe('site and seo', () => { expect(config.site).toEqual({ origin: null }); expect(config.seo).toEqual({ robots: { disallow: [] }, - sitemap: { extra: [], lastmod: 'none' }, + sitemap: { extra: [], lastmod: 'none', sections: [], pageSize: 50_000 }, }); }); diff --git a/packages/core/src/config-site.ts b/packages/core/src/config-site.ts index 0e2d7a0f2..0a7d68598 100644 --- a/packages/core/src/config-site.ts +++ b/packages/core/src/config-site.ts @@ -3,6 +3,11 @@ // Split from `config.ts` for `config-pwa.ts`' reason: that file sits at its 500-line ceiling. import { type Input, layered } from './config-merge'; +import { + SITEMAP_PAGE_SIZE_MAX, + type SitemapSectionConfig, + sitemapSectionIssues, +} from './config-sitemap'; import { describeValue } from './error-render'; export interface SiteConfig { @@ -39,6 +44,14 @@ export interface SeoSitemapConfig { */ readonly extra: readonly string[]; readonly lastmod: SitemapLastmod; + /** + * Named parts of a sitemap index. Empty (the default): one `/sitemap.xml` urlset, split only past + * a cap. Any section declared: `/sitemap.xml` is a `` of `/sitemaps/-.xml`, + * the pages no section matches under `pages`. + */ + readonly sections: readonly SitemapSectionConfig[]; + /** `` entries per sitemap file. Default and ceiling: 50,000, the sitemaps.org cap. */ + readonly pageSize: number; } export interface SeoConfig { @@ -75,7 +88,7 @@ export function mergeSite(layers: readonly SiteSectionsInput[]): SiteSections { layers.map((layer) => layer.seo?.robots), ), sitemap: layered( - { extra: [], lastmod: 'none' }, + { extra: [], lastmod: 'none', sections: [], pageSize: SITEMAP_PAGE_SIZE_MAX }, layers.map((layer) => layer.seo?.sitemap), ), }, @@ -136,4 +149,5 @@ export function siteIssues(config: SiteSections, issues: string[]): void { `seo.sitemap.lastmod "${String(config.seo.sitemap.lastmod)}" must be one of ${SITEMAP_LASTMOD_SOURCES.join(', ')}`, ); } + issues.push(...sitemapSectionIssues(config.seo.sitemap.sections, config.seo.sitemap.pageSize)); } diff --git a/packages/core/src/config-sitemap.ts b/packages/core/src/config-sitemap.ts new file mode 100644 index 000000000..b0ad22d2f --- /dev/null +++ b/packages/core/src/config-sitemap.ts @@ -0,0 +1,118 @@ +// Single responsibility: `seo.sitemap.sections` and `seo.sitemap.pageSize` — the shape of a named +// sitemap section and the one validator both `defineConfig` and `@ultimat3/seo` refuse it with. +// Split from `config-site.ts`: that file owns the origin and the crawler rules. + +import { describeValue } from './error-render'; + +/** + * One named part of a sitemap index. Every listed page whose UNPREFIXED path matches goes to + * `/sitemaps/-.xml`; the first section that matches wins, in declaration order. + */ +export interface SitemapSectionConfig { + /** Lower-case slug — it is a file name: `blog` → `/sitemaps/blog-1.xml`. */ + readonly name: string; + /** + * A path pattern, or several: `/precios` (that page), `/blog/*` (one segment below), `/blog/**` + * (`/blog` and everything below it). Matched against the page's path without its locale prefix, + * so `/en/blog/a` files with `/blog/a`. + */ + readonly match: string | readonly string[]; +} + +/** Where a page no section matches goes. Reserved: an app's section may not take the name. */ +export const SITEMAP_DEFAULT_SECTION = 'pages'; + +/** The sitemaps.org cap on `` entries in one file — the default and the ceiling of `pageSize`. */ +export const SITEMAP_PAGE_SIZE_MAX = 50_000; + +const SECTION_NAME = /^[a-z0-9]+(?:-[a-z0-9]+)*$/; +const SECTION_NAME_MAX = 64; + +/** The patterns of one section as a list, whichever form it was written in. */ +export function sectionPatterns(section: SitemapSectionConfig): readonly string[] { + return typeof section.match === 'string' ? [section.match] : section.match; +} + +function patternIssue(pattern: string): string | undefined { + if (!pattern.startsWith('/') || pattern.startsWith('//') || /[?#]/.test(pattern)) { + return 'must be a path starting with / — no origin, query or fragment'; + } + const segments = pattern.split('/').slice(1); + for (const [index, segment] of segments.entries()) { + if (segment === '**' && index !== segments.length - 1) { + return 'may use ** only as its last segment'; + } + if (segment !== '*' && segment !== '**' && segment.includes('*')) { + return 'may use * only as a whole segment'; + } + } + return undefined; +} + +/** + * Every refusal `sections` and `pageSize` earn, as sentences. `unknown` in: an untyped config + * reaches here with whatever it wrote, and a validator that throws natively names nothing. + */ +export function sitemapSectionIssues(sections: unknown, pageSize: unknown): readonly string[] { + const issues: string[] = []; + if ( + typeof pageSize !== 'number' || + !Number.isSafeInteger(pageSize) || + pageSize < 1 || + pageSize > SITEMAP_PAGE_SIZE_MAX + ) { + issues.push( + `seo.sitemap.pageSize must be a whole number from 1 to ${String(SITEMAP_PAGE_SIZE_MAX)} (the sitemaps.org cap), not ${describeValue(pageSize)}`, + ); + } + if (!Array.isArray(sections)) { + issues.push( + `seo.sitemap.sections must be a list of { name, match }, not ${describeValue(sections)}`, + ); + return issues; + } + const seen = new Set(); + for (const section of sections as readonly unknown[]) { + if (typeof section !== 'object' || section === null || Array.isArray(section)) { + issues.push( + `seo.sitemap.sections entry must be { name, match }, not ${describeValue(section)}`, + ); + continue; + } + const { name, match } = section as { readonly name?: unknown; readonly match?: unknown }; + if (typeof name !== 'string' || !SECTION_NAME.test(name) || name.length > SECTION_NAME_MAX) { + issues.push( + `seo.sitemap.sections name ${describeValue(name)} must be a lower-case slug (a-z, 0-9, single hyphens, at most ${String(SECTION_NAME_MAX)} characters) — it becomes the file name /sitemaps/-.xml`, + ); + } else if (name === SITEMAP_DEFAULT_SECTION) { + issues.push( + `seo.sitemap.sections name "${name}" is reserved for the pages no section matches`, + ); + } else if (seen.has(name)) { + issues.push(`seo.sitemap.sections names "${name}" twice — two sections would write one file`); + } else { + seen.add(name); + } + const label = typeof name === 'string' ? `"${name}"` : 'entry'; + const patterns = typeof match === 'string' ? [match] : match; + if (!Array.isArray(patterns) || patterns.length === 0) { + issues.push( + `seo.sitemap.sections ${label} match must be a path pattern or a non-empty list of them, not ${describeValue(match)}`, + ); + continue; + } + for (const pattern of patterns as readonly unknown[]) { + if (typeof pattern !== 'string') { + issues.push( + `seo.sitemap.sections ${label} match entry must be a path pattern string, not ${describeValue(pattern)}`, + ); + continue; + } + const issue = patternIssue(pattern); + if (issue !== undefined) { + issues.push(`seo.sitemap.sections ${label} match "${pattern}" ${issue}`); + } + } + } + return issues; +} diff --git a/packages/core/src/index.ts b/packages/core/src/index.ts index 2b0d9ffd8..c685c1423 100644 --- a/packages/core/src/index.ts +++ b/packages/core/src/index.ts @@ -195,6 +195,13 @@ export type { SiteSectionsInput, } from './config-site'; export { SITEMAP_LASTMOD_SOURCES } from './config-site'; +export type { SitemapSectionConfig } from './config-sitemap'; +export { + SITEMAP_DEFAULT_SECTION, + SITEMAP_PAGE_SIZE_MAX, + sectionPatterns, + sitemapSectionIssues, +} from './config-sitemap'; export type { ConflictPolicy, ResolveConflictOptions, Row } from './conflict-policy'; export { resolveConflict } from './conflict-policy'; export type { Ctx, CtxFacts, CtxInit, CtxPatch, CtxServices, ServiceBag } from './context'; diff --git a/packages/http/src/error-map-tier-1.ts b/packages/http/src/error-map-tier-1.ts index a553605ea..da51927a4 100644 --- a/packages/http/src/error-map-tier-1.ts +++ b/packages/http/src/error-map-tier-1.ts @@ -80,4 +80,6 @@ export const TIER_1_ERROR_STATUS = { // 409 with the rest of promotion's refusals: an earlier call moved these bytes (a doubled or // retried confirm), a state the caller answers by reading the row that holds `meta.attachedKey`. X_STORAGE_ALREADY_PROMOTED: 409, + // @ultimat3/seo — a sitemap entry carries a lastmod or a locale the sitemap cannot write + X_SITEMAP_ENTRY_INVALID: 500, } satisfies Readonly>; diff --git a/packages/render/README.md b/packages/render/README.md index 568cf4542..49647c27e 100644 --- a/packages/render/README.md +++ b/packages/render/README.md @@ -26,6 +26,14 @@ export const config = defineRoute({ export function Page(props: { data: Post }) { /* the SAME object meta was given */ } ``` +`prerender()` answers the pages to build: a bare string (a one-param route), a params object, or an +entry — `{ params, lastmod?, locales? }` (`PrerenderEntry`) — mixed freely. The build reads `params` +alone and renders every routed locale; `lastmod` (a `Date` or ISO 8601 with a zone) and `locales` +are what the page says of itself for the sitemap: its ``, and the locales it is listed and +cross-linked in. `enumeratePrerender()` answers params, `enumeratePrerenderEntries()` the entries. +A key outside the three, a `lastmod` that is neither a string nor a `Date`, or `locales` that is +not a list of names is `X_PRERENDER_FAILED`, naming the route file. + ## `load` is the one server-side data seam Optional, and the only way a page gets data. It runs **once per render** and the result is handed diff --git a/packages/render/src/index.ts b/packages/render/src/index.ts index 1382eef0d..5562ddf8c 100644 --- a/packages/render/src/index.ts +++ b/packages/render/src/index.ts @@ -182,7 +182,9 @@ export { export type { InvalidateMode, LoadRequirement, + PrerenderEntry, PrerenderFn, + PrerenderItem, RenderResult, RevalidateConfig, RouteAlternate, diff --git a/packages/render/src/prerender-enumerate.test.ts b/packages/render/src/prerender-enumerate.test.ts new file mode 100644 index 000000000..441f59fbf --- /dev/null +++ b/packages/render/src/prerender-enumerate.test.ts @@ -0,0 +1,107 @@ +// `prerender()` answers pages in three forms — a bare string, a params object, and an entry that +// also says when the page changed and which locales list it. All three reach a build as params; +// only the last reaches a sitemap with more than a path. + +import { beforeEach, describe, expect, test } from 'bun:test'; +import { PrerenderFailedError } from './errors'; +import { enumeratePrerender, enumeratePrerenderEntries } from './prerender-enumerate'; +import { clearRoutes, registerRoute } from './registry'; +import type { PrerenderFn, RouteConfig, RouteMetaFn } from './route'; +import { defineRoute } from './route'; + +const meta: RouteMetaFn = () => ({ title: 'T', description: 'd'.repeat(60) }); + +function config(prerender?: PrerenderFn): RouteConfig { + return defineRoute({ + render: 'static', + offline: 'precache', + hydrate: 'never', + meta, + ...(prerender ? { prerender } : {}), + }); +} + +const blog = (prerender?: PrerenderFn) => + registerRoute({ file: 'apps/web/site/blog/[slug]/page.tsx', config: config(prerender) }); + +const refusal = async (prerender: unknown): Promise => { + const entry = blog(prerender as PrerenderFn); + try { + await enumeratePrerenderEntries(entry); + } catch (error) { + return error as PrerenderFailedError; + } + return expect.unreachable('the entry was accepted'); +}; + +beforeEach(() => { + clearRoutes(); +}); + +describe('enumeratePrerenderEntries', () => { + test('the three forms are one list, in the order prerender() answered', async () => { + const changed = new Date('2026-10-01T15:04:05.000Z'); + const entry = blog(async () => [ + 'a', + { slug: 'b' }, + { params: { slug: 'c' }, lastmod: changed, locales: ['es-co'] }, + { params: { slug: 'd' }, lastmod: '2026-10-02T00:00:00Z' }, + { params: { slug: 'e' } }, + ]); + expect(await enumeratePrerenderEntries(entry)).toEqual([ + { params: { slug: 'a' } }, + { params: { slug: 'b' } }, + { params: { slug: 'c' }, lastmod: changed, locales: ['es-co'] }, + { params: { slug: 'd' }, lastmod: '2026-10-02T00:00:00Z' }, + { params: { slug: 'e' } }, + ]); + }); + + test('enumeratePrerender answers the same pages as bare params — what a build renders', async () => { + const entry = blog(() => ['a', { params: { slug: 'b' }, locales: [], lastmod: '2026-10-02' }]); + expect(await enumeratePrerender(entry)).toEqual([{ slug: 'a' }, { slug: 'b' }]); + }); + + test('a route with no params and no prerender is its one page', async () => { + const entry = registerRoute({ file: 'apps/web/site/precios/page.tsx', config: config() }); + expect(await enumeratePrerenderEntries(entry)).toEqual([{ params: {} }]); + }); + + // `params` is only an entry's key when it holds an OBJECT: a route may name its own param that. + test('a route whose dynamic param is called `params` still takes a plain params object', async () => { + const entry = registerRoute({ + file: 'apps/web/site/q/[params]/page.tsx', + config: config(() => [{ params: 'a' }, 'b']), + }); + expect(await enumeratePrerenderEntries(entry)).toEqual([ + { params: { params: 'a' } }, + { params: { params: 'b' } }, + ]); + }); +}); + +describe('enumeratePrerenderEntries — refusals', () => { + test('a lastmod that is neither a string nor a Date names the route and its file', async () => { + const error = await refusal(() => [{ params: { slug: 'a' }, lastmod: 1_760_000_000 }]); + expect(error).toBeInstanceOf(PrerenderFailedError); + expect(error.code).toBe('X_PRERENDER_FAILED'); + expect(error.message).toContain('/blog/:slug'); + expect(error.message).toContain('lastmod'); + expect(error.fix).toContain('apps/web/site/blog/[slug]/page.tsx'); + }); + + test('locales that is not a list of strings is refused', async () => { + for (const locales of ['es-co', [5], null]) { + const error = await refusal(() => [{ params: { slug: 'a' }, locales }]); + expect(error).toBeInstanceOf(PrerenderFailedError); + expect(error.message).toContain('locales'); + } + }); + + test('a key an entry does not have is refused by name — `lastMod` is not `lastmod`', async () => { + const error = await refusal(() => [{ params: { slug: 'a' }, lastMod: '2026-10-01' }]); + expect(error).toBeInstanceOf(PrerenderFailedError); + expect(error.message).toContain('"lastMod"'); + expect(error.message).toContain('params, lastmod, locales'); + }); +}); diff --git a/packages/render/src/prerender-enumerate.ts b/packages/render/src/prerender-enumerate.ts new file mode 100644 index 000000000..982f17114 --- /dev/null +++ b/packages/render/src/prerender-enumerate.ts @@ -0,0 +1,108 @@ +// Single responsibility: what a route's `prerender()` answered, normalised — every page as its +// params, plus what the page said about itself for a sitemap (`lastmod`, `locales`). + +import { renderCauseValue, renderThrowable } from '@ultimat3/core'; +import { PrerenderFailedError } from './errors'; +import type { RouteEntry } from './registry'; +import type { PrerenderEntry, RouteParams } from './route'; + +const ENTRY_KEYS: readonly string[] = ['params', 'lastmod', 'locales']; + +const isRecord = (value: unknown): value is Readonly> => + typeof value === 'object' && value !== null && !Array.isArray(value); + +/** + * Is `item` the ENTRY form — `{ params: { … }, lastmod?, locales? }` — rather than a params object? + * Decided by `params` holding an object: a param VALUE is always a string, so `{ params: 'a' }` on a + * route whose own dynamic segment is `[params]` stays the plain params it has always been. + */ +const isEntry = (item: Readonly>): boolean => isRecord(item['params']); + +function entryOf(entry: RouteEntry, item: Readonly>): PrerenderEntry { + const refuse = (problem: string): never => { + throw new PrerenderFailedError( + `prerender() for ${entry.path} returned an entry ${problem}`, + `x build --target static --json # after prerender in ${entry.file} returns { params, lastmod?: Date | ISO 8601 string, locales?: string[] }`, + ); + }; + const extra = Object.keys(item).find((key) => !ENTRY_KEYS.includes(key)); + if (extra !== undefined) { + refuse(`with the key ${JSON.stringify(extra)}, which is not one of ${ENTRY_KEYS.join(', ')}`); + } + const { lastmod, locales } = item; + if (lastmod !== undefined && typeof lastmod !== 'string' && !(lastmod instanceof Date)) { + refuse(`whose lastmod is ${renderCauseValue(lastmod)}, neither a Date nor an ISO 8601 string`); + } + if ( + locales !== undefined && + !(Array.isArray(locales) && locales.every((locale) => typeof locale === 'string')) + ) { + refuse(`whose locales is ${renderCauseValue(locales)}, not a list of locale names`); + } + return { + params: item['params'] as RouteParams, + ...(lastmod === undefined ? {} : { lastmod: lastmod as string | Date }), + ...(locales === undefined ? {} : { locales: locales as readonly string[] }), + }; +} + +/** + * Every page `prerender()` enumerates, as an entry. A bare string fills the route's one dynamic + * param; a params object is an entry with nothing to add. `lastmod` is carried AS WRITTEN — whether + * it names an instant is the sitemap's question (`@ultimat3/seo`, `X_SITEMAP_ENTRY_INVALID`). + */ +export async function enumeratePrerenderEntries( + entry: RouteEntry, +): Promise { + const prerender = entry.config.prerender; + if (prerender === undefined) { + return entry.pattern.keys.length === 0 ? [{ params: {} }] : []; + } + + let produced: unknown; + try { + produced = await prerender(); + } catch (error) { + throw new PrerenderFailedError( + // `renderThrowable`, never `.message`/`String()`: `prerender` is app code and may throw a + // value whose read raises in turn — this frame is what makes the build failure a coded one. + `prerender() for ${entry.path} threw: ${renderThrowable(error)}`, + `fix prerender in ${entry.file} — it runs at build time with no request context`, + ); + } + + if (!Array.isArray(produced)) { + throw new PrerenderFailedError( + `prerender() for ${entry.path} returned ${typeof produced}, expected an array`, + `return an array of params from prerender in ${entry.file}`, + ); + } + + const keys = entry.pattern.keys; + return produced.map((item: unknown): PrerenderEntry => { + if (typeof item === 'string') { + const only = keys[0]; + if (keys.length !== 1 || only === undefined) { + throw new PrerenderFailedError( + `prerender() for ${entry.path} returned the bare string ${renderCauseValue(item)} ` + + `but the route has ${keys.length} dynamic params (${keys.join(', ')})`, + `return objects from prerender in ${entry.file}, e.g. { ${keys.join(': …, ')}: … }`, + ); + } + return { params: { [only]: item } }; + } + // A `null` reached `Object.hasOwn` in `static-path.ts` and left the build as a bare `TypeError`. + if (!isRecord(item)) { + throw new PrerenderFailedError( + `prerender() for ${entry.path} returned ${renderCauseValue(item)} as one of its items, which is neither a params object nor a string`, + `return an array of params objects (or strings, for a one-param route) from prerender in ${entry.file}`, + ); + } + return isEntry(item) ? entryOf(entry, item) : { params: item as RouteParams }; + }); +} + +/** The params of every page `prerender()` enumerates — what a build renders, one document each. */ +export async function enumeratePrerender(entry: RouteEntry): Promise { + return (await enumeratePrerenderEntries(entry)).map((page) => page.params); +} diff --git a/packages/render/src/render-static.ts b/packages/render/src/render-static.ts index a14a60f01..5d7142e8e 100644 --- a/packages/render/src/render-static.ts +++ b/packages/render/src/render-static.ts @@ -4,12 +4,16 @@ * the precache revision in `sw.js`, and the asset filename suffix. */ -import { BUILD_ID_HEADER, renderCauseValue, renderThrowable, useContext } from '@ultimat3/core'; +import { BUILD_ID_HEADER, renderThrowable, useContext } from '@ultimat3/core'; import { PrerenderFailedError, RouteModeInvalidError } from './errors'; +import { enumeratePrerender } from './prerender-enumerate'; import type { RouteEntry } from './registry'; import type { RenderResult, RouteParams } from './route'; import { filePathOf, filledSegments, urlPathOf } from './static-path'; +// Re-exported where it has always been imported from; the enumeration is `prerender-enumerate.ts`'s. +export { enumeratePrerender } from './prerender-enumerate'; + export interface StaticArtifact { readonly path: string; readonly params: RouteParams; @@ -61,59 +65,6 @@ export function assertNoPerRequestState(file: string): void { } } -/** Normalize `prerender()` output. A bare string fills the route's one dynamic param. */ -export async function enumeratePrerender(entry: RouteEntry): Promise { - const prerender = entry.config.prerender; - if (prerender === undefined) { - return entry.pattern.keys.length === 0 ? [{}] : []; - } - - let produced: readonly (string | RouteParams)[]; - try { - produced = await prerender(); - } catch (error) { - throw new PrerenderFailedError( - // `renderThrowable`, never `.message`/`String()`: `prerender` is app code and may throw a - // value whose read raises in turn — this frame is what makes the build failure a coded one. - `prerender() for ${entry.path} threw: ${renderThrowable(error)}`, - `fix prerender in ${entry.file} — it runs at build time with no request context`, - ); - } - - if (!Array.isArray(produced)) { - throw new PrerenderFailedError( - `prerender() for ${entry.path} returned ${typeof produced}, expected an array`, - `return an array of params from prerender in ${entry.file}`, - ); - } - - const keys = entry.pattern.keys; - return produced.map((item: unknown) => { - if (typeof item !== 'string') return paramsOf(entry, item); - const only = keys[0]; - if (keys.length !== 1 || only === undefined) { - throw new PrerenderFailedError( - `prerender() for ${entry.path} returned the bare string ${renderCauseValue(item)} ` + - `but the route has ${keys.length} dynamic params (${keys.join(', ')})`, - `return objects from prerender in ${entry.file}, e.g. { ${keys.join(': …, ')}: … }`, - ); - } - return { [only]: item }; - }); -} - -/** - * One `prerender()` item that is not a bare string: a params object, or a coded refusal. A `null` - * reached `Object.hasOwn` in `static-path.ts` and left the build as a bare `TypeError`. - */ -function paramsOf(entry: RouteEntry, item: unknown): RouteParams { - if (typeof item === 'object' && item !== null && !Array.isArray(item)) return item as RouteParams; - throw new PrerenderFailedError( - `prerender() for ${entry.path} returned ${renderCauseValue(item)} as one of its items, which is neither a params object nor a string`, - `return an array of params objects (or strings, for a one-param route) from prerender in ${entry.file}`, - ); -} - export interface StaticBuildOptions { readonly buildId: string; /** Extension-less output files get `/index.html` appended. */ diff --git a/packages/render/src/route.ts b/packages/render/src/route.ts index 03b4206c6..ea596f592 100644 --- a/packages/render/src/route.ts +++ b/packages/render/src/route.ts @@ -186,10 +186,29 @@ export type RouteMetaAsyncFn = ( ctx: RouteMetaContext, ) => Promise; -/** Returns the params to build at deploy time. Bare strings fill a single dynamic param. */ -export type PrerenderFn = () => - | readonly (string | RouteParams)[] - | Promise; +/** + * One page `prerender()` answers, with what the page says about itself for the SITEMAP. The build + * reads `params` alone: every routed locale's document is still rendered. + */ +export interface PrerenderEntry { + readonly params: RouteParams; + /** + * When this page last changed — the sitemap's `` for it. A `Date`, or ISO 8601 with an + * explicit zone (`row.updatedAt`); a moment in the future is written as now. + */ + readonly lastmod?: string | Date; + /** + * The locales this page is LISTED in: one sitemap `` each, cross-linked with hreflang among + * themselves only. Omitted: every routed locale. Empty: built, and listed nowhere. + */ + readonly locales?: readonly string[]; +} + +/** One item of a `prerender()` answer: a bare string, a params object, or an entry. */ +export type PrerenderItem = string | RouteParams | PrerenderEntry; + +/** Returns the pages to build at deploy time. Bare strings fill a single dynamic param. */ +export type PrerenderFn = () => readonly PrerenderItem[] | Promise; /** * The rule that makes the no-`load` fallback true: a route that loads nothing renders diff --git a/packages/render/src/server.ts b/packages/render/src/server.ts index 986d33da8..1de55d060 100644 --- a/packages/render/src/server.ts +++ b/packages/render/src/server.ts @@ -36,10 +36,10 @@ export { stylesheetsRevision, transformTsx, } from './module-loader'; +export { enumeratePrerenderEntries } from './prerender-enumerate'; // ---- the render pipeline: one entry point per mode ---------------------------------------------- export type { RenderHtmlOptions } from './render-html'; export { ROOT_ELEMENT_ID, renderComponent, renderToHtml } from './render-html'; - export { DEFAULT_ISR_REGENERATE_DEADLINE_MS, invalidateAndRevalidate, diff --git a/packages/seo/CLAUDE.md b/packages/seo/CLAUDE.md index 1af6518a7..e8ca3ff90 100644 --- a/packages/seo/CLAUDE.md +++ b/packages/seo/CLAUDE.md @@ -46,6 +46,15 @@ Tier 1. May import `@ultimat3/core`, `@ultimat3/schema`, `@ultimat3/i18n`. Nothi which costs the alternates that WERE right. `defaultLocaleUrl` answers `undefined` unless the default locale's own URL is among the ones this route emits — the same explicit-fallback shape `meta.ts`'s `hreflangSet` takes. Never re-derive it from `path`. +- **No `sections`, no byte changes.** The unsectioned sitemap is pinned by hash in + `sitemap-sections.test.ts`. Sorting, a child's own index `` and `-` naming + belong to the sectioned path only; a fix that must reach both paths re-pins the hash on purpose. +- **A page's `lastmod` never reads the ambient zone, and a future one is clamped.** + `sitemapDatetime` refuses a zoneless time before `Date.parse` can read it in the process's zone. + Refusing a future date would fail a whole enumeration for one skewed row; clamping to `now` is + the true upper bound. A route's `lastmod` string is the caller's and is written as given. +- **The hreflang cluster, `x-default` and the ``s read ONE trimmed list** — the page's + `locales`, minus what `disallow` blocks. Never trim one of the three alone. - **Errors name the file, not the URL.** `RouteRecord.file` is in every cause and every fix; an agent must be able to open the source without guessing. - **Fail closed, and core reads the key.** `isIndexable()` is `environment === 'production'` and nothing else — `staging`, a laptop, a typo and an unset variable all disallow. `ULTIMATE_ENV` has diff --git a/packages/seo/README.md b/packages/seo/README.md index 08c9086a0..997e209eb 100644 --- a/packages/seo/README.md +++ b/packages/seo/README.md @@ -18,6 +18,7 @@ enforced where they are built: each builder throws. | `X_SEO_CANONICAL_MISMATCH` | `meta.canonical` does not resolve to the route's own URL | a wrong canonical de-indexes the page in favour of another | | `X_LD_INVALID` | a JSON-LD node missing a required schema.org field | invalid structured data drops the rich result silently | | `X_SITEMAP_TOO_LARGE` | the sitemap index exceeds 50,000 files | past the protocol limit the whole sitemap is discarded | +| `X_SITEMAP_ENTRY_INVALID` | a page's `lastmod` is not a `Date` or ISO 8601 with a zone, or its `locales` names a locale the sitemap is not built for | a wrong date teaches a crawler to distrust every date in the file; a typo'd locale drops the page from one silently | Performance budgets are **not** here: `x verify`'s `budgets` step is `@ultimat3/cli`'s `checkBudgets`, over the route manifest and the build's own stats, and it throws @@ -47,8 +48,11 @@ a job boundary the class is gone and the `code` is what survives — match on th | `meta.ts` | the metadata model, `renderMeta()` → head tags: title template, canonical, robots, `og:*`, `twitter:*`, hreflang + `x-default`, `theme-color` per colour scheme. **`robots: { index: false }` withdraws every `og:*`, `article:*` and `twitter:*` tag** — declared or derived; there is no `social` switch | | `validate.ts` | the build gate — `validateMeta()` (`--json`-shaped) and `assertMeta()` | | `ld.ts` | typed JSON-LD builders; required fields are required in the **input type** | -| `sitemap.ts` | `buildSitemap()` from the route table + each route's `prerender()`, per-locale alternates, automatic index splitting past 50k — the index stays `/sitemap.xml`, the parts are `/sitemaps/.xml` (`SITEMAP_PARTS_DIR`) | -| `robots.ts` | `buildRobots()`, environment-aware and fail-closed; `disallow` reaches **every** group, and a `User-agent: *` group is emitted for it when none is declared | +| `sitemap.ts` | `buildSitemap()` from the route table + each route's `prerender()`, per-locale alternates. One urlset at `/sitemap.xml`; past a cap (`SITEMAP_MAX_URLS` 50,000, `SITEMAP_MAX_BYTES` 50 MB) the index stays `/sitemap.xml` and the parts are `/sitemaps/.xml` (`SITEMAP_PARTS_DIR`). With `sections` it is always an index of `/sitemaps/-.xml` — see below | +| `sitemap-sections.ts` | `sectionOf(path, sections)`: which named section a page files under (`/blog/**`, `/blog/*`, exact), first match wins, else `pages` | +| `sitemap-datetime.ts` | `sitemapDatetime(value, now)`: a page's `lastmod` as W3C Datetime in UTC, or `undefined` — a zoneless time is not a date here | +| `sitemap-xml.ts` | the `` and `` XML, and `paginate()` — files that each fit both caps | +| `robots.ts` | `buildRobots()`, environment-aware and fail-closed; `disallow` reaches **every** group, and a `User-agent: *` group is emitted for it when none is declared |. `disallowMatcher(rules)` reads those rules as a crawler does — what `buildSitemap({ disallow })` leaves out | `rss.ts` | `buildFeed()` → RSS 2.0 + Atom + JSON Feed from one item list. Channel `author`/`copyright`/`icon` → Atom ``/``/``; an item `author` → Atom ``, RSS `` when it has an email, `` when it does not; an item `image` → Atom ``, RSS ``. Extra RSS namespaces are declared only when used | | `feed-dates.ts` | the one place a feed timestamp is parsed or formatted — an item date that will not parse is *absent*, never `Invalid Date` and never a crash | | `images.ts` | `srcset` widths (never past the intrinsic width or `MAX_IMAGE_WIDTH`, 8192), modern formats before the original — `DEFAULT_FORMATS` is what the built-in driver encodes (`webp`); pass `formats: FORMAT_ORDER` for AVIF behind a CDN driver — inlined intrinsic dimensions, and `parseImageQuery()` — reads a minted URL back into a transform request | @@ -64,6 +68,33 @@ ld.Article({ headline: 'Ship it', author: { name: 'Ada' } }); A missing `datePublished` is a compile error, not a Search Console warning three weeks later. +## Sitemap: pages, sections, caps + +```ts +import { buildSitemap, type RouteRecord } from '@ultimat3/seo'; + +declare const routes: readonly RouteRecord[]; + +await buildSitemap(routes, { + baseUrl: 'https://notificado.co', + locales: ['es-co', 'en'], + defaultLocale: 'es-co', + maxUrls: 1000, // page size; default 50,000 + sections: [{ name: 'blog', match: '/blog/**' }], // omitted: one urlset + disallow: ['/panel'], // robots.txt rules: never listed +}); +// → index /sitemap.xml · files /sitemaps/pages-1.xml, /sitemaps/blog-1.xml, /sitemaps/blog-2.xml … +``` + +| Rule | | +|---|---| +| a page | `RouteRecord.prerender()` answers `'/blog/a'` or a `SitemapPage` — `{ path, lastmod?, locales? }`. `expandRoutePages()` reads both; `expandRoute()` still answers paths | +| `lastmod` | a `Date` or ISO 8601 with a zone, written in UTC. No zone, or no date: `X_SITEMAP_ENTRY_INVALID`. After `now`: clamped to `now` — one row from a clock a second ahead must not fail every page, and a future date is one a crawler stops believing | +| `locales` | per page, else `RouteRecord.locales`, else all. One `` per listed locale, hreflang among those only; `x-default` only when the default is listed; a page left in one locale of several carries no alternates; `[]` lists it nowhere. An unknown name is `X_SITEMAP_ENTRY_INVALID` | +| sections | matched on the unprefixed path, so a page and its locale copies share a file. `pages` (reserved) first, then declaration order; inside one, sorted by path. Numbered from 1; an empty one writes nothing; the index `` of a child is the newest inside it (`SitemapFile.lastmod`) | +| no sections | byte-for-byte the output before sections existed — `sitemap-sections.test.ts` pins the hash | +| names | `sections` and `maxUrls` are refused by `@ultimat3/core`'s `sitemapSectionIssues`, the validator `defineConfig` runs for `seo.sitemap` | + ## robots.txt is fail-closed Only the literal string `production` in `ULTIMATE_ENV` / `NODE_ENV` opts a deploy diff --git a/packages/seo/src/errors.ts b/packages/seo/src/errors.ts index 673fccafe..a4cdcb532 100644 --- a/packages/seo/src/errors.ts +++ b/packages/seo/src/errors.ts @@ -19,6 +19,7 @@ export const SEO_ERROR_CODES = { sitemapTooLarge: 'X_SITEMAP_TOO_LARGE', imageQueryInvalid: 'X_IMAGE_QUERY_INVALID', linkInvalid: 'X_SEO_LINK_INVALID', + sitemapEntryInvalid: 'X_SITEMAP_ENTRY_INVALID', } as const; export type SeoErrorCode = (typeof SEO_ERROR_CODES)[keyof typeof SEO_ERROR_CODES]; @@ -42,6 +43,9 @@ registerErrorCodes({ X_SITEMAP_TOO_LARGE: { title: 'sitemap exceeds the 50,000-entry protocol limit' }, X_IMAGE_QUERY_INVALID: { title: 'an image transform query parameter is present but unusable' }, X_SEO_LINK_INVALID: { title: 'a route meta link is unsafe or would fetch nothing' }, + X_SITEMAP_ENTRY_INVALID: { + title: 'a sitemap entry carries a lastmod or a locale the sitemap cannot write', + }, }); export interface SeoErrorInit { @@ -148,6 +152,19 @@ export function sitemapTooLarge(count: number, max: number): SeoError { }); } +/** + * A page a sitemap was asked to list with a `lastmod` or a locale it cannot write. `file` is the + * route's source — where the `prerender()` that answered the entry lives. + */ +export function sitemapEntryInvalid(file: string, path: string, problem: string): SeoError { + return new SeoError({ + code: SEO_ERROR_CODES.sitemapEntryInvalid, + cause: `${file} lists "${path}" in the sitemap with ${problem}`, + fix: `x verify --json # after prerender() in ${file} answers lastmod as a Date or an ISO 8601 string with a zone (row.updatedAt), and locales naming only locales the app routes`, + meta: { file, path, problem }, + }); +} + /** * `parseImageQuery`'s only refusal: a `?w=`/`?q=`/`?f=` value present but not usable — serving * the untransformed original against a URL that asked for a size would be the layout shift this diff --git a/packages/seo/src/index.ts b/packages/seo/src/index.ts index fdc07cbf1..1e1a0f82b 100644 --- a/packages/seo/src/index.ts +++ b/packages/seo/src/index.ts @@ -13,6 +13,7 @@ export { notImplementedDriver, SEO_ERROR_CODES, SeoError, + sitemapEntryInvalid, sitemapTooLarge, } from './errors'; export type { @@ -98,9 +99,9 @@ export { TITLE_MAX_LENGTH, } from './meta'; export type { RobotsConfig, RobotsGroup } from './robots'; -export { buildRobots, isIndexable } from './robots'; -export type { ChangeFreq, RouteRecord, Surface } from './routes'; -export { expandRoute, indexableRoutes, isDynamic } from './routes'; +export { buildRobots, disallowMatcher, isIndexable } from './robots'; +export type { ChangeFreq, RouteRecord, SitemapPage, Surface } from './routes'; +export { expandRoute, expandRoutePages, indexableRoutes, isDynamic } from './routes'; export type { BuildFeedOptions, Feed, FeedAuthor, FeedChannel, FeedItem } from './rss'; export { buildFeed } from './rss'; export type { @@ -114,10 +115,14 @@ export { buildSitemap, chunk, SITEMAP_INDEX_MAX_FILES, + SITEMAP_MAX_BYTES, SITEMAP_MAX_URLS, SITEMAP_PARTS_DIR, sitemapUrls, } from './sitemap'; +export { sitemapDatetime } from './sitemap-datetime'; +export type { SitemapSection } from './sitemap-sections'; +export { sectionOf } from './sitemap-sections'; export type { MetaIssue, MetaValidationReport, ValidateMetaOptions } from './validate'; export { assertMeta, validateMeta } from './validate'; export { absoluteUrl, escapeXml } from './xml'; diff --git a/packages/seo/src/robots.ts b/packages/seo/src/robots.ts index b03bc5e26..3a53a5a24 100644 --- a/packages/seo/src/robots.ts +++ b/packages/seo/src/robots.ts @@ -96,3 +96,27 @@ export function buildRobots(config: RobotsConfig): string { return `${lines.join('\n').trimEnd()}\n`; } + +/** + * `rule` as a crawler reads it: a PREFIX of the path, `*` standing for any run of characters and a + * trailing `$` anchoring the end. An empty rule disallows nothing. + */ +function disallowPattern(rule: string): RegExp | undefined { + if (rule === '') return undefined; + const anchored = rule.endsWith('$'); + const body = (anchored ? rule.slice(0, -1) : rule) + .split('*') + .map((literal) => literal.replace(/[.+?^${}()|[\]\\]/g, '\\$&')) + .join('.*'); + return new RegExp(`^${body}${anchored ? '$' : ''}`); +} + +/** + * A test for "do these `Disallow:` rules keep a crawler out of this path". What the sitemap asks + * before it lists a URL: one that `robots.txt` blocks is reported as "submitted URL blocked by + * robots.txt", and the two files are written from one config. + */ +export function disallowMatcher(rules: readonly string[]): (path: string) => boolean { + const patterns = rules.flatMap((rule) => disallowPattern(rule) ?? []); + return (path) => patterns.some((pattern) => pattern.test(path)); +} diff --git a/packages/seo/src/routes.ts b/packages/seo/src/routes.ts index 925a7414e..d9086ae4d 100644 --- a/packages/seo/src/routes.ts +++ b/packages/seo/src/routes.ts @@ -10,6 +10,25 @@ export type Surface = 'site' | 'app' | 'api'; export type ChangeFreq = 'always' | 'hourly' | 'daily' | 'weekly' | 'monthly' | 'yearly' | 'never'; +/** + * One concrete page of a route, as a sitemap lists it. A bare string is the same thing with nothing + * to add: `'/blog/a'` is `{ path: '/blog/a' }`. + */ +export interface SitemapPage { + /** The concrete path, WITHOUT a locale prefix: `/blog/a`. */ + readonly path: string; + /** + * When this page last changed: a `Date`, or ISO 8601 with an explicit zone (`2026-10-01`, + * `2026-10-01T15:04:05Z`). Written as W3C Datetime in UTC; wins over the route's own `lastmod`. + */ + readonly lastmod?: string | Date; + /** + * The locales this page is LISTED in — one `` each, cross-linked with hreflang among + * themselves only. Omitted: every locale the sitemap is built for. Empty: listed nowhere. + */ + readonly locales?: readonly string[]; +} + export interface RouteRecord { /** URL pattern, e.g. `/blog/:slug`. */ path: string; @@ -18,8 +37,13 @@ export interface RouteRecord { surface: Surface; render: RenderMode; meta?: RouteMeta; - /** Concrete paths for a dynamic route, from `defineRoute({ prerender })`. */ - prerender?: () => readonly string[] | Promise; + /** The route's concrete pages, from `defineRoute({ prerender })` — required for a dynamic one. */ + prerender?: () => readonly (string | SitemapPage)[] | Promise; + /** + * The locales every page of this route is listed in, unless the page names its own + * (`SitemapPage.locales`). Omitted: every locale the sitemap is built for. + */ + locales?: readonly string[]; /** Keep out of the sitemap and emit `noindex`. */ noindex?: boolean; /** @@ -47,9 +71,20 @@ export function indexableRoutes(routes: readonly RouteRecord[]): readonly RouteR ); } +/** + * Every concrete page a route resolves to, each in the one object form. A route that declares + * `prerender()` is the pages it answers — with or without params: a page with none may still say + * its own `lastmod` and `locales` there. Without one, a static path is its one page and a dynamic + * one is none. + */ +export async function expandRoutePages(route: RouteRecord): Promise { + if (route.prerender === undefined) return isDynamic(route.path) ? [] : [{ path: route.path }]; + return (await route.prerender()).map((page) => + typeof page === 'string' ? { path: page } : page, + ); +} + /** Every concrete URL a route resolves to, expanding `prerender()`. */ export async function expandRoute(route: RouteRecord): Promise { - if (!isDynamic(route.path)) return [route.path]; - if (route.prerender === undefined) return []; - return await route.prerender(); + return (await expandRoutePages(route)).map((page) => page.path); } diff --git a/packages/seo/src/sitemap-datetime.ts b/packages/seo/src/sitemap-datetime.ts new file mode 100644 index 000000000..4148f0c9a --- /dev/null +++ b/packages/seo/src/sitemap-datetime.ts @@ -0,0 +1,33 @@ +// Single responsibility: one page's `lastmod`, as the W3C Datetime a sitemap writes — or the +// answer that it is not a date. No ambient zone is ever read: a time without one is refused. + +/** + * ISO 8601 as a sitemap accepts it, with the zone REQUIRED on a time. `Date.parse` alone is not the + * test: it reads `2026-10-01T10:00:00` and `October 1, 2026` in the process's own zone, so one row + * would be two instants on a laptop in Bogotá and in a container on UTC. + */ +const ISO_ZONED = + /^\d{4}-\d{2}-\d{2}(?:T\d{2}:\d{2}(?::\d{2}(?:\.\d{1,9})?)?(?:Z|[+-]\d{2}:\d{2}))?$/; + +/** What a refusal quotes back, so the fix names the forms that are accepted. */ +export const SITEMAP_DATETIME_FORMS = + 'a Date or an ISO 8601 date with an explicit zone (2026-10-01, 2026-10-01T15:04:05Z, 2026-10-01T10:04:05-05:00)'; + +/** + * `value` in UTC (`2026-10-01T15:04:05.000Z`), or `undefined` when it names no instant. + * + * A moment after `now` is CLAMPED to `now`, never refused and never written as it stands. Written, + * it is a date a crawler learns to distrust for the whole file. Refused, one row stamped by a + * database a second ahead of this process fails the enumeration for every page. Clamped, it says + * the one thing that is certainly true of a change that already happened: no later than now. + */ +export function sitemapDatetime(value: unknown, now: Date): string | undefined { + const time = + value instanceof Date + ? value.getTime() + : typeof value === 'string' && ISO_ZONED.test(value) + ? Date.parse(value) + : Number.NaN; + if (!Number.isFinite(time)) return undefined; + return new Date(Math.min(time, now.getTime())).toISOString(); +} diff --git a/packages/seo/src/sitemap-pages.test.ts b/packages/seo/src/sitemap-pages.test.ts new file mode 100644 index 000000000..9e2b55cf9 --- /dev/null +++ b/packages/seo/src/sitemap-pages.test.ts @@ -0,0 +1,289 @@ +// What ONE page may say about itself in a sitemap: when it last changed, and which locales it is +// listed in — beside the plain path a `prerender()` has always answered. Plus the URLs a sitemap +// must not carry at all: the ones `robots.txt` keeps crawlers out of. + +import { describe, expect, test } from 'bun:test'; +import { SEO_ERROR_CODES } from './errors'; +import type { RouteRecord, SitemapPage } from './routes'; +import { expandRoute, expandRoutePages } from './routes'; +import { buildSitemap, sitemapUrls } from './sitemap'; +import { sitemapDatetime } from './sitemap-datetime'; + +const ORIGIN = 'https://notificado.co'; +const NOW = new Date('2026-10-11T12:00:00.000Z'); +const BASE = { baseUrl: ORIGIN, now: NOW } as const; +const LOCALES = { locales: ['es-co', 'en'], defaultLocale: 'es-co' } as const; + +function route(partial: Partial & Pick): RouteRecord { + return { surface: 'site', render: 'static', ...partial }; +} + +const blog = (pages: readonly (string | SitemapPage)[]): RouteRecord => + route({ path: '/blog/:slug', file: 'site/blog/[slug]/page.tsx', prerender: () => pages }); + +async function refusal( + run: () => Promise, +): Promise<{ code?: string; cause?: string; fix?: string }> { + try { + await run(); + } catch (error) { + return error as { code?: string; cause?: string; fix?: string }; + } + return {}; +} + +describe('expandRoutePages', () => { + test('a plain path and a page object are one list; expandRoute still answers paths', async () => { + const record = blog(['/blog/a', { path: '/blog/b', lastmod: '2026-10-01' }]); + expect(await expandRoutePages(record)).toEqual([ + { path: '/blog/a' }, + { path: '/blog/b', lastmod: '2026-10-01' }, + ]); + expect(await expandRoute(record)).toEqual(['/blog/a', '/blog/b']); + }); + + test('a static route is its own one page; a dynamic one with no prerender is none', async () => { + expect( + await expandRoutePages(route({ path: '/precios', file: 'site/precios/page.tsx' })), + ).toEqual([{ path: '/precios' }]); + expect(await expandRoutePages(route({ path: '/b/:id', file: 'site/b/[id]/page.tsx' }))).toEqual( + [], + ); + }); +}); + +describe('a route with no params', () => { + // The blog's index is one page whose date is its newest post's: a route with no params may + // answer `prerender()` too, and what it says of its one page is read like any other entry. + test('says its page`s lastmod and locales through prerender(), when it declares one', async () => { + const urls = await sitemapUrls( + [ + route({ + path: '/blog', + file: 'site/blog/page.tsx', + prerender: () => [{ path: '/blog', lastmod: '2026-10-01T00:00:00Z', locales: ['es-co'] }], + }), + route({ path: '/borradores', file: 'site/borradores/page.tsx', prerender: () => [] }), + ], + { ...BASE, ...LOCALES }, + ); + expect(urls).toEqual([{ loc: `${ORIGIN}/blog`, lastmod: '2026-10-01T00:00:00.000Z' }]); + }); +}); + +describe('lastmod per page', () => { + test('a Date, a zoned ISO string and a date are each written as W3C Datetime in UTC', async () => { + const urls = await sitemapUrls( + [ + blog([ + { path: '/blog/a', lastmod: new Date('2026-10-01T15:04:05.000Z') }, + { path: '/blog/b', lastmod: '2026-10-05T08:30:00-05:00' }, + { path: '/blog/c', lastmod: '2026-09-30' }, + '/blog/d', + ]), + ], + BASE, + ); + expect(urls.map((url) => url.lastmod)).toEqual([ + '2026-10-01T15:04:05.000Z', + '2026-10-05T13:30:00.000Z', + '2026-09-30T00:00:00.000Z', + undefined, + ]); + }); + + test('the page`s own lastmod wins over the route`s, which wins over the build`s', async () => { + const urls = await sitemapUrls( + [ + route({ + path: '/blog/:slug', + file: 'site/blog/[slug]/page.tsx', + lastmod: '2026-01-01T00:00:00.000Z', + prerender: () => [{ path: '/blog/a', lastmod: '2026-10-01T00:00:00Z' }, '/blog/b'], + }), + route({ path: '/precios', file: 'site/precios/page.tsx' }), + ], + { ...BASE, lastmod: '2025-05-05T00:00:00.000Z' }, + ); + expect(urls.map((url) => url.lastmod)).toEqual([ + '2026-10-01T00:00:00.000Z', + '2026-01-01T00:00:00.000Z', + '2025-05-05T00:00:00.000Z', + ]); + }); + + test('every locale copy of a page carries the page`s lastmod', async () => { + const result = await buildSitemap( + [blog([{ path: '/blog/a', lastmod: '2026-10-01T00:00:00Z' }])], + { + ...BASE, + ...LOCALES, + }, + ); + expect( + result.files[0]?.xml.match(/2026-10-01T00:00:00\.000Z<\/lastmod>/g), + ).toHaveLength(2); + }); + + // CLAMPED, not refused: one row stamped by a database a second ahead of this process would + // otherwise fail the whole enumeration — and "no later than now" is the true bound on a past event. + test('a lastmod in the future is clamped to now', async () => { + const urls = await sitemapUrls( + [blog([{ path: '/blog/a', lastmod: '2031-01-01T00:00:00Z' }])], + BASE, + ); + expect(urls[0]?.lastmod).toBe('2026-10-11T12:00:00.000Z'); + }); + + test('a lastmod that is no date is refused, naming the route file and the page', async () => { + for (const lastmod of ['yesterday', '', '2026-13-45', new Date(Number.NaN)]) { + const error = await refusal(() => sitemapUrls([blog([{ path: '/blog/a', lastmod }])], BASE)); + expect(error.code).toBe(SEO_ERROR_CODES.sitemapEntryInvalid); + expect(error.cause).toContain('site/blog/[slug]/page.tsx'); + expect(error.cause).toContain('/blog/a'); + expect(error.fix).toContain('site/blog/[slug]/page.tsx'); + } + }); + + // `Date.parse` reads a zoneless time in the PROCESS's zone: the same row would be a different + // instant on a laptop in Bogotá and in a container on UTC. + test('a time with no zone is refused rather than read in the ambient one', async () => { + for (const lastmod of ['2026-10-01T10:00:00', 'October 1, 2026', '10/01/2026']) { + const error = await refusal(() => sitemapUrls([blog([{ path: '/blog/a', lastmod }])], BASE)); + expect(error.code).toBe(SEO_ERROR_CODES.sitemapEntryInvalid); + expect(error.cause).toContain('zone'); + } + }); + + test('sitemapDatetime is the one normaliser', () => { + expect(sitemapDatetime('2026-10-01T00:00:00+02:00', NOW)).toBe('2026-09-30T22:00:00.000Z'); + expect(sitemapDatetime('2026-10-01T00:00Z', NOW)).toBe('2026-10-01T00:00:00.000Z'); + expect(sitemapDatetime('nope', NOW)).toBeUndefined(); + }); +}); + +describe('locales per page', () => { + test('a page listed in one locale is one , with no alternates pointing at the other', async () => { + const result = await buildSitemap( + [blog([{ path: '/blog/solo', locales: ['es-co'] }, '/blog/ambos'])], + { ...BASE, ...LOCALES }, + ); + const xml = result.files[0]?.xml ?? ''; + expect(xml).toContain(`${ORIGIN}/blog/solo`); + expect(xml).not.toContain('/en/blog/solo'); + const solo = xml.slice(xml.indexOf('/blog/solo'), xml.indexOf('')); + expect(solo).not.toContain('xhtml:link'); + // The page in both locales is untouched: two s, the full cluster on each. + expect(xml).toContain(`${ORIGIN}/en/blog/ambos`); + expect(xml.split(`hreflang="x-default" href="${ORIGIN}/blog/ambos"`).length - 1).toBe(2); + expect(result.urlCount).toBe(3); + }); + + test('a cluster that leaves the default locale out names no x-default', async () => { + const urls = await sitemapUrls([blog([{ path: '/blog/a', locales: ['en', 'pt'] }])], { + ...BASE, + locales: ['es-co', 'en', 'pt'], + defaultLocale: 'es-co', + }); + expect(urls.map((url) => url.loc)).toEqual([`${ORIGIN}/en/blog/a`, `${ORIGIN}/pt/blog/a`]); + expect(urls[0]?.alternates?.map((alternate) => alternate.hreflang)).toEqual(['en', 'pt']); + }); + + test('the cluster keeps the app`s locale order, whatever order the page wrote', async () => { + const urls = await sitemapUrls([blog([{ path: '/blog/a', locales: ['en', 'es-co'] }])], { + ...BASE, + ...LOCALES, + }); + expect(urls.map((url) => url.loc)).toEqual([`${ORIGIN}/blog/a`, `${ORIGIN}/en/blog/a`]); + }); + + test('`locales: []` lists the page nowhere', async () => { + const urls = await sitemapUrls([blog([{ path: '/blog/borrador', locales: [] }, '/blog/a'])], { + ...BASE, + ...LOCALES, + }); + expect(urls.map((url) => url.loc)).toEqual([`${ORIGIN}/blog/a`, `${ORIGIN}/en/blog/a`]); + // And on a single-locale site, where there is no cluster to trim: still nowhere. + const single = await sitemapUrls( + [blog([{ path: '/blog/borrador', locales: [] }, '/blog/a'])], + BASE, + ); + expect(single.map((url) => url.loc)).toEqual([`${ORIGIN}/blog/a`]); + }); + + test('a locale the app does not route is refused, naming the file — never a silent drop', async () => { + const error = await refusal(() => + sitemapUrls([blog([{ path: '/blog/a', locales: ['es-co', 'en-us'] }])], { + ...BASE, + ...LOCALES, + }), + ); + expect(error.code).toBe(SEO_ERROR_CODES.sitemapEntryInvalid); + expect(error.cause).toContain('"en-us"'); + expect(error.cause).toContain('site/blog/[slug]/page.tsx'); + expect(error.cause).toContain('es-co, en'); + }); + + test('a route`s own `locales` is the default for its pages; a page`s wins', async () => { + const urls = await sitemapUrls( + [ + route({ path: '/legal', file: 'site/legal/page.tsx', locales: ['es-co'] }), + route({ + path: '/blog/:slug', + file: 'site/blog/[slug]/page.tsx', + locales: ['es-co'], + prerender: () => ['/blog/a', { path: '/blog/b', locales: ['es-co', 'en'] }], + }), + ], + { ...BASE, ...LOCALES }, + ); + expect(urls.map((url) => url.loc)).toEqual([ + `${ORIGIN}/legal`, + `${ORIGIN}/blog/a`, + `${ORIGIN}/blog/b`, + `${ORIGIN}/en/blog/b`, + ]); + expect(urls[0]?.alternates).toBeUndefined(); + }); +}); + +describe('disallow', () => { + test('a URL robots.txt keeps crawlers out of is not listed, in any locale', async () => { + const urls = await sitemapUrls( + [ + route({ path: '/panel', file: 'site/panel/page.tsx' }), + route({ path: '/panel-de-control', file: 'site/panel-de-control/page.tsx' }), + route({ path: '/precios', file: 'site/precios/page.tsx' }), + ], + { ...BASE, disallow: ['/panel/', '/panel$', '/*/panel$'], ...LOCALES }, + ); + expect(urls.map((url) => url.loc)).toEqual([ + `${ORIGIN}/panel-de-control`, + `${ORIGIN}/en/panel-de-control`, + `${ORIGIN}/precios`, + `${ORIGIN}/en/precios`, + ]); + }); + + test('a rule that keeps one locale copy out trims the cluster to what is left', async () => { + const urls = await sitemapUrls([route({ path: '/precios', file: 'site/precios/page.tsx' })], { + ...BASE, + ...LOCALES, + disallow: ['/en/'], + }); + expect(urls.map((url) => url.loc)).toEqual([`${ORIGIN}/precios`]); + expect(urls[0]?.alternates).toBeUndefined(); + }); + + test('a plain rule is a prefix, as a crawler reads it', async () => { + const urls = await sitemapUrls( + [ + route({ path: '/api', file: 'site/api/page.tsx' }), + route({ path: '/apis-abiertas', file: 'site/apis-abiertas/page.tsx' }), + route({ path: '/', file: 'site/page.tsx' }), + ], + { ...BASE, disallow: ['/api'] }, + ); + expect(urls.map((url) => url.loc)).toEqual([ORIGIN]); + }); +}); diff --git a/packages/seo/src/sitemap-sections.test.ts b/packages/seo/src/sitemap-sections.test.ts new file mode 100644 index 000000000..b25e29cb2 --- /dev/null +++ b/packages/seo/src/sitemap-sections.test.ts @@ -0,0 +1,313 @@ +// Named, paginated sitemap sections: `/sitemap.xml` as an index by CONFIGURATION, each child a +// valid urlset, the caps still splitting a section — and, with no section declared, the bytes +// every release before this one wrote. + +import { describe, expect, test } from 'bun:test'; +import { SEO_ERROR_CODES } from './errors'; +import type { RouteRecord } from './routes'; +import { buildSitemap, SITEMAP_MAX_BYTES } from './sitemap'; +import { sectionOf } from './sitemap-sections'; + +const ORIGIN = 'https://notificado.co'; +const BASE = { baseUrl: ORIGIN } as const; + +function route(partial: Partial & Pick): RouteRecord { + return { surface: 'site', render: 'static', ...partial }; +} + +const locs = (xml: string | undefined) => + [...(xml ?? '').matchAll(/([^<]+)<\/loc>/g)].map((m) => m[1] ?? ''); +const sha = (text: string) => new Bun.CryptoHasher('sha256').update(text).digest('hex'); + +const SECTIONS = [ + { name: 'blog', match: '/blog/**' }, + { name: 'ciudades', match: ['/juzgados/**', '/ciudades/*'] }, +] as const; + +const site = (): RouteRecord[] => [ + route({ path: '/', file: 'site/page.tsx', lastmod: '2026-10-01T00:00:00.000Z' }), + route({ path: '/precios', file: 'site/precios/page.tsx', changefreq: 'weekly', priority: 0.8 }), + route({ + path: '/blog/:slug', + file: 'site/blog/[slug]/page.tsx', + render: 'isr', + prerender: () => ['/blog/b', '/blog/a?x=1&y=2'], + }), +]; + +describe('no sections configured', () => { + // Captured from the release before sections existed (27.8.1), over this exact fixture. + test('the single urlset is byte-identical to what it always was', async () => { + const result = await buildSitemap(site(), { + ...BASE, + locales: ['es-co', 'en'], + defaultLocale: 'es-co', + }); + expect(result.index).toBeUndefined(); + expect(result.files.map((file) => file.path)).toEqual(['/sitemap.xml']); + expect(result.files[0]?.xml.length).toBe(3047); + expect(sha(result.files[0]?.xml ?? '')).toBe( + 'e17df8fe22fec2d3f10140132f883e707bfbcc934f55f98f455188e745b36c7e', + ); + }); + + test('a split past the cap is byte-identical too: /sitemaps/.xml, an index with no lastmod', async () => { + const result = await buildSitemap(site(), { ...BASE, maxUrls: 3 }); + const all = [result.index, ...result.files].map((file) => `${file?.path}\n${file?.xml}`); + expect(sha(all.join('\n'))).toBe( + 'de7c14d80e71b8b804596b91a108ad0dc3ea7cd4d5bf2d8f4eb2fc406a25a457', + ); + }); + + test('an empty `sections` is no sections', async () => { + const result = await buildSitemap(site(), { ...BASE, sections: [] }); + expect(result.index).toBeUndefined(); + expect(result.files.map((file) => file.path)).toEqual(['/sitemap.xml']); + }); +}); + +describe('sections', () => { + test('/sitemap.xml is an index of named, numbered children; unmatched pages go to `pages`', async () => { + const result = await buildSitemap( + [ + ...site(), + route({ + path: '/juzgados/:ciudad', + file: 'site/juzgados/[ciudad]/page.tsx', + prerender: () => ['/juzgados/medellin', '/juzgados/bogota'], + }), + ], + { ...BASE, sections: SECTIONS }, + ); + expect(result.index?.path).toBe('/sitemap.xml'); + expect(result.index?.xml).toBe( + '\n' + + '\n' + + ' \n' + + ` ${ORIGIN}/sitemaps/pages-1.xml\n` + + ' 2026-10-01T00:00:00.000Z\n' + + ' \n' + + ' \n' + + ` ${ORIGIN}/sitemaps/blog-1.xml\n` + + ' \n' + + ' \n' + + ` ${ORIGIN}/sitemaps/ciudades-1.xml\n` + + ' \n' + + '\n', + ); + expect(result.files.map((file) => file.path)).toEqual([ + '/sitemaps/pages-1.xml', + '/sitemaps/blog-1.xml', + '/sitemaps/ciudades-1.xml', + ]); + expect(result.urlCount).toBe(6); + for (const file of result.files) { + expect(file.xml.startsWith('\n\n')).toBe(true); + } + expect(locs(result.files[0]?.xml)).toEqual([ORIGIN, `${ORIGIN}/precios`]); + // Sorted by path inside a section, whatever order `prerender()` answered in. + expect(locs(result.files[1]?.xml)).toEqual([ + `${ORIGIN}/blog/a?x=1&y=2`, + `${ORIGIN}/blog/b`, + ]); + expect(locs(result.files[2]?.xml)).toEqual([ + `${ORIGIN}/juzgados/bogota`, + `${ORIGIN}/juzgados/medellin`, + ]); + }); + + test('pageSize paginates each section on its own', async () => { + const posts = Array.from({ length: 5 }, (_unused, index) => `/blog/p${index}`); + const result = await buildSitemap( + [ + route({ path: '/', file: 'site/page.tsx' }), + route({ path: '/blog/:slug', file: 'site/blog/[slug]/page.tsx', prerender: () => posts }), + ], + { ...BASE, sections: SECTIONS, maxUrls: 2 }, + ); + expect(result.files.map((file) => [file.path, file.urlCount])).toEqual([ + ['/sitemaps/pages-1.xml', 1], + ['/sitemaps/blog-1.xml', 2], + ['/sitemaps/blog-2.xml', 2], + ['/sitemaps/blog-3.xml', 1], + ]); + expect(locs(result.index?.xml)).toEqual(result.files.map((file) => `${ORIGIN}${file.path}`)); + }); + + test('a section that would be empty is omitted, the default one included', async () => { + const result = await buildSitemap( + [ + route({ + path: '/blog/:slug', + file: 'site/blog/[slug]/page.tsx', + prerender: () => ['/blog/a'], + }), + ], + { ...BASE, sections: SECTIONS }, + ); + expect(result.files.map((file) => file.path)).toEqual(['/sitemaps/blog-1.xml']); + expect(locs(result.index?.xml)).toEqual([`${ORIGIN}/sitemaps/blog-1.xml`]); + }); + + test('a site with no URL at all is the empty urlset, never an index of nothing', async () => { + const result = await buildSitemap([], { ...BASE, sections: SECTIONS }); + expect(result.index).toBeUndefined(); + expect(result.files.map((file) => file.path)).toEqual(['/sitemap.xml']); + expect(result.files[0]?.xml).toContain(' { + const build = (slugs: readonly string[]) => + buildSitemap( + [ + route({ path: '/blog/:slug', file: 'site/blog/[slug]/page.tsx', prerender: () => slugs }), + route({ path: '/precios', file: 'site/precios/page.tsx' }), + route({ path: '/', file: 'site/page.tsx' }), + ], + { + ...BASE, + sections: SECTIONS, + maxUrls: 2, + locales: ['es-co', 'en'], + defaultLocale: 'es-co', + }, + ); + const first = await build(['/blog/c', '/blog/a', '/blog/b']); + const second = await build(['/blog/b', '/blog/c', '/blog/a']); + expect(second.files).toEqual(first.files); + expect(second.index).toEqual(first.index); + }); + + test('a page and its locale copies file together, each carrying the whole hreflang cluster', async () => { + const result = await buildSitemap( + [ + route({ path: '/', file: 'site/page.tsx' }), + route({ + path: '/blog/:slug', + file: 'site/blog/[slug]/page.tsx', + prerender: () => ['/blog/a'], + }), + ], + { ...BASE, sections: SECTIONS, locales: ['es-co', 'en'], defaultLocale: 'es-co' }, + ); + const blog = result.files.find((file) => file.path === '/sitemaps/blog-1.xml')?.xml ?? ''; + expect(locs(blog)).toEqual([`${ORIGIN}/blog/a`, `${ORIGIN}/en/blog/a`]); + expect(blog).toContain('xmlns:xhtml="http://www.w3.org/1999/xhtml"'); + for (const tag of [ + `hreflang="es-CO" href="${ORIGIN}/blog/a"`, + `hreflang="en" href="${ORIGIN}/en/blog/a"`, + `hreflang="x-default" href="${ORIGIN}/blog/a"`, + ]) { + expect(blog.split(tag).length - 1).toBe(2); + } + }); + + test('the index lastmod of a child is the newest lastmod inside it', async () => { + const result = await buildSitemap( + [ + route({ + path: '/blog/:slug', + file: 'site/blog/[slug]/page.tsx', + prerender: () => [ + { path: '/blog/a', lastmod: '2026-09-01T10:00:00Z' }, + { path: '/blog/b', lastmod: '2026-10-05T08:30:00-05:00' }, + { path: '/blog/c' }, + ], + }), + ], + { ...BASE, sections: SECTIONS, now: new Date('2026-10-11T00:00:00Z') }, + ); + expect(result.index?.xml).toContain('2026-10-05T13:30:00.000Z'); + expect(result.index?.xml.match(//g)).toHaveLength(1); + }); + + test('an invalid section is refused before any file is built', async () => { + for (const sections of [ + [{ name: 'Blog', match: '/blog/**' }], + [{ name: 'pages', match: '/a' }], + [{ name: 'blog', match: 'blog' }], + ]) { + let code: unknown; + try { + await buildSitemap(site(), { ...BASE, sections }); + } catch (error) { + code = (error as { code?: unknown }).code; + } + expect(code).toBe('X_INVARIANT'); + } + }); + + test('more children than an index can hold is refused', async () => { + const posts = Array.from({ length: 50_001 }, (_unused, index) => `/blog/${index}`); + let code: unknown; + try { + await buildSitemap( + [route({ path: '/blog/:slug', file: 'site/blog/[slug]/page.tsx', prerender: () => posts })], + { ...BASE, sections: SECTIONS, maxUrls: 1 }, + ); + } catch (error) { + code = (error as { code?: unknown }).code; + } + expect(code).toBe(SEO_ERROR_CODES.sitemapTooLarge); + }); +}); + +describe('the byte cap', () => { + test('is the protocol`s 50 MB, uncompressed', () => { + expect(SITEMAP_MAX_BYTES).toBe(52_428_800); + }); + + test('a section over the byte cap splits further, each file under it', async () => { + const posts = Array.from({ length: 40 }, (_unused, index) => `/blog/post-${index}`); + const maxBytes = 1200; + const result = await buildSitemap( + [route({ path: '/blog/:slug', file: 'site/blog/[slug]/page.tsx', prerender: () => posts })], + { ...BASE, sections: SECTIONS, maxBytes }, + ); + expect(result.files.length).toBeGreaterThan(1); + expect(result.urlCount).toBe(40); + for (const file of result.files) { + expect(new TextEncoder().encode(file.xml).length).toBeLessThanOrEqual(maxBytes); + } + expect(result.files.map((file) => file.path)).toEqual( + result.files.map((_file, index) => `/sitemaps/blog-${index + 1}.xml`), + ); + }); + + test('without sections the byte cap splits the one urlset the same way', async () => { + const posts = Array.from({ length: 40 }, (_unused, index) => `/blog/post-${index}`); + const result = await buildSitemap( + [route({ path: '/blog/:slug', file: 'site/blog/[slug]/page.tsx', prerender: () => posts })], + { ...BASE, maxBytes: 1200 }, + ); + expect(result.index?.path).toBe('/sitemap.xml'); + expect(result.files[0]?.path).toBe('/sitemaps/1.xml'); + for (const file of result.files) { + expect(new TextEncoder().encode(file.xml).length).toBeLessThanOrEqual(1200); + } + }); +}); + +describe('sectionOf', () => { + test('exact, one-segment and subtree patterns; the first section that matches wins', () => { + const sections = [ + { name: 'guias', match: '/blog/guias/**' }, + { name: 'blog', match: ['/blog/**'] }, + { name: 'ciudades', match: '/ciudades/*' }, + { name: 'precios', match: '/precios' }, + ]; + expect(sectionOf('/blog', sections)).toBe('blog'); + expect(sectionOf('/blog/a', sections)).toBe('blog'); + expect(sectionOf('/blog/a/b', sections)).toBe('blog'); + expect(sectionOf('/blog/guias/x', sections)).toBe('guias'); + expect(sectionOf('/blogs', sections)).toBe('pages'); + expect(sectionOf('/ciudades/cali', sections)).toBe('ciudades'); + expect(sectionOf('/ciudades', sections)).toBe('pages'); + expect(sectionOf('/ciudades/cali/juzgados', sections)).toBe('pages'); + expect(sectionOf('/precios', sections)).toBe('precios'); + expect(sectionOf('/precios/', sections)).toBe('precios'); + expect(sectionOf('/precios?plan=a', sections)).toBe('precios'); + expect(sectionOf('/', sections)).toBe('pages'); + }); +}); diff --git a/packages/seo/src/sitemap-sections.ts b/packages/seo/src/sitemap-sections.ts new file mode 100644 index 000000000..ef7628351 --- /dev/null +++ b/packages/seo/src/sitemap-sections.ts @@ -0,0 +1,50 @@ +// Single responsibility: which named section of a sitemap index a page files under. The shape and +// its refusals are `@ultimat3/core`'s (`seo.sitemap.sections`), so `defineConfig` and a direct +// `buildSitemap` call reject the same things in the same words. + +import { + assert, + SITEMAP_DEFAULT_SECTION, + type SitemapSectionConfig, + sectionPatterns, + sitemapSectionIssues, +} from '@ultimat3/core'; + +export type SitemapSection = SitemapSectionConfig; + +/** A page's path as a pattern reads it: no query, no fragment, no trailing slash. */ +function segmentsOf(path: string): readonly string[] { + const bare = path.replace(/[?#].*$/, '').replace(/\/+$/, ''); + return bare === '' ? [] : bare.slice(1).split('/'); +} + +function matches(pattern: string, page: readonly string[]): boolean { + const wanted = segmentsOf(pattern); + const subtree = wanted.at(-1) === '**'; + const fixed = subtree ? wanted.slice(0, -1) : wanted; + // `/blog/**` is `/blog` and everything below it: a section's own landing page files with it. + if (subtree ? page.length < fixed.length : page.length !== fixed.length) return false; + return fixed.every((segment, index) => segment === '*' || segment === page[index]); +} + +/** + * The section `path` files under: the first one, in declaration order, with a pattern that matches + * — else the default. `path` is UNPREFIXED, so a page and its locale copies share one file. + */ +export function sectionOf(path: string, sections: readonly SitemapSection[]): string { + const page = segmentsOf(path); + for (const section of sections) { + if (sectionPatterns(section).some((pattern) => matches(pattern, page))) return section.name; + } + return SITEMAP_DEFAULT_SECTION; +} + +/** Refuses what `defineConfig` refuses, for a caller that built the options by hand. */ +export function assertSections(sections: readonly SitemapSection[], pageSize: number): void { + const issues = sitemapSectionIssues(sections, pageSize); + assert( + issues.length === 0, + issues.join('; '), + "pass sections as [{ name: 'blog', match: '/blog/**' }] — a lower-case slug and a path pattern — and maxUrls from 1 to 50000", + ); +} diff --git a/packages/seo/src/sitemap-xml.ts b/packages/seo/src/sitemap-xml.ts new file mode 100644 index 000000000..89ed40080 --- /dev/null +++ b/packages/seo/src/sitemap-xml.ts @@ -0,0 +1,134 @@ +// Single responsibility: the XML of a sitemap — one ``, one `` — and the +// split of a list of URLs into files that each fit BOTH protocol caps, entries and bytes. + +import { assert } from '@ultimat3/core'; +import type { ChangeFreq } from './routes'; +import { attributes, escapeXml } from './xml'; + +export interface SitemapAlternate { + hreflang: string; + href: string; +} + +export interface SitemapUrl { + loc: string; + lastmod?: string; + changefreq?: ChangeFreq; + priority?: number; + alternates?: readonly SitemapAlternate[]; +} + +const HEAD = '\n'; +const SITEMAP_NS = ' xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"'; +const XHTML_NS = ' xmlns:xhtml="http://www.w3.org/1999/xhtml"'; +/** The envelope around the `` entries, at its largest: every byte of it is ASCII. */ +const ENVELOPE_BYTES = `${HEAD}\n\n\n`.length; + +function urlXml(url: SitemapUrl): string { + const parts = [` ${escapeXml(url.loc)}`]; + if (url.lastmod !== undefined) parts.push(` ${escapeXml(url.lastmod)}`); + if (url.changefreq !== undefined) parts.push(` ${url.changefreq}`); + if (url.priority !== undefined) { + parts.push(` ${url.priority.toFixed(1)}`); + } + for (const alternate of url.alternates ?? []) { + parts.push( + ` `, + ); + } + return ` \n${parts.join('\n')}\n `; +} + +export function urlSetXml(urls: readonly SitemapUrl[]): string { + const needsXhtml = urls.some((url) => (url.alternates?.length ?? 0) > 0); + const ns = needsXhtml ? `${SITEMAP_NS}${XHTML_NS}` : SITEMAP_NS; + return `${HEAD}\n${urls.map(urlXml).join('\n')}\n\n`; +} + +export interface SitemapIndexEntry { + /** Absolute. */ + readonly loc: string; + readonly lastmod?: string | undefined; +} + +export function indexXml(entries: readonly SitemapIndexEntry[]): string { + const body = entries + .map((entry) => { + const lastmod = + entry.lastmod === undefined ? '' : `\n ${escapeXml(entry.lastmod)}`; + return ` \n ${escapeXml(entry.loc)}${lastmod}\n `; + }) + .join('\n'); + return `${HEAD}\n${body}\n\n`; +} + +export function chunk(items: readonly T[], size: number): T[][] { + // The loop advances by `size`, so a non-positive one never moves the cursor: `maxUrls: 0` in a + // route config turned a build into an infinite loop allocating empty slices until the box ran + // out of memory. A fractional size is refused for a quieter reason — `slice` truncates it, so + // the groups silently stop being the size that was asked for. + assert( + Number.isSafeInteger(size) && size > 0, + `a chunk size must be a positive integer, got ${String(size)}: a non-positive step never advances and the loop cannot end`, + 'pass a positive integer — buildSitemap(routes, { baseUrl, maxUrls: 50000 }), the sitemaps.org bound SITEMAP_MAX_URLS already carries', + ); + const out: T[][] = []; + for (let index = 0; index < items.length; index += size) { + out.push(items.slice(index, index + size)); + } + return out; +} + +const encoder = new TextEncoder(); +const bytesOf = (text: string): number => encoder.encode(text).length; + +/** One group of at most `maxUrls`, cut again wherever its XML would pass `maxBytes`. */ +function splitByBytes(urls: readonly SitemapUrl[], maxBytes: number): SitemapUrl[][] { + const out: SitemapUrl[][] = []; + let current: SitemapUrl[] = []; + let size = ENVELOPE_BYTES; + for (const url of urls) { + const cost = bytesOf(urlXml(url)) + 1; + // A lone URL larger than the cap still gets a file: there is no smaller unit to write. + if (current.length > 0 && size + cost > maxBytes) { + out.push(current); + current = []; + size = ENVELOPE_BYTES; + } + current.push(url); + size += cost; + } + if (current.length > 0) out.push(current); + return out; +} + +export interface SitemapPageXml { + readonly urls: readonly SitemapUrl[]; + readonly xml: string; +} + +/** + * `urls` as the files that hold them: at most `maxUrls` entries and at most `maxBytes` of XML each, + * in order. No URL at all is ONE empty urlset — `/sitemap.xml` always answers a document. + */ +export function paginate( + urls: readonly SitemapUrl[], + maxUrls: number, + maxBytes: number, +): readonly SitemapPageXml[] { + if (urls.length === 0) return [{ urls, xml: urlSetXml(urls) }]; + const pages: SitemapPageXml[] = []; + for (const group of chunk(urls, maxUrls)) { + const xml = urlSetXml(group); + // UTF-8 is at most three bytes per UTF-16 unit: a file this short cannot pass the cap, and the + // common case never pays for an encode. + if (xml.length * 3 <= maxBytes || bytesOf(xml) <= maxBytes) { + pages.push({ urls: group, xml }); + continue; + } + for (const part of splitByBytes(group, maxBytes)) { + pages.push({ urls: part, xml: urlSetXml(part) }); + } + } + return pages; +} diff --git a/packages/seo/src/sitemap.ts b/packages/seo/src/sitemap.ts index 8f18fc9b6..ca2d2132b 100644 --- a/packages/seo/src/sitemap.ts +++ b/packages/seo/src/sitemap.ts @@ -1,47 +1,51 @@ -// Sitemap generation from the route table. Dynamic routes contribute the URLs -// their `prerender()` enumerates, so the sitemap can never drift from what the -// build actually produced. Splits into an index past the 50,000-URL protocol cap. +// Sitemap generation from the route table. Dynamic routes contribute the pages their `prerender()` +// enumerates, so the sitemap can never drift from what the build actually produced. One urlset — +// split into an index past a protocol cap — or, with `sections`, an index of named, paginated parts. -import { assert } from '@ultimat3/core'; -import { sitemapTooLarge } from './errors'; +import { assert, SITEMAP_DEFAULT_SECTION } from '@ultimat3/core'; +import { sitemapEntryInvalid, sitemapTooLarge } from './errors'; import { hreflangTag } from './locale-tags'; -import { type ChangeFreq, expandRoute, indexableRoutes, type RouteRecord } from './routes'; -import { absoluteUrl, attributes, escapeXml } from './xml'; +import { disallowMatcher } from './robots'; +import { expandRoutePages, indexableRoutes, type RouteRecord, type SitemapPage } from './routes'; +import { SITEMAP_DATETIME_FORMS, sitemapDatetime } from './sitemap-datetime'; +import { assertSections, type SitemapSection, sectionOf } from './sitemap-sections'; +import { + indexXml, + paginate, + type SitemapAlternate, + type SitemapPageXml, + type SitemapUrl, +} from './sitemap-xml'; +import { absoluteUrl } from './xml'; -/** Both limits are from the sitemaps.org protocol. */ +export type { SitemapAlternate, SitemapUrl } from './sitemap-xml'; +export { chunk } from './sitemap-xml'; + +/** All three limits are from the sitemaps.org protocol; the byte one is of the UNCOMPRESSED file. */ export const SITEMAP_MAX_URLS = 50_000; export const SITEMAP_INDEX_MAX_FILES = 50_000; +export const SITEMAP_MAX_BYTES = 52_428_800; /** - * Where the parts of a split sitemap live: `/sitemaps/1.xml`, `/sitemaps/2.xml`, … — one directory, - * so a running server answers every part from ONE route (`/sitemaps/:file`). `/sitemap-N.xml` needed - * a parameter inside a segment, which no router here has, and every part 404'd from the web role. + * Where the parts of a sitemap index live: `/sitemaps/1.xml`, … for a split, `/sitemaps/blog-1.xml`, + * … for a section — one directory, so a running server answers every part from ONE route + * (`/sitemaps/:file`). `/sitemap-N.xml` needed a parameter inside a segment, which no router here + * has, and every part 404'd from the web role. */ export const SITEMAP_PARTS_DIR = '/sitemaps'; -export interface SitemapAlternate { - hreflang: string; - href: string; -} - -export interface SitemapUrl { - loc: string; - lastmod?: string; - changefreq?: ChangeFreq; - priority?: number; - alternates?: readonly SitemapAlternate[]; -} - export interface SitemapFile { - /** Path relative to the site root: `/sitemap.xml`, or a part, `/sitemaps/1.xml`. */ + /** Path relative to the site root: `/sitemap.xml`, or a part, `/sitemaps/blog-1.xml`. */ path: string; xml: string; urlCount: number; + /** The newest `` inside a urlset — what the index says of it. Absent when none has one. */ + lastmod?: string; } export interface SitemapResult { readonly files: readonly SitemapFile[]; - /** Present only when the URLs did not fit in a single file. */ + /** Present when `/sitemap.xml` is an index: sections were declared, or one file did not fit. */ readonly index: SitemapFile | undefined; readonly urlCount: number; } @@ -57,8 +61,23 @@ export interface BuildSitemapOptions { * emitted at all** — there is no unprefixed URL in the sitemap for it to name. */ defaultLocale?: string; + /** `` entries per file — the page size of a section. Defaults to the protocol's 50,000. */ maxUrls?: number; + /** Bytes of XML per file. Defaults to the protocol's 50 MB; a test passes less. */ + maxBytes?: number; lastmod?: string; + /** + * Named parts. Declared, `/sitemap.xml` is ALWAYS an index — of `/sitemaps/-.xml`, the + * pages no section matches under `pages` — whatever the URL count. Omitted or empty: one urlset. + */ + sections?: readonly SitemapSection[]; + /** + * `robots.txt` `Disallow:` rules. A URL one of them matches is not listed, and leaves its + * hreflang cluster: a crawler told to stay out of a URL must not be handed it to fetch. + */ + disallow?: readonly string[]; + /** The moment a page's `lastmod` may not be later than. Omitted: when the sitemap is built. */ + now?: Date; } function localize(path: string, locale: string, options: BuildSitemapOptions): string { @@ -85,29 +104,83 @@ function defaultLocaleUrl( return localised.find((entry) => entry.locale === options.defaultLocale)?.path; } -/** Every concrete URL the route table produces, with per-locale alternates. */ -export async function sitemapUrls( +/** One page and every `` it is listed as — its locale copies, which file and move together. */ +interface Cluster { + /** The page's UNPREFIXED path: what a section matches and what a section sorts by. */ + readonly page: string; + readonly urls: readonly SitemapUrl[]; +} + +/** + * The locales `page` is listed in, in the SITEMAP's order. A name outside `locales` is refused: a + * typo (`en-us` for `en`) would otherwise drop the page from a locale without a word. + */ +function listedLocales( + route: RouteRecord, + page: SitemapPage, + locales: readonly string[], +): readonly string[] { + const wanted = page.locales ?? route.locales; + if (wanted === undefined) return locales; + const unknown = wanted.find((locale) => !locales.includes(locale)); + if (unknown !== undefined) { + throw sitemapEntryInvalid( + route.file, + page.path, + `the locale "${unknown}", which is not one of the routed locales (${locales.join(', ')})`, + ); + } + return locales.filter((locale) => wanted.includes(locale)); +} + +function lastmodOf( + route: RouteRecord, + page: SitemapPage, + options: BuildSitemapOptions, + now: Date, +): string | undefined { + if (page.lastmod === undefined) return route.lastmod ?? options.lastmod; + const lastmod = sitemapDatetime(page.lastmod, now); + if (lastmod !== undefined) return lastmod; + const written = + page.lastmod instanceof Date ? 'an invalid Date' : JSON.stringify(String(page.lastmod)); + throw sitemapEntryInvalid( + route.file, + page.path, + `the lastmod ${written}, which is not ${SITEMAP_DATETIME_FORMS}`, + ); +} + +async function sitemapClusters( routes: readonly RouteRecord[], options: BuildSitemapOptions, -): Promise { - const urls: SitemapUrl[] = []; +): Promise { + const clusters: Cluster[] = []; const locales = options.locales ?? []; + const disallowed = disallowMatcher(options.disallow ?? []); + const now = options.now ?? new Date(); for (const route of indexableRoutes(routes)) { - for (const path of await expandRoute(route)) { + for (const page of await expandRoutePages(route)) { + const path = page.path; // Localised once, then read three times — the alternates, the `x-default` candidate and the // ``s below are three questions with one answer, and they drifted apart when each - // computed its own. - const localised = locales.map((locale) => ({ - locale, - path: localize(path, locale, options), - })); - const alternates: SitemapAlternate[] = localised.map((entry) => ({ - // BCP 47 region form (`es-CO`), the spelling `renderMeta` puts in the head: one cluster, - // one vocabulary, whether a crawler reads the page or the sitemap. - hreflang: hreflangTag(entry.locale), - href: absoluteUrl(options.baseUrl, entry.path), - })); + // computed its own. Trimmed HERE, to the locales the page is listed in and the URLs a + // crawler may fetch, so all three read the same trimmed list. + const localised = listedLocales(route, page, locales) + .map((locale) => ({ locale, path: localize(path, locale, options) })) + .filter((entry) => !disallowed(entry.path)); + // A page left in ONE locale of several has no alternate to name: the cluster is the page. + // (A sitemap built for one locale keeps its cluster of one — it always wrote it.) + const alone = localised.length === 1 && locales.length > 1; + const alternates: SitemapAlternate[] = alone + ? [] + : localised.map((entry) => ({ + // BCP 47 region form (`es-CO`), the spelling `renderMeta` puts in the head: one cluster, + // one vocabulary, whether a crawler reads the page or the sitemap. + hreflang: hreflangTag(entry.locale), + href: absoluteUrl(options.baseUrl, entry.path), + })); // `x-default` only when it names a URL THIS sitemap lists. It was the bare `path` for every // route: with `locales` set and no `defaultLocale`, every `` is prefixed and the // unprefixed path is one the sitemap never mentions — an hreflang cluster pointing at a URL @@ -119,77 +192,81 @@ export async function sitemapUrls( alternates.push({ hreflang: 'x-default', href: absoluteUrl(options.baseUrl, fallback) }); } - const lastmod = route.lastmod ?? options.lastmod; - const emitFor = locales.length === 0 ? [path] : localised.map((entry) => entry.path); - for (const emitted of emitFor) { - urls.push({ - loc: absoluteUrl(options.baseUrl, emitted), - ...(lastmod === undefined ? {} : { lastmod }), - ...(route.changefreq === undefined ? {} : { changefreq: route.changefreq }), - ...(route.priority === undefined ? {} : { priority: route.priority }), - ...(alternates.length === 0 ? {} : { alternates }), - }); - } + const lastmod = lastmodOf(route, page, options, now); + // No locales at all: the page is its one URL — unless it asked to be listed nowhere. + const nowhere = (page.locales ?? route.locales)?.length === 0; + const unlocalised = nowhere || disallowed(path) ? [] : [path]; + const emitFor = locales.length === 0 ? unlocalised : localised.map((entry) => entry.path); + const urls = emitFor.map((emitted) => ({ + loc: absoluteUrl(options.baseUrl, emitted), + ...(lastmod === undefined ? {} : { lastmod }), + ...(route.changefreq === undefined ? {} : { changefreq: route.changefreq }), + ...(route.priority === undefined ? {} : { priority: route.priority }), + ...(alternates.length === 0 ? {} : { alternates }), + })); + if (urls.length > 0) clusters.push({ page: path, urls }); } } - return urls; + return clusters; } -function renderUrlSet(urls: readonly SitemapUrl[]): string { - const needsXhtml = urls.some((url) => (url.alternates?.length ?? 0) > 0); - const ns = needsXhtml - ? ' xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:xhtml="http://www.w3.org/1999/xhtml"' - : ' xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"'; - const body = urls - .map((url) => { - const parts = [` ${escapeXml(url.loc)}`]; - if (url.lastmod !== undefined) parts.push(` ${escapeXml(url.lastmod)}`); - if (url.changefreq !== undefined) { - parts.push(` ${url.changefreq}`); - } - if (url.priority !== undefined) { - parts.push(` ${url.priority.toFixed(1)}`); - } - for (const alternate of url.alternates ?? []) { - parts.push( - ` `, - ); - } - return ` \n${parts.join('\n')}\n `; - }) - .join('\n'); - return `\n\n${body}\n\n`; +/** Every concrete URL the route table produces, with per-locale alternates. */ +export async function sitemapUrls( + routes: readonly RouteRecord[], + options: BuildSitemapOptions, +): Promise { + return (await sitemapClusters(routes, options)).flatMap((cluster) => cluster.urls); } -function renderIndex(files: readonly SitemapFile[], options: BuildSitemapOptions): string { - const body = files - .map((file) => { - const loc = escapeXml(absoluteUrl(options.baseUrl, file.path)); - const lastmod = - options.lastmod === undefined - ? '' - : `\n ${escapeXml(options.lastmod)}`; - return ` \n ${loc}${lastmod}\n `; - }) - .join('\n'); - return `\n\n${body}\n\n`; +/** The newest `` among `urls`, as written — compared as instants, not as text. */ +function newestLastmod(urls: readonly SitemapUrl[]): string | undefined { + let newest: string | undefined; + let at = Number.NEGATIVE_INFINITY; + for (const url of urls) { + const time = url.lastmod === undefined ? Number.NaN : Date.parse(url.lastmod); + if (Number.isFinite(time) && time > at) { + at = time; + newest = url.lastmod; + } + } + return newest; } -export function chunk(items: readonly T[], size: number): T[][] { - // The loop advances by `size`, so a non-positive one never moves the cursor: `maxUrls: 0` in a - // route config turned a build into an infinite loop allocating empty slices until the box ran - // out of memory. A fractional size is refused for a quieter reason — `slice` truncates it, so - // the groups silently stop being the size that was asked for. - assert( - Number.isSafeInteger(size) && size > 0, - `a chunk size must be a positive integer, got ${String(size)}: a non-positive step never advances and the loop cannot end`, - 'pass a positive integer — buildSitemap(routes, { baseUrl, maxUrls: 50000 }), the sitemaps.org bound SITEMAP_MAX_URLS already carries', - ); - const out: T[][] = []; - for (let index = 0; index < items.length; index += size) { - out.push(items.slice(index, index + size)); - } - return out; +const fileOf = (path: string, page: SitemapPageXml): SitemapFile => { + const lastmod = newestLastmod(page.urls); + return { + path, + xml: page.xml, + urlCount: page.urls.length, + ...(lastmod === undefined ? {} : { lastmod }), + }; +}; + +const comparePages = (a: Cluster, b: Cluster): number => + a.page < b.page ? -1 : a.page > b.page ? 1 : 0; + +/** + * The files of a sectioned sitemap, the default section first and the rest in declaration order. + * Inside a section pages are SORTED by path: the bytes then depend on which pages exist and never + * on the order a query happened to return them in, so a crawler re-reading a page of a section + * diffs it against the same neighbours. A section with nothing in it writes no file. + */ +function sectionFiles( + clusters: readonly Cluster[], + sections: readonly SitemapSection[], + maxUrls: number, + maxBytes: number, +): readonly SitemapFile[] { + const names = [SITEMAP_DEFAULT_SECTION, ...sections.map((section) => section.name)]; + const filed = new Map(names.map((name) => [name, []])); + for (const cluster of clusters) filed.get(sectionOf(cluster.page, sections))?.push(cluster); + return names.flatMap((name) => { + const urls = (filed.get(name) ?? []).sort(comparePages).flatMap((cluster) => cluster.urls); + if (urls.length === 0) return []; + return paginate(urls, maxUrls, maxBytes).map((page, position) => + fileOf(`${SITEMAP_PARTS_DIR}/${name}-${String(position + 1)}.xml`, page), + ); + }); } export async function buildSitemap( @@ -197,6 +274,7 @@ export async function buildSitemap( options: BuildSitemapOptions, ): Promise { const maxUrls = options.maxUrls ?? SITEMAP_MAX_URLS; + const maxBytes = options.maxBytes ?? SITEMAP_MAX_BYTES; // Refused here and not only in `chunk`, so the answer does not depend on how many URLs the site // happens to have today: `maxUrls: 2.5` is a typo whether or not this build has enough routes to // reach the split, exactly as a metric refuses `maxSeries: 1.5` at declaration. @@ -205,34 +283,62 @@ export async function buildSitemap( `buildSitemap({ maxUrls }) must be a positive integer, got ${String(maxUrls)}`, 'pass a positive integer — buildSitemap(routes, { baseUrl, maxUrls: 50000 }) — or omit it and take SITEMAP_MAX_URLS, the sitemaps.org bound', ); - const urls = await sitemapUrls(routes, options); + assert( + Number.isSafeInteger(maxBytes) && maxBytes > 0, + `buildSitemap({ maxBytes }) must be a positive integer, got ${String(maxBytes)}`, + 'pass a positive integer — buildSitemap(routes, { baseUrl, maxBytes: 52428800 }) — or omit it and take SITEMAP_MAX_BYTES, the sitemaps.org bound', + ); + const sections = options.sections ?? []; + if (sections.length > 0) assertSections(sections, Math.min(maxUrls, SITEMAP_MAX_URLS)); + + const clusters = await sitemapClusters(routes, options); + const urlCount = clusters.reduce((count, cluster) => count + cluster.urls.length, 0); + // Counted before a single file is rendered: a site this far past the index cap would otherwise + // build tens of thousands of documents on the way to a refusal. + if (Math.ceil(urlCount / maxUrls) > SITEMAP_INDEX_MAX_FILES) { + throw sitemapTooLarge(Math.ceil(urlCount / maxUrls), SITEMAP_INDEX_MAX_FILES); + } - if (urls.length <= maxUrls) { + if (sections.length > 0 && urlCount > 0) { + const files = sectionFiles(clusters, sections, maxUrls, maxBytes); + if (files.length > SITEMAP_INDEX_MAX_FILES) { + throw sitemapTooLarge(files.length, SITEMAP_INDEX_MAX_FILES); + } + const entries = files.map((file) => ({ + loc: absoluteUrl(options.baseUrl, file.path), + lastmod: file.lastmod, + })); return { - files: [{ path: '/sitemap.xml', xml: renderUrlSet(urls), urlCount: urls.length }], - index: undefined, - urlCount: urls.length, + files, + index: { path: '/sitemap.xml', xml: indexXml(entries), urlCount: files.length }, + urlCount, }; } - const groups = chunk(urls, maxUrls); - if (groups.length > SITEMAP_INDEX_MAX_FILES) { - throw sitemapTooLarge(groups.length, SITEMAP_INDEX_MAX_FILES); + const pages = paginate( + clusters.flatMap((cluster) => cluster.urls), + maxUrls, + maxBytes, + ); + const [only] = pages; + if (pages.length === 1 && only !== undefined) { + return { files: [fileOf('/sitemap.xml', only)], index: undefined, urlCount }; } - - const files: SitemapFile[] = groups.map((group, position) => ({ - path: `${SITEMAP_PARTS_DIR}/${position + 1}.xml`, - xml: renderUrlSet(group), - urlCount: group.length, + if (pages.length > SITEMAP_INDEX_MAX_FILES) { + throw sitemapTooLarge(pages.length, SITEMAP_INDEX_MAX_FILES); + } + const files = pages.map((page, position) => + fileOf(`${SITEMAP_PARTS_DIR}/${String(position + 1)}.xml`, page), + ); + // An unsectioned split says of every part what the caller said of the build (`options.lastmod`): + // the bytes it always wrote. Sections are where a part's own newest date is read. + const entries = files.map((file) => ({ + loc: absoluteUrl(options.baseUrl, file.path), + lastmod: options.lastmod, })); - return { files, - index: { - path: '/sitemap.xml', - xml: renderIndex(files, options), - urlCount: files.length, - }, - urlCount: urls.length, + index: { path: '/sitemap.xml', xml: indexXml(entries), urlCount: files.length }, + urlCount, }; } diff --git a/wiki/Configuration.md b/wiki/Configuration.md index 20ef6a0e9..e9793664c 100644 --- a/wiki/Configuration.md +++ b/wiki/Configuration.md @@ -280,6 +280,8 @@ seo: { | `seo.robots.disallow` | `string[]` | `[]` | each starts with `/`. Added to **every** production group — a crawler obeys only the group that names it — with a `User-agent: *` group emitted when none is declared. Any other environment still emits `Disallow: /` alone | | `seo.sitemap.extra` | `string[]` | `[]` | public pages outside `site/` to list — a path (`/verificar`) answered by an `app/` route with no `policy`; listed per routed locale with hreflang alternates like a `site/` page. No origin, query or fragment (`X_CONFIG_INVALID`); a path no ungated `app/` route answers is `X_SITEMAP_EXTRA_INVALID` when the sitemap is built. `As of 22.10` | | `seo.sitemap.lastmod` | `'none' \| 'git' \| 'mtime' \| 'build'` | `'none'` | each ``: `'git'` the last commit touching the route's source file (its mtime where there is no work tree — a container image), `'mtime'` the file's mtime, `'build'` one timestamp for every URL. Read once per file per process. `As of 22.10` | +| `seo.sitemap.sections` | `{ name, match }[]` | `[]` | any declared: `/sitemap.xml` is an index of `/sitemaps/-.xml`, pages no section matches under the reserved `pages`. `name` a lower-case slug, unique; `match` a path pattern or a list (`/precios`, `/blog/*`, `/blog/**`), read against the path without its locale prefix. Anything else is `X_CONFIG_INVALID`. [SEO](SEO#sections-an-index-by-configuration). `As of 2026-10` | +| `seo.sitemap.pageSize` | `number` | `50000` | `` entries per sitemap file, 1–50,000 (the sitemaps.org cap). A file past 50 MB splits further whatever this says. `As of 2026-10` | ## `pwa` @@ -408,7 +410,7 @@ An **embedder** that builds its own server — `httpServer({ routes, config: def | Call | Options | Notes | |---|---|---| | `buildRobots(config)` | `baseUrl`, `environment?`, `groups?`, `disallow?`, `sitemaps?`, `extra?` | **fail-closed**: only the exact string `production` opts a deploy into indexing, so staging, a laptop, a typo and an unset `ULTIMATE_ENV` all emit `Disallow: /` — a branch deploy that gets indexed outranks and cannibalises the real site. `environment` omitted resolves from `ULTIMATE_ENV`, and an unreadable one falls back to core's default rather than 500ing a `robots.txt` | -| `buildSitemap(routes, options)` | `baseUrl`, `locales?`, `localizePath?`, `defaultLocale?`, `maxUrls?`, `lastmod?` | splits past `SITEMAP_MAX_URLS` (50,000) into the index `/sitemap.xml` and parts `/sitemaps/.xml` (`SITEMAP_PARTS_DIR`), which the web role serves from `GET /sitemaps/:file`. `maxUrls` must be a positive integer — `0` never advances the chunk cursor and used to allocate empty slices until the box ran out of memory | +| `buildSitemap(routes, options)` | `baseUrl`, `locales?`, `localizePath?`, `defaultLocale?`, `maxUrls?`, `maxBytes?`, `lastmod?`, `sections?`, `disallow?`, `now?` | splits past `SITEMAP_MAX_URLS` (50,000) or `SITEMAP_MAX_BYTES` (50 MB) into the index `/sitemap.xml` and parts `/sitemaps/.xml` (`SITEMAP_PARTS_DIR`), which the web role serves from `GET /sitemaps/:file`; with `sections`, always an index of `/sitemaps/-.xml`. `maxUrls` must be a positive integer — `0` never advances the chunk cursor and used to allocate empty slices until the box ran out of memory | `baseUrl` is the argument every one of them takes, so the canonical origin stays an env key the app reads (`APP_URL`) and never a config field. There is no `seo.lighthouse` gate and no `seo.ogImage` renderer `As of 2026-08-22`. diff --git a/wiki/Error-Codes.md b/wiki/Error-Codes.md index c66e93f3e..7850c4a72 100644 --- a/wiki/Error-Codes.md +++ b/wiki/Error-Codes.md @@ -555,6 +555,7 @@ No secret and no signature reaches a `cause`, a `fix` or a `meta`, and a URL is | `X_SITEMAP_TOO_LARGE` | sitemap exceeds the 50,000-entry limit | too many prerendered URLs in one file | enable sitemap index splitting in `app.config.ts` | | `X_IMAGE_QUERY_INVALID` | a minted image URL's `?w=`/`?q=` value is present but unusable | `?w=0`, `?w=-5`, `?q=150` on a URL `responsiveImage()` minted | request a positive integer, e.g. `?w=640` (quality is `1`-`100`, e.g. `?q=75`) | | `X_SEO_LINK_INVALID` | a route meta link is unsafe or would fetch nothing | a javascript:/data: href, a preload with no as, or a font preload with no crossorigin | edit links in the route's defineRoute({ meta }): an http(s) or relative href, as on every preload, crossorigin on a font preload | +| `X_SITEMAP_ENTRY_INVALID` | a sitemap entry carries a lastmod or a locale the sitemap cannot write | a prerender() entry (or a page handed to buildSitemap) has a lastmod that is not an ISO 8601 date with an explicit zone, or names a locale the app does not route | `x verify --json` after `prerender()` in the route file named in cause answers `lastmod` as a `Date` or an ISO 8601 string with a zone (`row.updatedAt`) and only routed locales in `locales` | ## PWA and build skew diff --git a/wiki/Routes-And-Render-Modes.md b/wiki/Routes-And-Render-Modes.md index dff9732a1..9985fd82d 100644 --- a/wiki/Routes-And-Render-Modes.md +++ b/wiki/Routes-And-Render-Modes.md @@ -363,7 +363,7 @@ A missing required property is a **type error**, not a Rich Results Test failure | Artifact | Derived from | Notes | |---|---|---| -| `sitemap.xml` | all indexable routes + `prerender()` results | one `` per page per locale with `xhtml:link` alternates; `lastmod` from the entity's `updatedAt`; auto-split at 50k URLs into a sitemap index | +| `sitemap.xml` | all indexable routes + `prerender()` results | one `` per page per locale with `xhtml:link` alternates; `lastmod` and the locales a page is listed in from its `prerender()` entry (`{ params, lastmod: row.updatedAt, locales }`); an index of named sections by `seo.sitemap.sections`, else auto-split at 50k URLs — [SEO](SEO#robotstxt-and-sitemapxml) | | `robots.txt` | `seo.robots.disallow` in `app.config.ts` | sitemap reference included; anything but `ULTIMATE_ENV=production` is `Disallow: /` | | `rss.xml` / `atom.xml` / `feed.json` | routes tagged as feed items | one declaration, three formats | | `llms.txt` | `site/` route titles + descriptions | machine-readable site summary for agents | diff --git a/wiki/SEO.md b/wiki/SEO.md index 01586f374..211d7a6e5 100644 --- a/wiki/SEO.md +++ b/wiki/SEO.md @@ -30,6 +30,7 @@ export const config = defineRoute({ | `X_SEO_CANONICAL_MISMATCH` | `meta.canonical` does not resolve to the route's own URL | | `X_LD_INVALID` | a JSON-LD node missing a required schema.org field — and a required field is required in the builder's **input type**, so `ld.Article` without `datePublished` does not compile. A CMS `null` in one is the same coded refusal | | `X_SITEMAP_TOO_LARGE` | the sitemap index past 50,000 files | +| `X_SITEMAP_ENTRY_INVALID` | a page's `lastmod` that is not a `Date` or ISO 8601 with a zone, or a `locales` name the app does not route — names the route file | | `X_SEO_LINK_INVALID` | a `meta.links` entry whose href is not a path or an `http(s)` URL (`javascript:`, `data:` …), a `preload` with no `as`, or a font preload with no `crossorigin`. Thrown while the page renders (SSR and prerender) and reported at the route file by the `seo` step | `x verify`'s `seo` step runs `validateMeta` over every `site/` route: missing, duplicate and @@ -71,7 +72,7 @@ a link's `href` is part of its identity when the head is deduped. | Call | Answers | |---|---| | `renderMeta()` | the head tags: title (a `$` in a title is kept verbatim), canonical, robots, `og:*`, `twitter:*`, hreflang + `x-default`, `theme-color` per scheme, `links` | -| `buildSitemap()` | from the route table and each route's `prerender()`, per-locale alternates, split into an index past 50k | +| `buildSitemap()` | from the route table and each route's `prerender()`, per-locale alternates; one urlset, an index of named `sections`, or a split past a cap (50,000 URLs, 50 MB) | | `buildRobots()` | **fail-closed**: only the literal `production` environment opts into indexing; anything else — staging, a laptop, an unset variable — is `Disallow: /` with no sitemap line. In production `disallow` is added to **every** group, and a `User-agent: *` group is emitted to carry it when none is declared | | `buildFeed()` | RSS 2.0, Atom and JSON Feed from one item list. The channel's `author`, `copyright` and `icon` reach Atom as ``, `` and ``; an item's `author` is `` in Atom and, in RSS, `` with an email or `` without one; an item's `image` is an Atom enclosure link and an RSS ``. An item date must be ISO-8601 with an offset or `Z`; one that is not is treated as absent (and an offsetless one is logged as `seo.feed.date_offsetless`), never read through the server's zone | | `builtinImageDriver({ read })` | resize and blur placeholder over core's pipeline — decodes `png`, `jpeg`, `webp`, `gif`; encodes `png`, `jpeg`, `webp`. `avif` needs a CDN driver (`X_IMAGE_UNSUPPORTED`), so `responsiveImage()` offers `webp` only unless given `formats`, and no width past 8192 (`MAX_IMAGE_WIDTH`). `As of 2026-10` | @@ -82,13 +83,98 @@ Every `X_SEO_*` and image code is in [Error codes](Error-Codes). ## `robots.txt` and `sitemap.xml` One answer, served two ways. `siteSeo()` from `@ultimat3/cli` builds both from the route table. It -takes the public `site/` routes (no `policy`), leaves out any page whose `meta` says -`robots: { index: false }`, and expands a dynamic route through its `prerender()`. +takes the public `site/` routes (no `policy`) and expands a dynamic route through its `prerender()`. | Where | How | |---|---| -| static export | the scaffolded `apps/web/prerender.ts` writes `siteSeo()`'s files into `.x/static`. A dynamic route lists exactly the pages the build emitted | -| web role (`x dev`, `runRole`) | `GET /robots.txt`, `GET /sitemap.xml` and its parts `GET /sitemaps/:file`, `public, max-age=3600`, built once per origin and kept for that same hour (every file shares one answer; a failed build is not kept). Absolute against `APP_URL`, else `SITE_ORIGIN`, else the request's own origin. `As of 2026-09-25` (22.2.2) | +| static export | the scaffolded `apps/web/prerender.ts` writes `siteSeo()`'s files into `.x/static` (`writeSiteSeo`). A dynamic route lists exactly the pages the build emitted | +| web role (`x dev`, `runRole`) | `GET /robots.txt`, `GET /sitemap.xml` and its parts `GET /sitemaps/:file`, `application/xml`, `public, max-age=3600`, built once per origin and kept for that same hour. Every file shares one answer: a refresh that fails or hangs keeps the previous index **and** its parts. Absolute against `APP_URL`, else `SITE_ORIGIN`, else the request's own origin | + +### What is left out + +| Left out | Decided by | +|---|---| +| a page whose `meta` says `robots: { index: false }` | the route's own `meta`, read **once per routed locale**: `noindex` in English only lists the other locales and drops English from their hreflang cluster. Read for a route with no params and no `load` — the only `meta` that resolves without a request | +| a page of a dynamic route, or of a route with `load` | its `prerender()` entry: `locales: ['es-co']` lists it there only, `locales: []` nowhere. Its `meta` is **not** read — that would be one `load` per page per refresh | +| a URL `seo.robots.disallow` matches | the rule, read as a crawler reads it (prefix, `*`, trailing `$`). A locale copy it blocks leaves the cluster. `As of 2026-10`: such a URL used to be listed — "submitted URL blocked by robots.txt" | +| a gated route, an `app/` page not in `extra` | `policy`, `seo.sitemap.extra` | + +### A page's own `lastmod` and locales + +`prerender()` answers a bare string, a params object, or an **entry** — the three mix freely: + +```ts +import { defineRoute } from '@ultimat3/render'; + +declare const posts: { + published(): Promise; +}; + +export const config = defineRoute({ + render: 'isr', + revalidate: { ttl: '10m' }, + offline: 'network-only', + prerender: async () => + (await posts.published()).map((post) => ({ + params: { slug: post.slug }, + lastmod: post.updatedAt, // a Date, or ISO 8601 with a zone + locales: post.englishReviewed ? ['es-co', 'en'] : ['es-co'], + })), + meta: ({ params }) => ({ title: params['slug'] ?? '', description: 'A note from the blog.' }), +}); +``` + +| Key | Rule | +|---|---| +| `params` | the page. The build renders it in **every** routed locale whatever `locales` says: `locales` is about listing, not about which documents exist | +| `lastmod` | written as W3C Datetime in UTC (`2026-10-01T15:04:05.000Z`). Wins over `seo.sitemap.lastmod`. A time with no zone (`2026-10-01T10:00:00`) or free text is `X_SITEMAP_ENTRY_INVALID`, never read in the server's zone. A moment in the future is **clamped to now**: refusing it would fail the whole sitemap for one row stamped by a clock a second ahead, and writing it teaches a crawler to distrust every date in the file | +| `locales` | one `` per named locale, hreflang among those only; `x-default` only when the default locale is one of them; a page left in one locale carries no alternates. Omitted: every routed locale. A name the app does not route is `X_SITEMAP_ENTRY_INVALID` | + +A route with **no params** may answer `prerender()` too — `() => [{ params: {}, lastmod, locales }]` +— which is how a page with a `load` (a blog index whose date is its newest post's) says either. + +A key that is none of the three (`lastMod`) is `X_PRERENDER_FAILED`. The static export still lists +only the pages it emitted; it asks `prerender()` again for what each says of itself. + +The `` is separate: a page's hreflang tags come from `meta.alternates`, or from every routed +locale when it declares none. A page listed in one locale declares its own: +`alternates: [{ hreflang: 'es-CO', href: url }]`. + +### Sections: an index by configuration + +```ts +import { defineConfig } from '@ultimat3/core'; + +export const config = defineConfig({ + name: 'notificado', + seo: { + sitemap: { + pageSize: 1000, + sections: [ + { name: 'blog', match: '/blog/**' }, + { name: 'ciudades', match: ['/juzgados/**', '/ciudades/*'] }, + ], + }, + }, +}); +``` + +`/sitemap.xml` becomes a ``; every listed page goes to `/sitemaps/-.xml`. + +| Rule | | +|---|---| +| match | against the page's path **without** its locale prefix, so `/en/blog/a` files with `/blog/a` and a cluster stays in one section. `/precios` that page · `/blog/*` one segment below · `/blog/**` `/blog` and everything below. First section that matches wins, in declaration order | +| unmatched | the default section, `pages` — a reserved name | +| order | `pages` first, then sections as declared; inside one, pages sorted by path, so the bytes depend on which pages exist and never on the order a query returned them | +| pagination | `pageSize` URLs per file (default and ceiling 50,000); a file past 50 MB splits further. Numbered from 1 even when there is one: `blog-1.xml` keeps its URL when `blog-2.xml` appears | +| empty | a section with no page writes no file and is not in the index; a site with no page at all answers one empty urlset | +| index `` | per child, the newest `` inside it; none when no URL in it has one | +| `robots.txt` | names `/sitemap.xml` alone | +| refused at boot | a `name` that is not a lower-case slug (`a-z`, `0-9`, single hyphens), a duplicate, `pages`, a `match` that is not a path pattern, a `pageSize` outside 1–50,000 — `X_CONFIG_INVALID` | + +With no `sections` the output is byte-for-byte what it was: one urlset at `/sitemap.xml`, and past +a cap (50,000 URLs, or 50 MB) the index with parts `/sitemaps/1.xml`, `/sitemaps/2.xml`, … +(`SITEMAP_PARTS_DIR`). A part the index does not name is a 404, `no-store`. `As of 2026-10`. **Beyond `site/`, and ``** (`As of 22.10`) — `seo.sitemap` in `app.config.ts`: @@ -104,7 +190,5 @@ seo: { sitemap: { extra: ['/verificar', '/estado'], lastmod: 'git' } }, The scaffolded `prerender.ts` passes `sitemap` and `root` from `loadSiteSettings(root)`; an app with its own `prerender.ts` adds `sitemap: settings.sitemap, root` to its `siteSeo({ … })` call. -Past 50,000 URLs, `/sitemap.xml` is the index and its parts are `/sitemaps/1.xml`, -`/sitemaps/2.xml`, … (`SITEMAP_PARTS_DIR`). The static export writes them and the web role serves -them (`GET /sitemaps/:file`, from the same answer as the index); a part the index does not name is a -404. `As of 2026-10`: the parts were `/sitemap-N.xml` before, which no running process could serve. +The parts of a split were `/sitemap-N.xml` once, which no running process could serve +([Upgrading](Upgrading)).