feat(seo): a sitemap index with named, paginated sections; lastmod and locales per prerendered page

CHANGELOG.md

@@ -8,7 +8,22 @@ Semver applies from 1.0.0. A breaking change to a documented API needs a major 
 
 ## [Unreleased]
 
-Nothing yet.
+### Added
+
+Tier 0 — core. Tier 1 — seo. Tier 4 — render. Tier 5 — cli.
+
+- **`seo.sitemap.sections` — a sitemap index with named, paginated sections, by configuration.** `/sitemap.xml` was one `<urlset>` until a site passed 50,000 URLs. `seo: { sitemap: { pageSize: 1000, sections: [{ name: 'blog', match: '/blog/**' }, { name: 'ciudades', match: '/juzgados/**' }] } }` makes it a `<sitemapindex>` of `/sitemaps/pages-1.xml`, `/sitemaps/blog-1.xml`, `/sitemaps/blog-2.xml`, …, each a valid urlset; a page no section matches goes to the reserved `pages`. `match` is a path pattern or a list (`/precios`, `/blog/*`, `/blog/**`), read against the path without its locale prefix, so a page and its locale copies share a file; the first section that matches wins. Inside a section pages are sorted by path — the bytes depend on which pages exist, not on the order a query returned them — files are numbered from 1, a section with nothing in it writes no file, and the index `<lastmod>` of a child is the newest `<lastmod>` inside it. The web role serves every child from the route it already had (`GET /sitemaps/:file`, `application/xml`, `public, max-age=3600`, from the same hourly answer as the index — a refresh that fails keeps the previous index and its children together), the static export writes the same files, and `robots.txt` still names `/sitemap.xml` alone. A `name` that is not a lower-case slug, a duplicate, `pages`, a `match` that is not a path pattern or a `pageSize` outside 1–50,000 is `X_CONFIG_INVALID` at boot. **With no `sections` the output is byte-for-byte what it was** (`sitemap-sections.test.ts` pins the hash). [SEO → Sections](https://github.com/developerz-ai/ultimate/wiki/SEO#sections-an-index-by-configuration).
+- **`prerender()` may say when a page changed and which locales list it.** Beside a bare string and a params object, an item may be an entry: `{ params: { slug }, lastmod: row.updatedAt, locales: ['es-co'] }` (`PrerenderEntry`; the three forms mix). `lastmod` — a `Date`, or ISO 8601 with an explicit zone — is that page's sitemap `<lastmod>`, written as W3C Datetime in UTC and winning over `seo.sitemap.lastmod`. A time with no zone or free text is the new `X_SITEMAP_ENTRY_INVALID`, naming the route file, never a date read in the server's zone; a moment in the future is clamped to now (refusing it would fail the whole sitemap for one row from a clock a second ahead; writing it is a date a crawler stops trusting). `locales` lists the page — one `<url>` each, hreflang among those only, `x-default` only when the default locale is one of them — in the named locales; `[]` lists it nowhere; a name the app does not route is `X_SITEMAP_ENTRY_INVALID`. A route with no params may answer one entry (`[{ params: {}, lastmod, locales }]`), which is how a page with a `load` says either. The BUILD is unchanged: every routed locale's document is still rendered, and `enumeratePrerender()` still answers params. An entry key outside the three, a `lastmod` that is neither a string nor a `Date`, or `locales` that is not a list of names is `X_PRERENDER_FAILED`. The static export still lists only the pages it emitted (`pagesFor`) and now asks `prerender()` once more for what each says of itself.
+- **The scaffolded `apps/web/prerender.ts` writes the SEO files through `writeSiteSeo(out, seo)`.** Nothing to change in an existing app: a `prerender.ts` that loops over `seo.sitemaps` itself already writes an index and its parts.
+- New names: `SitemapSectionConfig`, `SITEMAP_DEFAULT_SECTION`, `SITEMAP_PAGE_SIZE_MAX`, `sectionPatterns`, `sitemapSectionIssues` (`@ultimat3/core`; `SeoSitemapConfig` gains `sections` and `pageSize`); `SitemapPage`, `SitemapSection`, `expandRoutePages`, `sectionOf`, `sitemapDatetime`, `disallowMatcher`, `SITEMAP_MAX_BYTES`, `sitemapEntryInvalid`, `RouteRecord.locales`, `SitemapFile.lastmod`, and `sections` / `maxBytes` / `disallow` / `now` on `buildSitemap` (`@ultimat3/seo`); `PrerenderEntry`, `PrerenderItem` (`@ultimat3/render`) and `enumeratePrerenderEntries` (`@ultimat3/render/server`); `writeSiteSeo`, `SiteSeoSitemap` (`@ultimat3/cli`). `PrerenderFn`'s return type is wider; code that only declares a `prerender` is unaffected, code that reads one's result itself should call `enumeratePrerender`.
+
+### Fixed
+
+Tier 1 — seo. Tier 5 — cli.
+
+- **A page that is `noindex` in one locale is no longer listed — or cross-linked — in that locale.** The sitemap read each route's `meta` once, in no locale, and emitted one `<url>` per routed locale: a page whose `meta` answers `robots: { index: false }` for English only was listed at `/en/…`, and every other copy's hreflang pointed at it ("alternate page with noindex"). `meta` is now read once per routed locale; the page is listed in the locales that may be indexed and its cluster names only those. As before, this reads a route with no params and no `load` — a dynamic page says it with `locales` on its `prerender()` entry.
+- **A URL `seo.robots.disallow` blocks is no longer in the sitemap.** `robots.txt` and the sitemap are written from one config, and the sitemap listed URLs the other told crawlers not to fetch ("submitted URL blocked by robots.txt"). A rule is read as a crawler reads it — a prefix, `*`, a trailing `$`; a locale copy it blocks leaves its hreflang cluster. An app whose sitemap shrinks on upgrade was listing a blocked page: narrow the rule to keep it.
+- **A sitemap file past 50 MB is split.** Only the 50,000-URL cap was enforced; a site with many locales (every `<url>` carries the whole hreflang cluster) could pass the protocol's 50 MB uncompressed limit below it, which a crawler answers with "sitemap could not be read". Sections and the unsectioned split both cut on bytes now (`SITEMAP_MAX_BYTES`).
 
 ## 27.8.1 - 2026-10-11
 

packages/cli/src/app-config-load.test.ts

@@ -52,7 +52,7 @@ export const config = {
   drain: { readinessGraceMs: 7000 },
   health: { readiness: 'process' },
   site: { origin: 'https://every.example' },
-  seo: { robots: { disallow: ['/panel'] }, sitemap: { extra: ['/about'], lastmod: 'build' } },
+  seo: { robots: { disallow: ['/panel'] }, sitemap: { extra: ['/about'], lastmod: 'build', sections: [{ name: 'blog', match: '/blog/**' }], pageSize: 1000 } },
   navigation: { client: ['app'], speculation: { prefetch: 'conservative', exclude: ['/x/*'] } },
   islands: { sharedChunks: true },
 };
@@ -95,7 +95,12 @@ describe('loadAppConfig', () => {
     expect(config?.health.readiness).toBe('process');
     expect(config?.site.origin).toBe('https://every.example');
     expect(config?.seo.robots.disallow).toEqual(['/panel']);
-    expect(config?.seo.sitemap).toEqual({ extra: ['/about'], lastmod: 'build' });
+    expect(config?.seo.sitemap).toEqual({
+      extra: ['/about'],
+      lastmod: 'build',
+      sections: [{ name: 'blog', match: '/blog/**' }],
+      pageSize: 1000,
+    });
     expect(config?.navigation.client).toEqual(['app']);
     expect(config?.navigation.speculation).toEqual({
       prefetch: 'conservative',
@@ -127,7 +132,12 @@ describe('loadAppConfig', () => {
     expect(config?.health.readiness).toBe('dependencies');
     expect(config?.site.origin).toBeNull();
     expect(config?.seo.robots.disallow).toEqual([]);
-    expect(config?.seo.sitemap).toEqual({ extra: [], lastmod: 'none' });
+    expect(config?.seo.sitemap).toEqual({
+      extra: [],
+      lastmod: 'none',
+      sections: [],
+      pageSize: 50_000,
+    });
     expect(config?.navigation.client).toEqual([]);
     expect(config?.navigation.speculation).toEqual(DEFAULT_SPECULATION);
     expect(config?.islands.sharedChunks).toBe(false);

packages/cli/src/cmd-new.test.ts

@@ -362,7 +362,9 @@ describe('unit · x new · the sitemap and robots.txt it wires', () => {
     const entry = emitted('apps/web/prerender.ts', true);
     expect(entry).toContain("from '@ultimat3/cli'");
     expect(entry).toContain('siteSeo(');
-    expect(entry).toContain("'robots.txt'");
+    // `writeSiteSeo` puts every file of that answer on disk: the sitemap, each part of an index
+    // (`/sitemaps/*.xml`) and `robots.txt` — `site-seo-sections.test.ts` holds it to the bytes.
+    expect(entry).toContain('writeSiteSeo(out, seo)');
     expect(emitted('package.json', true)).toContain('"@ultimat3/seo"');
   });
 });

packages/cli/src/index.ts

@@ -157,8 +157,9 @@ export {
 export { quoteArg } from './shell-quote';
 export type { SiteSettings } from './site-config';
 export { loadSiteSettings, publicOrigin } from './site-config';
-export type { SiteSeo, SiteSeoOptions } from './site-seo';
+export type { SiteSeo, SiteSeoOptions, SiteSeoSitemap } from './site-seo';
 export { ROBOTS_PATH, SITEMAP_PATH, siteSeo } from './site-seo';
+export { writeSiteSeo } from './site-seo-write';
 export { eachSourceFile, isGenerated, isTest, SOURCE_GLOBS } from './source-files';
 export type { SkippedRoute, StaticReport } from './static-report';
 export { parseStaticReport } from './static-report';

packages/cli/src/seo-meta.ts

@@ -5,6 +5,8 @@
 // this is the only tier that can see both: `@ultimat3/seo` is tier 1 and may not import the route
 // registry, which is tier 4.
 
+import { ctxOf, runWithContext } from '@ultimat3/core';
+import { localizedPath } from '@ultimat3/i18n';
 import { metaContextFor, type RouteEntry, routeDataFor, routeEntries } from '@ultimat3/render';
 import type { RouteRecord } from '@ultimat3/seo';
 import { loadApp } from './app-load';
@@ -66,11 +68,19 @@ export async function scanSiteMeta(root: string): Promise<SiteMetaScan> {
 /**
  * The same scan over the registry as it stands, without loading anything.
  *
+ * `locale` runs every `meta` inside that locale's context; omitted, in none (the gate's read).
+ *
  * Split from `scanSiteMeta` so the resolution rules are testable against routes registered by hand:
  * the half worth pinning is which routes are reachable and what happens when one throws, and
  * neither of those is a fact about globbing a directory.
  */
-export async function readSiteMeta(): Promise<SiteMetaScan> {
+export async function readSiteMeta(locale?: string): Promise<SiteMetaScan> {
+  // In a locale: each `meta` runs as that locale's document would — `ctx.locale`, `t` and the URL
+  // are its own — which is what lets a page that is `noindex` in one language say so.
+  const scoped = <T>(run: () => Promise<T>): Promise<T> =>
+    locale === undefined ? run() : runWithContext(ctxOf({ role: 'web', locale }), run);
+  const pathOf = (path: string): string =>
+    locale === undefined ? path : localizedPath(path, locale);
   const records: RouteRecord[] = [];
   const unresolved: UnresolvedRoute[] = [];
   const findings: Finding[] = [];
@@ -82,10 +92,10 @@ export async function readSiteMeta(): Promise<SiteMetaScan> {
       unresolved.push({ path: entry.path, file: entry.file, reason });
       continue;
     }
-    const ctx = { url: `${PROBE_ORIGIN}${entry.path}`, params: {} };
+    const ctx = { url: `${PROBE_ORIGIN}${pathOf(entry.path)}`, params: {} };
     try {
-      const meta = await entry.config.meta(
-        metaContextFor(ctx, await routeDataFor(entry.config, ctx)),
+      const meta = await scoped(async () =>
+        entry.config.meta(metaContextFor(ctx, await routeDataFor(entry.config, ctx))),
       );
       records.push({
         path: entry.path,

packages/cli/src/seo-routes-sections.test.ts

@@ -0,0 +1,153 @@
+// A sectioned sitemap from a RUNNING web role: `/sitemap.xml` is the index, every child it names
+// answers at `/sitemaps/<name>-<n>.xml` with the same content type and cache policy, and a refresh
+// that fails keeps the whole previous answer — index and children together.
+
+import { afterEach, beforeEach, describe, expect, test } from 'bun:test';
+import { defineHttpConfig, httpServer } from '@ultimat3/http';
+import { clearRoutes, defineRoute, registerRoute } from '@ultimat3/render';
+import { seoRoutes } from './seo-routes';
+import type { SiteSettings } from './site-config';
+
+const BUILD_ID = 'seo-sections-under-test';
+const ENV = { ULTIMATE_ENV: 'production', APP_URL: 'https://www.example.com/' };
+const ORIGIN = 'https://www.example.com';
+const HOUR_MS = 3_600_000;
+const SITE: SiteSettings = {
+  origin: null,
+  disallow: [],
+  sitemap: {
+    extra: [],
+    lastmod: 'none',
+    pageSize: 2,
+    sections: [{ name: 'blog', match: '/blog/**' }],
+  },
+};
+
+let slugs: readonly string[] | 'down' = ['a', 'b', 'c'];
+let clock = 0;
+
+const page = (file: string, prerender?: () => readonly string[] | Promise<readonly string[]>) =>
+  registerRoute({
+    file,
+    component: () => 'page',
+    config: defineRoute({
+      render: 'isr',
+      revalidate: { ttl: '5m' },
+      hydrate: 'never',
+      offline: 'network-only',
+      ...(prerender === undefined ? {} : { prerender }),
+      meta: () => ({ title: 'Page', description: 'a page' }),
+    }),
+  });
+
+beforeEach(() => {
+  clearRoutes();
+  slugs = ['a', 'b', 'c'];
+  clock = 0;
+  page('apps/web/site/precios/page.tsx');
+  page('apps/web/site/blog/[slug]/page.tsx', () =>
+    // A foreign failure handed to the code under test: the driver's, not this test's verdict.
+    slugs === 'down' ? Promise.reject(new Error('the pool refused a connection')) : slugs,
+  );
+});
+
+afterEach(() => {
+  clearRoutes();
+});
+
+const server = (site: SiteSettings = SITE) =>
+  httpServer({
+    routes: seoRoutes({ env: ENV, site, now: () => clock }),
+    role: 'web',
+    config: defineHttpConfig({ dev: true, buildId: BUILD_ID, rateLimit: { scope: 'process' } }),
+  });
+
+const locs = (xml: string) => [...xml.matchAll(/<loc>([^<]+)<\/loc>/g)].map((m) => m[1] ?? '');
+const get = (app: ReturnType<typeof server>, path: string) =>
+  app.fetch(new Request(`http://10.0.0.7:3000${path}`));
+
+describe('unit · the web role serves a sectioned sitemap', () => {
+  test('/sitemap.xml is the index; each child it names answers its own urlset', async () => {
+    const app = server();
+    const index = await get(app, '/sitemap.xml');
+    expect(index.status).toBe(200);
+    expect(index.headers.get('content-type')).toBe('application/xml; charset=utf-8');
+    expect(index.headers.get('cache-control')).toBe('public, max-age=3600');
+    const body = await index.text();
+    expect(body).toContain('<sitemapindex');
+    expect(locs(body)).toEqual([
+      `${ORIGIN}/sitemaps/pages-1.xml`,
+      `${ORIGIN}/sitemaps/blog-1.xml`,
+      `${ORIGIN}/sitemaps/blog-2.xml`,
+    ]);
+
+    const expected: Readonly<Record<string, string[]>> = {
+      '/sitemaps/pages-1.xml': [`${ORIGIN}/precios`],
+      '/sitemaps/blog-1.xml': [`${ORIGIN}/blog/a`, `${ORIGIN}/blog/b`],
+      '/sitemaps/blog-2.xml': [`${ORIGIN}/blog/c`],
+    };
+    for (const [path, urls] of Object.entries(expected)) {
+      const child = await get(app, path);
+      expect(child.status).toBe(200);
+      expect(child.headers.get('content-type')).toBe('application/xml; charset=utf-8');
+      expect(child.headers.get('cache-control')).toBe(index.headers.get('cache-control'));
+      const xml = await child.text();
+      expect(xml).toContain('<urlset');
+      expect(locs(xml)).toEqual(urls);
+    }
+  });
+
+  test('a child the index does not name is a 404 that is never cached', async () => {
+    const app = server();
+    for (const path of ['/sitemaps/blog-3.xml', '/sitemaps/nada-1.xml', '/sitemaps/1.xml']) {
+      const missing = await get(app, path);
+      expect(missing.status).toBe(404);
+      expect(missing.headers.get('cache-control')).toContain('no-store');
+    }
+  });
+
+  test('robots.txt names the index and no child', async () => {
+    const robots = await (await get(server(), '/robots.txt')).text();
+    expect(robots.match(/^Sitemap: .*$/gm)).toEqual([`Sitemap: ${ORIGIN}/sitemap.xml`]);
+  });
+
+  test('with no sections the same site is one urlset and /sitemaps/* answers nothing', async () => {
+    const app = server({
+      ...SITE,
+      sitemap: { extra: [], lastmod: 'none', sections: [], pageSize: 50_000 },
+    });
+    const body = await (await get(app, '/sitemap.xml')).text();
+    expect(body).toContain('<urlset');
+    expect(locs(body)).toHaveLength(4);
+    expect((await get(app, '/sitemaps/blog-1.xml')).status).toBe(404);
+  });
+});
+
+describe('unit · a sectioned sitemap refreshes behind the answer it would replace', () => {
+  test('a failed prerender keeps the previous index AND its children, then the retry replaces both', async () => {
+    const app = server();
+    const before = await (await get(app, '/sitemap.xml')).text();
+    expect(locs(before)).toHaveLength(3);
+
+    slugs = 'down';
+    clock += HOUR_MS + 1;
+    const during = await get(app, '/sitemap.xml');
+    expect(during.status).toBe(200);
+    expect(await during.text()).toBe(before);
+    await Bun.sleep(5);
+    const child = await get(app, '/sitemaps/blog-2.xml');
+    expect(child.status).toBe(200);
+    expect(locs(await child.text())).toEqual([`${ORIGIN}/blog/c`]);
+
+    // The read recovers with one page fewer: the second blog page is gone from index and role alike.
+    slugs = ['a', 'b'];
+    clock += 60_001;
+    await get(app, '/sitemap.xml');
+    await Bun.sleep(5);
+    expect(locs(await (await get(app, '/sitemap.xml')).text())).toEqual([
+      `${ORIGIN}/sitemaps/pages-1.xml`,
+      `${ORIGIN}/sitemaps/blog-1.xml`,
+    ]);
+    expect((await get(app, '/sitemaps/blog-2.xml')).status).toBe(404);
+  });
+});