feat(warmup): drain the tail the cap excluded, a batch at a time

CHANGELOG.md

@@ -175,6 +175,12 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/).
   `timberkit_warmup_priority_weights`. Off by default.
 - Site Health check `preload_chain_healthy` — reports a Breeze preload chain
   that has stopped making progress.
+- `$breeze_warmup_tail` — keep warming the URLs the cap excluded, a batch at a
+  time, in score order, pausing whenever Breeze is draining its own preload
+  queue. Batch size is `$breeze_warmup_tail_batch` (default 100 per five-minute
+  tick) and filterable via `timberkit_warmup_tail_batch`; the stored tail is
+  capped by `timberkit_warmup_tail_max_urls` (default 5000). Off by default,
+  requires `$breeze_warmup_priority`.
 
 ### Changed
 

README.md

@@ -767,6 +767,79 @@ stored URL list itself is trusted before a refresh is scheduled. A practical
 way to size the cap: count how many URLs got at least one pageview
 yesterday; that number is the cap.
 
+### Draining the tail the cap left behind
+
+`$breeze_warmup_priority` picks what gets warmed first; it does not warm what
+the cap excludes. `$breeze_warmup_tail` (default `false`) keeps going after
+the purge: every five minutes it dispatches another batch of the excluded
+URLs to Breeze's preloader, in the same score order, until the tail runs out
+or the next purge resets the run. Requires both `$breeze_warmup_sitemap` and
+`$breeze_warmup_priority` — on its own there is no ordering to drain.
+
+```php
+class Base extends StarterBase {
+    public function __construct() {
+        $this->breeze_warmup_sitemap     = true;
+        $this->breeze_warmup_priority    = true;
+        $this->breeze_warmup_tail        = true;
+        $this->breeze_warmup_tail_batch  = 100;
+
+        parent::__construct();
+    }
+}
+```
+
+Each tick checks Breeze's own preload queue (`breeze_preload_queue`) first
+and stands aside while it is non-empty, so the tail drain never competes with
+Breeze for the same origin renders. A skipped tick still schedules its
+successor — the chain only ends when the tail itself is exhausted.
+
+**This never reaches "done", and that is by design.** A full purge can arrive
+several times a day, and each one starts the tail over from its head. On a
+busy site, only part of the tail is ever covered in one run — but because the
+tail is in score order, the part covered is always the most valuable part.
+Do not size this feature expecting the whole sitemap to eventually go warm;
+size it expecting the top of the tail to stay warm continuously.
+
+**The batch size does not multiply out the way it looks.** `$breeze_warmup_tail_batch`
+of 100 per five-minute tick reads as 1200 URLs an hour, but that arithmetic
+only holds if every tick fires — and a tick fires only when Breeze's own
+queue is idle. Real throughput on a site that purges and warms constantly is
+lower. Size the batch by the origin-render budget you can afford in a tick
+that does run, not by the hourly total the multiplication suggests.
+
+**The cursor counts URLs dispatched, not URLs warmed.** Each tick hands its
+batch to `Breeze_Cache_Preloader::preload_url()`, which returns `void` and
+may reject a URL outright. The tail advancing past a URL means it was handed
+to Breeze, not that the origin rendered it. There is no confirmation signal
+to build on: Breeze does not report back.
+
+**The five-minute interval is fixed, not a filter.** `$breeze_warmup_tail_batch`
+is the only knob; the tick's own cadence stays constant so the brake against
+Breeze's queue behaves predictably. `$breeze_warmup_tail_batch`, like
+`$breeze_warmup_priority_weights`, is read once, at registration — changing
+it at runtime after that has no effect. Filterable independently:
+`timberkit_warmup_tail_batch` (also applied once, at registration) and
+`timberkit_warmup_tail_max_urls` (default 5000) which caps how many excluded
+URLs are stored as the tail in the first place — a safety bound distinct from
+the sitemap URL cap.
+
+**A cold start rescues itself.** A purge schedules the first tick and resets
+the tail's cursor immediately, but the tail's *contents* are only written
+later, by the deferred refresh. If the first tick runs before that refresh
+has written anything, it finds an empty tail and ends the chain — nothing
+else would ever restart it. To close that gap, the refresh itself schedules a
+tick whenever it writes a non-empty tail, so the chain resumes once the
+tail actually has something to drain.
+
+**Tail draining refuses to wire on multisite.** The brake reads Breeze's
+`breeze_preload_queue` option, and Breeze scopes that option per blog on
+multisite — every site's brake would read "idle" regardless of what any
+other site's queue is doing, and the drain from every site would pile onto
+whatever origin actually serves the requests. Rather than run without a
+working brake, `register()` detects `is_multisite()` and leaves the tail
+hooks unwired there, even when `$breeze_warmup_tail` is `true`.
+
 ### Preload chain health
 
 The Site Health check `preload_chain_healthy` (category `caching`, needs

phpstan.neon

@@ -18,6 +18,9 @@ parameters:
         - '#Call to static method .* on an unknown class Timber#'
         # WP-CLI is a separate package, not in the WordPress stubs (cf. Timber above).
         - '#Call to static method .* on an unknown class WP_CLI#'
+        # Breeze_Cache_Preloader is the Breeze plugin's class, not a dependency
+        # of this package (cf. WP-CLI and Timber above).
+        - '#Call to static method .* on an unknown class Breeze_Cache_Preloader#'
         # wp_get_attachment_metadata stub shape omits original_image; presence is
         # guarded by `! empty( $metadata['original_image'] )` before the unset.
         -

src/Breeze/TailPlanner.php

@@ -0,0 +1,88 @@
+<?php
+
+declare(strict_types=1);
+
+namespace Parisek\TimberKit\Breeze;
+
+/**
+ * The tail's pure arithmetic: which URLs the cap excluded, how that list is
+ * fingerprinted, and how a batch is sliced off it.
+ *
+ * Deliberately free of WordPress. The property test that pins the batching
+ * invariant lives under `tests/Property/`, which is reserved for functions
+ * isolated from Brain\Monkey.
+ *
+ * This class never sorts. Its caller hands it an already-ordered set, because
+ * the tail's order IS the feature's promise: a run cut short by the next purge
+ * must still have warmed the most valuable pages first.
+ */
+final class TailPlanner {
+
+	/** @var int Safety cap on stored tail URLs, filterable by the caller. */
+	public const DEFAULT_MAX_TAIL_URLS = 5000;
+
+	/**
+	 * URLs the cap excluded, in the order they were given.
+	 *
+	 * @param array<int, array<string, mixed>> $scored   Already-sorted scored records.
+	 * @param array<int, string>               $keptUrls URLs that made it under the cap.
+	 * @param int                              $maxTail  Upper bound on the result.
+	 * @return array<int, string>
+	 */
+	public static function split( array $scored, array $keptUrls, int $maxTail ): array {
+		$kept = array_fill_keys( $keptUrls, true );
+		$tail = array();
+
+		foreach ( $scored as $record ) {
+			if ( ! isset( $record['url'] ) || ! is_string( $record['url'] ) || '' === $record['url'] ) {
+				continue;
+			}
+			if ( isset( $kept[ $record['url'] ] ) ) {
+				continue;
+			}
+
+			$tail[] = $record['url'];
+		}
+
+		return array_slice( $tail, 0, max( 0, $maxTail ) );
+	}
+
+	/**
+	 * Fingerprint of a tail, used to invalidate a cursor that points into a
+	 * different plan. Order participates: a reordered tail is a new plan.
+	 *
+	 * @param array<int, string> $urls
+	 * @return string
+	 */
+	public static function hash( array $urls ): string {
+		$urls    = array_values( $urls );
+		$encoded = json_encode( $urls );
+
+		// json_encode() returns false on invalid UTF-8. Falling through to ''
+		// would hash every unencodable tail to the same md5(''), making the
+		// cursor believe nothing changed. The concatenation is a fallback,
+		// not the primary encoding: it still varies with content and order,
+		// which is all a fingerprint needs.
+		if ( false === $encoded ) {
+			$encoded = implode( "\n", $urls );
+		}
+
+		return md5( $encoded );
+	}
+
+	/**
+	 * One batch, starting at the cursor.
+	 *
+	 * A negative index is clamped to zero rather than passed to array_slice(),
+	 * which would read from the end of the array — a corrupted cursor must not
+	 * silently warm the wrong pages.
+	 *
+	 * @param array<int, string> $urls
+	 * @param int                $index
+	 * @param int                $batch
+	 * @return array<int, string>
+	 */
+	public static function nextBatch( array $urls, int $index, int $batch ): array {
+		return array_slice( $urls, max( 0, $index ), max( 0, $batch ) );
+	}
+}

src/Breeze/TailStore.php

@@ -0,0 +1,152 @@
+<?php
+
+declare(strict_types=1);
+
+namespace Parisek\TimberKit\Breeze;
+
+/**
+ * The tail and the cursor into it, in two option rows the purge-time filter
+ * never reads.
+ *
+ * Keeping them out of the priority row is not tidiness: that row is
+ * deserialized inside the request that emptied the cache, and a tail can hold
+ * thousands of URLs.
+ *
+ * The cursor has two writers — the purge resets it, the tick advances it — so
+ * advancing is conditional. A tick that started before a purge and finished
+ * after it must not overwrite the fresh reset, or every URL the purge
+ * invalidated would sit unwarmed behind a cursor claiming otherwise. That is
+ * the race this guards against, and it is the one that actually happens.
+ *
+ * It does not guard two genuinely concurrent ticks. Both can read the same
+ * cursor, both can pass the check in advanceCursor(), and both can call
+ * update_option() — the later one wins, and both calls return true. Closing
+ * that would need a conditional UPDATE through $wpdb matched against the
+ * serialized option value: fragile, and disproportionate to the risk. The
+ * tail advance is driven by one scheduled tick, not by concurrent actors, so
+ * overlapping ticks are rare; when they do overlap, the result is a repeated
+ * batch of a handful of extra warm requests, not corrupted state.
+ */
+final class TailStore {
+
+	/** @var string wp_options key holding the ordered tail, autoload off. */
+	public const TAIL_OPTION = 'timber_kit_breeze_warmup_tail';
+
+	/** @var string wp_options key holding the cursor into it, autoload off. */
+	public const CURSOR_OPTION = 'timber_kit_breeze_warmup_tail_cursor';
+
+	/**
+	 * @return array{urls: array<int, string>, hash: string}
+	 */
+	public static function readTail(): array {
+		$empty = array( 'urls' => array(), 'hash' => '' );
+
+		if ( ! function_exists( 'get_option' ) ) {
+			return $empty;
+		}
+
+		$data = get_option( self::TAIL_OPTION, null );
+		if ( ! is_array( $data ) || ! isset( $data['urls'], $data['hash'] ) || ! is_array( $data['urls'] ) ) {
+			return $empty;
+		}
+
+		return array(
+			'urls' => array_values( array_filter( $data['urls'], 'is_string' ) ),
+			'hash' => (string) $data['hash'],
+		);
+	}
+
+	/**
+	 * @param array<int, string> $urls
+	 * @return void
+	 */
+	public static function writeTail( array $urls ): void {
+		if ( ! function_exists( 'update_option' ) ) {
+			return;
+		}
+
+		$urls = array_values( $urls );
+
+		update_option(
+			self::TAIL_OPTION,
+			array(
+				'urls' => $urls,
+				'hash' => TailPlanner::hash( $urls ),
+			),
+			false
+		);
+	}
+
+	/**
+	 * @return array{index: int, hash: string}
+	 */
+	public static function readCursor(): array {
+		if ( ! function_exists( 'get_option' ) ) {
+			return array( 'index' => 0, 'hash' => '' );
+		}
+
+		$data = get_option( self::CURSOR_OPTION, null );
+		if ( ! is_array( $data ) || ! isset( $data['index'], $data['hash'] ) ) {
+			return array( 'index' => 0, 'hash' => '' );
+		}
+
+		return array(
+			'index' => (int) $data['index'],
+			'hash'  => (string) $data['hash'],
+		);
+	}
+
+	/**
+	 * Start the tail over.
+	 *
+	 * Writes an empty hash on purpose: the tick stamps the real one when it
+	 * reads the tail anyway. Reading the tail here would drag a payload of
+	 * thousands of URLs into the request an editor is waiting on.
+	 *
+	 * @return void
+	 */
+	public static function resetCursor(): void {
+		if ( ! function_exists( 'update_option' ) ) {
+			return;
+		}
+
+		update_option( self::CURSOR_OPTION, array( 'index' => 0, 'hash' => '' ), false );
+	}
+
+	/**
+	 * Advance the cursor, but only if nobody moved it since it was read.
+	 *
+	 * @param array{index: int, hash: string} $expected Cursor as the caller read it.
+	 * @param int                             $newIndex
+	 * @param string                          $hash     Hash of the tail actually used.
+	 * @return bool True when written, false when a concurrent write won.
+	 */
+	public static function advanceCursor( array $expected, int $newIndex, string $hash ): bool {
+		if ( ! function_exists( 'update_option' ) ) {
+			return false;
+		}
+
+		$current = self::readCursor();
+		if ( $current['index'] !== (int) $expected['index'] || $current['hash'] !== (string) $expected['hash'] ) {
+			return false;
+		}
+
+		$wanted = array( 'index' => max( 0, $newIndex ), 'hash' => $hash );
+
+		update_option( self::CURSOR_OPTION, $wanted, false );
+
+		// The return value is read back rather than taken from update_option(),
+		// which answers false for a write that changed nothing as well as for
+		// one that failed. Both happen here: a concurrent tick can have stored
+		// the identical cursor already, and that is success, not failure. So
+		// the question asked is the one the caller actually cares about --
+		// does the stored cursor now say what this tick wanted?
+		//
+		// It matters because the caller acts on the answer. A failed write left
+		// reported as success means every later tick repeats the same batch
+		// forever, with nothing in any log to say so.
+		$stored = self::readCursor();
+
+		return $stored['index'] === $wanted['index'] && $stored['hash'] === $wanted['hash'];
+	}
+}