feat: dynamic include-values source for the keep-filter

[jghoman]

What

The keep-filter's allowlist can now be sourced at runtime from an HTTP endpoint instead of being fixed at startup. MILLPOND_INCLUDE_VALUES_URL names a URL returning a JSON array of scalars, polled on a background thread (60s default, ±10% jitter, optional auth header). Millpond stays endpoint-agnostic — nothing in the code knows what the values mean.

Two modes, designed for shadow-first rollout:

• shadow (default): the static MILLPOND_FILTER_VALUES list stays authoritative; the endpoint is polled purely for observability (shadow_only_static/shadow_only_remote diff gauges, staleness timestamp). Flip gate: symmetric difference at zero across the fleet.
• authoritative: the polled set drives the filter; the static list seeds the initial held set.

See the README's new "Dynamic allowlist source" section for the full static × dynamic interaction table, and AGENT.md for the invariants.

Safety semantics

Shaped by the consequence asymmetry — an erroneous addition writes surplus rows a downstream reader ignores; an erroneous removal silently drops records with no recovery:

• Additions apply on the first successful poll. Removals need REMOVAL_POLLS (default 5) consecutive successful accepted polls absent; failed and refused polls advance nothing.
• Poll failures keep the last-known-good set indefinitely; staleness is a gauge.
• Refusal guards reject the entire poll (refused_total{reason}): empty (an empty array is never removal evidence and never an acceptable first state), bulk_removal (> half of a multi-value set confirmed at once), type_flip (int↔str — one junk element would flip the normalized set to strings, the filter's values→column cast then fails, and whole batches drop as filter_field_missing with offsets committed: the worst silent-loss path, now refused instead).
• Authoritative startup blocks for the first successful poll and fails the pod on timeout — no proceed-on-stale-bootstrap. The damping state is in-memory, so a restart during an endpoint outage that ran on the bootstrap would perform de-facto removals with zero polls of evidence. A halt is recoverable from Kafka; a silent drop is not.
• millpond_include_values_mode{mode} reports what each replica actually runs so fleet-level flip gates can't pass vacuously on a replica that never got the URL.
• _fetch refuses redirects (urllib re-sends the custom auth header cross-host), caps responses at 4 MB.

Testing

Authored with an agent; adversarial lead-QE and principal-SWE review agents ran against the initial version — their findings (refusal pre-charge attack, type-flip batch-drop chain, restart amnesia, vacuous flip gate, redirect header leak, response cap, jitter, mode-without-URL validation) are all incorporated with regression tests. Checks actually run per the AGENT.md pre-push checklist:

• just lint / just fmt-check clean
• just test — 604 unit tests passed (35 new)
• just test-integration — 23 passed
• just test-e2e — 4 passed (full DuckLake stack)
• SSL-Kafka verification skipped by agreement: no Kafka client config paths touched.

Deployment

Inert until a URL is configured — with MILLPOND_INCLUDE_VALUES_* unset, behavior is byte-identical to today (verified by the untouched filter test suite). Chart wiring and the endpoint itself ship separately.