marketingFilter is bypassed on bulk ingest: cap fails open on backfill, absent entirely on /batch

[zachdunn]

Noticed while auditing marketingFilter sources for #2168 (item 5c). No action needed today — filing so the interaction is written down before someone runs a backfill and is surprised by it.

The observation

clickhouse-blog's RSS window exposes far more history than we've ingested:

source	feed items	stored	gap
clickhouse-blog	822	379	443
val-town-blog	109	174	—
pulumi-blog	50	100	—
dbt-blog	25	71	—

It's the only one of the six active feed/video marketingFilter sources with a gap; the others have stored ≥ feed window. The newest 8 ClickHouse items are all stored and there are no marketing-filter-cap-tripped events in 14 days, so steady-state ingest is healthy — the 443 are old backlog outside the ingest window, not a stuck fetch. Nothing is broken.

I did not check whether this generalizes beyond the marketingFilter set. If someone wants to know how much feed history we're leaving on the floor registry-wide, that's the sweep to run.

Why it's worth writing down: backfilling this source would bypass the marketing filter

clickhouse-blog has marketingFilter on for a reason — its feed mixes product news with newsletters, competency announcements, and customer stories. Both backfill routes drop that protection:

1. Bulk fetch trips the cap and inserts unfiltered. MARKETING_CLASSIFIER_MAX_PER_FIRE = 20 (poll-fetch.ts:1147). Above it, classifyMarketingForReleases logs marketing-filter-cap-tripped and returns an empty map (:1226-1233). Downstream, marketingMap.get(index) is then undefined for every row (:1498), so nothing is suppressed and everything inserts visibly. The cap is a cost guard, and its failure mode is fail-open by design — fine for a 0–5 item feed delta, wrong for a 443-item backfill.
2. The /batch and workflow paths never classify at all. Neither routes/releases-batch.ts, workflows/backfill-source.ts, nor adapters/scrape-persister.ts reference the classifier — it lives only in the fetchOne path. So a local-ingest or backfill-source run skips it outright.

Net: any obvious way to pull those 443 in lands ~443 unfiltered marketing posts in the index — the exact outcome the filter exists to prevent. The existing escape hatch is the one named in the code comment: insert, then suppress after the fact via the suppress API.

If we ever do want this history

Options, roughly in order of preference:

• Leave it. 443 mostly-marketing posts from a source we already track live is low value against the cleanup risk. This is my recommendation.
• Backfill with an explicit classification pass — raise the per-fire cap for a one-off, or run the classifier over the candidate set before insert. Costs ~443 google/gemini-2.5-flash-lite calls, which is cents, but needs the code path to exist.
• Backfill unfiltered into suppressed=1, then unsuppress the real product news. Safe default, manual review.

Reproducing the numbers

['releases-cloudflare-logs']
| where ['body'] contains 'marketing-filter-cap-tripped'
| extend p = parse_json(['body'])
| project _time, slug = tostring(p['sourceSlug']), candidates = toint(p['candidateCount'])

The gap itself is feed-vs-DB: pull the source's metadata.feedUrl, extract item-level <link>s, and diff against SELECT url FROM releases WHERE source_id = .... Note a feed's <link> set includes non-item links — scope the regex to inside <item> / <entry> blocks or the count comes out ~2× too high.

[zachdunn]

Verified the two mechanisms behind this. Worth keeping open — the trap is real and the "leave it" recommendation is right. But the second claim is wrong in a way that changes what a fix would look like, so correcting it before it's acted on.

Claim 1 — the cap fails open. Confirmed. MARKETING_CLASSIFIER_MAX_PER_FIRE = 20 (poll-fetch.ts:1148, not 1147). Over the cap, classifyMarketingForReleases logs marketing-filter-cap-tripped and returns the untouched empty result map (:1225-1234), so every row reads undefined downstream and inserts visibly. Exactly as described.

Claim 2 — "the /batch and workflow paths never classify at all" — half wrong.

The classifier call at poll-fetch.ts:1408 is inside ingestRawReleases (which spans 1345–1612), not inside fetchOne. And every backfill route calls ingestRawReleases:

• workflows/backfill-source.ts:329 (durable path)
• routes/workflows.ts:2721 (synchronous path)
• workflows/firecrawl-ingest.ts:325
• lib/video-materialize.ts:169

So backfill does classify. The codebase states this as a deliberate invariant — video-materialize.ts:60: "Backfill runs through ingestRawReleases so the marketing classifier, summarizer-trigger, and dedup all apply to the initial batch." Grepping for classifyMarketing finds one call site and misses this, because the coupling is one level up.

/batch (routes/releases-batch.ts) is correctly identified — it has its own insert path, never touches ingestRawReleases, and never classifies. That's the local-ingest write path.

Why the correction matters. The conclusion survives — a 443-item backfill still lands unfiltered — but for a different reason, and the two failure modes want different fixes:

Path	Mechanism	Fix shape
Backfill workflows	Classifier runs, then the 20-item cap trips and fails open	Chunk or raise the cap for backfill; the code path already exists
/batch	No classifier at any point	Would need the pass to be built there

Framed as "neither path references the classifier," someone would go build a classification pass for backfill that is already there and just needs its cap handled — the cheap fix mistaken for the expensive one.

Unverified: the 443 gap itself and the peer table. I didn't re-pull the feeds; the diff method described has the <item>-scoping caveat already noted, so the number is plausible but uncorroborated.

Verdict: keep open, low priority, recommendation unchanged. Suggest retitling to something like "marketingFilter is bypassed on bulk ingest: cap fails open on backfill, absent entirely on /batch" — the current title points at only one of the two paths and at the wrong mechanism for it.