DEVOPS-5014: Cloudflare Workers Static Assets (no Worker script) - #2122
Draft
Shabablinchikow wants to merge 8 commits into
Draft
DEVOPS-5014: Cloudflare Workers Static Assets (no Worker script)#2122Shabablinchikow wants to merge 8 commits into
Shabablinchikow wants to merge 8 commits into
Conversation
Ports the Netlify setup to Cloudflare Workers + Static Assets (DEVOPS-5014). Additive: netlify.toml is untouched and Netlify stays the live site until the DNS cutover is verified, so rollback is "do not switch DNS". - wrangler.jsonc: assets served from site/, ASSETS binding, run_worker_first limited to extension-less page URLs so images/fonts/CSS stay on the asset fast path (no billed Worker invocation, and _headers keeps applying). - src/worker.ts: port of netlify/edge-functions/markdown-negotiation.ts. Keeps Accept: text/markdown negotiation, Vary: Accept and the RFC 8288 Link headers. Drops the User-Agent/IP scraper blocks — those become Cloudflare WAF rules, which run in front of the Worker and cost nothing. - docs/6.x/_headers: port of the netlify.toml [[headers]] blocks. Cloudflare's splat is greedy, so one /*.md rule replaces Netlify's /*.md + /**/*.md pair. - docs/6.x/_redirects: ports the two netlify.toml 200-rewrites (favicon.ico and the sitemap.xml catch-all). - scripts/build.sh: the netlify.toml [build] command, extracted so both platforms run the same steps. Keep in sync until netlify.toml is removed. - .github/workflows/cloudflare.yml: PR to master uploads a version with a stable per-PR preview alias and comments the link; merge to master deploys and purges the zone cache. Verified locally with `wrangler dev`: markdown negotiation on pages, HTML fallback when no .md companion exists, _headers applied to assets that bypass the Worker, and _redirects (301s and 200-rewrites) taking precedence over it.
✅ Deploy Preview for pensive-dubinsky-5f7a00 ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
wrangler.jsonc gains env.staging (wallarm-docs-staging), served at docs-staging.wallarm.com so the whole flow can be exercised before docs.wallarm.com moves. A separate Worker rather than a preview version of the production one, so staging can be redeployed without touching whatever production is serving. Routes stay out of wrangler and live in infra/cloudflare-iac - two systems must not both own them. build.sh gains SKIP_IMAGE_OPTIMISATION. The image pass downloads x86_64 Linux binaries, so a production-mode build could not run on a developer machine at all - and since the .md companions the Worker serves are only generated in production mode, the Worker's whole reason for existing was untestable locally.
Cloudflare rejects the whole deployment, not just the overflow:
Maximum number of dynamic _redirects rules limit of 100 exceeded [100324]
Three separate problems, found by actually deploying:
1. ORDERING, which was the real one. Cloudflare charges every rule appearing
*after* the first wildcard rule against the 100-"dynamic" budget, whatever
that rule itself looks like. A wildcard on line 22 made ~700 plain rules
count as dynamic. Its own warning said so - "could be made more performant
by bringing it above any lines with splats" - and I read that as a
performance hint rather than a hard limit. Static rules now come first and
wildcards are grouped at the end, which is also better semantics: exact
matches win over wildcards.
2. 128 per-version boilerplate rules (/4.8/admin-en/* -> /4.8 and friends,
across fifteen retired versions) are replaced by one Cloudflare Redirect
Rule in infra/cloudflare-iac. Agreed with the docs team: those paths now
land on the version-list page instead of a per-version stub that is not
built and 404s today. The fifteen legacy rules that map onto live pages
(/2.12/api/* -> /api/:splat) stay here, and the Cloudflare rule explicitly
excludes their prefixes so it cannot shadow them.
3. 20 rules used Netlify's `301!` force syntax, which Cloudflare parses as
status 0 and rejects. Redirects already run before assets there, so a plain
301 is equivalent. 3 rules duplicated a path already defined earlier and
were dropped - Cloudflare treats /foo and /foo/ as distinct, so only exact
collisions were removed.
Netlify is unaffected: it is first-match-wins in file order either way.
src/worker.ts existed for Accept: text/markdown negotiation. That is now
three zone rules in infra/cloudflare-iac, which cost nothing to run where a
Worker script is billed per request - and the script ran on every HTML page
view, since a plain page request has no Accept header to rewrite on.
Measured on staging: 24 requests produced 13 invocations, all from paths the
rules did not yet cover.
Where each responsibility went:
negotiation -> URL rewrite rules
fallback on a missing companion -> scripts/check_markdown_companions.py,
which fails the build instead. A rewrite is blind, so a missing .md is
not a fallback to HTML but a 404 mislabelled Content-Type: text/markdown.
Vary: Accept, Link -> response-header rules
CSP / HSTS / nosniff -> _headers, which applies again now. Those were only
re-set in code because _headers does not apply to Worker responses.
Lost, accepted: X-Markdown-Tokens (computed from body length, not expressible
in a rule) and the per-page Link rel=alternate (response-header expressions
reject wildcard_replace). The HTML head still carries the alternate link.
Recoverable from git history if the rules disappoint on production traffic.
My local production build ran the pngquant/oxipng pass over images/, which rewrites the files in place, and git add -A swept 1115 of them into the previous commit. The optimisation is meant to run only on the ephemeral CI checkout - netlify.toml and scripts/build.sh both say full-resolution originals stay in git - so committing the lossy output would have destroyed the sources. Restored from master. Nothing else in that commit is affected.
Superseded before it ever ran. The agreed deploy path is Cloudflare Workers Builds, which is the direct Netlify equivalent - git integration, build command, production branch deploys, per-branch preview URLs - and needs none of the four repo secrets this workflow required. Workers Builds is configured in the Cloudflare dashboard rather than here, so this leaves the repo with no CI of its own until that is connected. Netlify continues to build and serve production in the meantime, which is the whole point of keeping netlify.toml untouched.
Shabablinchikow
changed the title
wrangler.jsonc: workers_dev = false. It would otherwise publish the whole site a second time on *.workers.dev, and zone rules do NOT apply there - verified against staging: Accept: text/markdown workers.dev -> text/html zone -> text/markdown /4.8/admin-en/foo/ workers.dev -> 404 zone -> 301 Vary workers.dev -> absent zone -> present So that copy had no redirects, no markdown negotiation, no WAF and no bot protection, and search engines could index it as duplicate content. After the change workers.dev returns 404 and the zone hostname still serves. preview_urls stays on: Workers Builds gives each non-production branch a versioned preview, which is how PR review works. Those are on workers.dev too, so previews show content accurately but not routing - a redirect or negotiation change has to be checked on docs-staging.wallarm.com. .python-version pins 3.14; the build image defaults to 3.13. DEPLOY-CLOUDFLARE.md records the Workers Builds settings, which live in the dashboard rather than the repo and would otherwise be undocumented: the submodule init the build needs, CONTEXT=production (without it the .md companions are never generated and negotiation 404s), and PUPPETEER_SKIP_CHROMIUM_DOWNLOAD, since package.json drags in ~170 MB of Chromium the docs build never uses. CLAUDE.md now describes both pipelines instead of Netlify alone, including the two things that hard-fail a Cloudflare build: redirect ordering and a missing .md companion.
Second time: timing the production build re-ran the pngquant/oxipng pass, which rewrites images/ in place, and git add -A committed the lossy output over the full-resolution sources. The behaviour is correct on CI, where the checkout is ephemeral. It is only dangerous locally, so build.sh now says so at the point it happens and points at SKIP_IMAGE_OPTIMISATION=1.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Moves docs.wallarm.com from Netlify to Cloudflare Workers Static Assets (DEVOPS-5014).
Additive.
netlify.tomlandnetlify/are untouched, so Netlify keeps building and serving production throughout.Verified end to end on https://docs-staging.wallarm.com — a real hostname on the same Cloudflare zone, delegated in
terraform-route53and wired up ininfra/cloudflare-iac.Six files
wrangler.jsoncdocs/6.x/_headersnetlify.toml [[headers]]blocksdocs/6.x/_redirectsscripts/build.shnetlify.toml [build]command, extractedscripts/check_markdown_companions.py.gitignoreThere is no Worker
The original port carried
src/worker.ts, a translation ofnetlify/edge-functions/markdown-negotiation.ts. It is gone.Accept: text/markdownnegotiation is now three zone rules in infra/cloudflare-iac (markdown_negotiation.tf).Why it mattered: a plain HTML page view has no
Acceptheader to rewrite on, so the Worker ran on every page view. Measured on staging with the Worker in place: 24 requests produced 13 invocations. After removing it, 80+ requests produced zero. Static asset requests are free and unlimited; Worker script requests are billed.Removing it also fixed a class of bug rather than an instance: the Worker had to re-set CSP, HSTS and
nosniffin code purely because_headersdoes not apply to responses returned from Worker code. With no Worker,_headersapplies directly.What was given up
X-Markdown-Tokens— computed from body length, not expressible in a rule.Link: rel=alternate— response-header expressions rejectwildcard_replace(API error 20087). Now a staticllms.txtservice-doc; the rendered HTML still carries<link rel="alternate">in<head>..mdcompanion is missing. This is the one that needed replacing, hencecheck_markdown_companions.py.The script is recoverable from git history if the rules disappoint under production traffic.
scripts/check_markdown_companions.pyA URL rewrite is blind: Cloudflare cannot test whether the
.mdexists. A page without a companion does not fall back to HTML — it returns the 404 page withContent-Type: text/markdown, because the_headersrule matches the request path. An agent gets 279 KB of HTML claiming to be Markdown.The Worker checked
md.okand fell back. Losing that is only safe if every page has a companion is enforced, so this fails the build instead. Current run: 931 pages checked, 3 exempt, 0 missing. The exempt three are version roots — theme-assembled landing pages — and its exemption list must stay in sync withlocal.md_no_companioninmarkdown_negotiation.tf._redirectswas a hard deploy blockerCloudflare rejects the entire deployment past 100 dynamic rules — not the overflow, the whole thing:
Three separate causes, all found by actually deploying:
1. Ordering — the real one. Cloudflare charges every rule appearing after the first wildcard rule against the dynamic budget, whatever that rule itself looks like. A wildcard on line 22 made ~700 plain rules count as dynamic. Its own warning said so, and I first read it as a performance hint. Static rules now come first, wildcards are grouped at the end. This is also better semantics — exact matches win over wildcards — and Netlify is unaffected, being first-match-wins in file order either way.
2. 128 per-version boilerplate rules (
/4.8/admin-en/*→/4.8, across fifteen retired versions) replaced by one Cloudflare Redirect Rule. Agreed with the docs team: those paths now land on the version-list page rather than a per-version stub that is not built and 404s today. The fifteen legacy rules that map onto live pages (/2.12/api/*→/api/:splat) stay here, and the Cloudflare rule explicitly excludes their prefixes so it cannot shadow them.3. Invalid syntax. 20 rules used Netlify's
301!force suffix, which Cloudflare parses as status0. Redirects already run before assets there, so a plain301is equivalent. 3 rules duplicated a path defined earlier and were dropped — Cloudflare treats/fooand/foo/as distinct, so only exact collisions were removed.Verified on docs-staging.wallarm.com
/200 text/markdown, authored companion200 text/htmlAccept: text/markdown200 text/htmlfallbacknosnifffrom_headersVary: Accept,Linkllms.txt, sitemap and favicon rewrites200, correct content types:splatrulesStill needed before cutover
cloudflare-iacare scoped todocs-staging.wallarm.com; widen todocs.wallarm.comat cutover.wallarm.comis a partial zone, so Cloudflare holds its own record for the proxied hostname:CNAME docs.wallarm.com → pensive-dubinsky-5f7a00.netlify.app. Rollback is restoring that value.cloudflare_workers_custom_domainwill not work — Custom Domains refuse a hostname that already carries a CNAME and are documented against full zones. Production needs the same route-based treatment staging uses.🤖 Generated with Claude Code