Add global Cloudflare throttle for player_crawl; cap player_crawl workers at 50

[Copilot]

• Create lib/web/bungie/cloudflare_throttle.go with global Cloudflare throttle mechanism
  • Throttle only activates after ≥3 Cloudflare errors within a 10-second sliding window
  • Fix gen counter bug: early-return when throttle is already active so gen is only incremented and a single timer is only spawned once per throttle activation (not per-error)
• Update lib/messaging/queue-workers/player_crawl.go to wait for and signal the Cloudflare throttle
  • Cap MaxWorkers at 50
• Revert Zeus WWW rate limit back to original (12 req/s, burst 25)

Original prompt

When I get 403 cloudflare errors, I need to add some sort of global throttling system. This usually happens on the player_crawl queue. I get a retry storm of messages which further exacerbates the issue, resulting in a small portion of messages getting dead lettered.

We can also reduce the Zeus throughout a bit for the www endpoint

---

💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.

[owens1127]

@copilot can you ensure this only activates above X events in N seconds? Also don't decrease Zeus anymore--undo that. Plus, cap player crawl workers at 50 if not already

[Copilot]

@copilot can you ensure this only activates above X events in N seconds? Also don't decrease Zeus anymore--undo that. Plus, cap player crawl workers at 50 if not already

Done in a2a5184:

• Threshold: The throttle now requires ≥3 Cloudflare errors within a 10-second sliding window before activating. Each call to SignalCloudflareThrottle records a timestamp, prunes entries older than the window, and only activates (or extends) the throttle once the count reaches the minimum.
• Zeus: Reverted the WWW rate limit back to the original 12 req/s (burst 25).
• Worker cap: MaxWorkers reduced from 70 → 50.

[owens1127]

@copilot address my review comment

[owens1127]

But don't we increase the gen on every call? If 3 errors come in at the same time, wouldn't they all have different gens? We don't early return if throttling is enabled.

[Copilot]

Fixed in 3bfd3a8. Added an early return when cloudflareIsThrottled is already true, so gen is only incremented and a timer is only spawned once per throttle activation — not on every subsequent error above the threshold. With the bug, 50 concurrent errors above the threshold would spawn 48 timers and increment gen 48 times; now only the first one to see !cloudflareIsThrottled does any work.

[Copilot]

@copilot address my review comment

Addressed in 3bfd3a8 — the gen counter bug is fixed.

[owens1127]

Superseded by #76, which handles Cloudflare blocks with a single quick in-process retry and unlimited Hermes queue backoff (no message drops) instead of global throttling.