Skip to content

DO NOT MERGE — fly.toml: min_machines_running 1->0 (Telegram webhook mode, Phase 3b) - #463

Open
jlunder00 wants to merge 1 commit into
devfrom
feature/infra-webhook-flytoml
Open

DO NOT MERGE — fly.toml: min_machines_running 1->0 (Telegram webhook mode, Phase 3b)#463
jlunder00 wants to merge 1 commit into
devfrom
feature/infra-webhook-flytoml

Conversation

@jlunder00

Copy link
Copy Markdown
Owner

⚠️ DO NOT MERGE until staging cold-start has been verified

This PR is staged for review/CI only. Merging flips prod's fly machine
scale-to-zero behavior — it must not land until the runbook below has been
executed against staging and confirmed successful.

What this does

fly.toml: min_machines_running 1 → 0. Currently the prod fly machine
never suspends because the bot's Telegram long-polling loop
(bot/message_handler.py run_polling, while True getUpdates) holds a
persistent connection open 24/7, which was the deliberate reason
min_machines_running was pinned to 1. Telegram webhook mode removes that
requirement — the machine can suspend when idle and auto_start_machines = true wakes it on the next inbound webhook POST.

Webhook mode itself is not new code — it's already fully wired and
merged:

  • Per-user webhook registration on boot: api/main.py lifespan (~lines
    97-134) iterates users with bot_token_encrypted + webhook_secret set,
    decrypts each token via the vault, calls
    bot/webhook_setup.py:register_webhook().
  • Receiving endpoint: POST /bot/telegram-webhook
    (api/routes/bot.py:166), resolves the user via the per-user
    X-Telegram-Bot-Api-Secret-Token header, dispatches through the same
    handle_message() pipeline as polling.
  • supervisord.conf [program:bot] already switches to sleep infinity
    when TELEGRAM_WEBHOOK_URL is set, instead of starting the polling loop.

A full parity audit (webhook _process_telegram_update vs. polling
run_polling's per-update logic) found no blocking gaps — see investigation
notes shared with the team. This PR is purely the config flip; no source
changes.

Why this needs staging verification first

Risk: if fly's cold start (suspended machine → auto_start_machines wakes
it → supervisord boots api/bot/mcp → app ready) is slower than Telegram's
webhook delivery retry window, inbound messages could be delayed or
dropped. Staging (tether-dev) already runs with min_machines_running = 0 (fly.dev.toml), so it already exercises the suspend/wake path — it's
the natural, zero-risk place to test the exact webhook cold-start flow prod
would go through.

Runbook (for Jason to execute)

Full step-by-step in
cc-context-store/tether/docs/infra/webhook-staging-verification.md.
Summary:

  1. Set TELEGRAM_WEBHOOK_URL on tether-dev, confirm a test user has
    bot_token_encrypted + webhook_secret in telegram_connections.
  2. Redeploy staging, confirm webhook registration in boot logs and that
    the bot process logs "Webhook mode active" (no polling loop started).
  3. Let the staging machine actually suspend (idle, don't poke it).
  4. Send a real Telegram message; confirm: machine wakes, webhook POST gets
    a 200 within Telegram's retry window, pipeline runs, a reply is sent,
    and the Postgres pool resumes on demand (ties to the Part A pool fix,
    PR fix(db): pool min_size=0 + idle lifetime so managed PG can scale to zero #462).
  5. Repeat 2-3x across different idle durations for confidence.

Success → merge this PR, deploy prod.
Failure (lag/drops/unreliable wake) → do not merge. Prod stays at
min_machines_running = 1. Part A (PR #462, pool min_size=0) already
fixes the Postgres compute-quota drain independent of this and is
unaffected either way.

Test plan

  • CI green (no source changes, config-only)
  • Jason: staging runbook executed, cold-start confirmed within
    Telegram's retry window, DB resumes on demand
  • Only then: merge + deploy prod

…se 3b)

Lets the fly machine itself suspend when idle instead of staying up 24/7
for Telegram bot long-polling. Webhook mode is already fully wired
(per-user secret registration in api/main.py lifespan, POST
/bot/telegram-webhook handler, supervisord [program:bot] already switches
to sleep infinity when TELEGRAM_WEBHOOK_URL is set) — this is the last
config flip.

DO NOT MERGE until staging cold-start is verified. Runbook:
cc-context-store/tether/docs/infra/webhook-staging-verification.md
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant