DO NOT MERGE — fly.toml: min_machines_running 1->0 (Telegram webhook mode, Phase 3b) - #463
Open
jlunder00 wants to merge 1 commit into
Open
DO NOT MERGE — fly.toml: min_machines_running 1->0 (Telegram webhook mode, Phase 3b)#463jlunder00 wants to merge 1 commit into
jlunder00 wants to merge 1 commit into
Conversation
…se 3b) Lets the fly machine itself suspend when idle instead of staying up 24/7 for Telegram bot long-polling. Webhook mode is already fully wired (per-user secret registration in api/main.py lifespan, POST /bot/telegram-webhook handler, supervisord [program:bot] already switches to sleep infinity when TELEGRAM_WEBHOOK_URL is set) — this is the last config flip. DO NOT MERGE until staging cold-start is verified. Runbook: cc-context-store/tether/docs/infra/webhook-staging-verification.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR is staged for review/CI only. Merging flips prod's fly machine
scale-to-zero behavior — it must not land until the runbook below has been
executed against staging and confirmed successful.
What this does
fly.toml:min_machines_running1 → 0. Currently the prod fly machinenever suspends because the bot's Telegram long-polling loop
(
bot/message_handler.pyrun_polling,while TruegetUpdates) holds apersistent connection open 24/7, which was the deliberate reason
min_machines_runningwas pinned to 1. Telegram webhook mode removes thatrequirement — the machine can suspend when idle and
auto_start_machines = truewakes it on the next inbound webhook POST.Webhook mode itself is not new code — it's already fully wired and
merged:
api/main.pylifespan (~lines97-134) iterates users with
bot_token_encrypted+webhook_secretset,decrypts each token via the vault, calls
bot/webhook_setup.py:register_webhook().POST /bot/telegram-webhook(
api/routes/bot.py:166), resolves the user via the per-userX-Telegram-Bot-Api-Secret-Tokenheader, dispatches through the samehandle_message()pipeline as polling.supervisord.conf[program:bot]already switches tosleep infinitywhen
TELEGRAM_WEBHOOK_URLis set, instead of starting the polling loop.A full parity audit (webhook
_process_telegram_updatevs. pollingrun_polling's per-update logic) found no blocking gaps — see investigationnotes shared with the team. This PR is purely the config flip; no source
changes.
Why this needs staging verification first
Risk: if fly's cold start (suspended machine →
auto_start_machineswakesit → supervisord boots api/bot/mcp → app ready) is slower than Telegram's
webhook delivery retry window, inbound messages could be delayed or
dropped. Staging (
tether-dev) already runs withmin_machines_running = 0(fly.dev.toml), so it already exercises the suspend/wake path — it'sthe natural, zero-risk place to test the exact webhook cold-start flow prod
would go through.
Runbook (for Jason to execute)
Full step-by-step in
cc-context-store/tether/docs/infra/webhook-staging-verification.md.Summary:
TELEGRAM_WEBHOOK_URLontether-dev, confirm a test user hasbot_token_encrypted+webhook_secretintelegram_connections.the bot process logs "Webhook mode active" (no polling loop started).
a 200 within Telegram's retry window, pipeline runs, a reply is sent,
and the Postgres pool resumes on demand (ties to the Part A pool fix,
PR fix(db): pool min_size=0 + idle lifetime so managed PG can scale to zero #462).
Success → merge this PR, deploy prod.
Failure (lag/drops/unreliable wake) → do not merge. Prod stays at
min_machines_running = 1. Part A (PR #462, poolmin_size=0) alreadyfixes the Postgres compute-quota drain independent of this and is
unaffected either way.
Test plan
Telegram's retry window, DB resumes on demand