Skip to content

fix(staker): never drop newly-activated orchs from NewRound fanout discovery - #7

Merged
mikezupper merged 1 commit into
mainfrom
fix/orch-profile-discovery-404
Aug 31, 2026
Merged

fix(staker): never drop newly-activated orchs from NewRound fanout discovery#7
mikezupper merged 1 commit into
mainfrom
fix/orch-profile-discovery-404

Conversation

@mikezupper

Copy link
Copy Markdown
Contributor

Problem

GET /api/v1/orchestrators/{address} returns 404 for orchestrators that are demonstrably live on-chain. Observed in prod:

  • 0x428989211607999bc53243e25862b9a2336e2e93 — TranscoderActivated 2026-08-28, earning tickets since
  • 0xfd6fe005f611bdaa195937c7b95e9c1fde118df7 — TranscoderActivated 2026-08-29, earning tickets since

Both have full event history in raw_protocol_events but zero rows in orch_stake_by_round, so the orchestrator_profile matview (which the endpoint serves) has never heard of them.

Root cause

load_known_orchestrators_before returns early with only matview-derived addresses whenever orchestrator_profile is non-empty — the raw lifecycle-event fallback scan only ever ran on a cold start. Discovery of new orchestrators is otherwise in-memory only, and the ProfileFollow loop rebuilds known_orchs from the matview on every iteration.

So at the chain tip: an iteration consumes TranscoderActivated, adds the orch to the in-memory set, advances the checkpoint past it — but no NewRound arrives in that batch (rounds are ~21h apart). The next iteration reseeds from the matview, which lacks the orch, and its activation event is now behind the checkpoint. It is excluded from every future NewRound fanout, permanently. Any orch that first activates mid-round hits this.

Fix

Union both discovery sources instead of early-returning. The event scan gains a contract_name = 'BondingManager' filter (all eight lifecycle event names occur only on BondingManager — verified in prod) so it is served by idx_events_contract_event; it resolves ~1,958 distinct addresses, the same cardinality the fanout already handles.

No API, matview, or migration changes needed: the matview is an unfiltered DISTINCT ON (address) over orch_stake_by_round and the daemon refreshes it every 30s, so one snapshot row suffices.

Deploy behavior

Self-heals: at the first NewRound after deploy (~round 4322), the re-discovered orchs enter the fanout, get their first orch_stake_by_round row, and the 404s resolve after the next matview refresh. For immediate repair, a staker_orch_profile checkpoint replay from before block 499300750 is idempotent.

🤖 Generated with Claude Code

https://claude.ai/code/session_01HPZTFAkTkMusMUe2nQBTUD

…scovery

load_known_orchestrators_before returned early with only matview-derived
addresses whenever orchestrator_profile was non-empty, skipping the raw
lifecycle-event scan. Discovery of new orchestrators is otherwise
in-memory only, so an orch whose TranscoderActivated event was consumed
in a follow iteration that ended before the next NewRound was silently
excluded from every future fanout: no orch_stake_by_round row, no
matview row, permanent 404 on /orchestrators/{address}.

Union both sources instead. The event scan is bounded by
idx_events_contract_event via the new contract_name filter (all eight
lifecycle event names are BondingManager events).

Observed in prod: 0x428989...2e93 (activated 2026-08-28) and
0xfd6fe0...8df7 (activated 2026-08-29) had full event history but no
profile row through rounds 4320-4321. Self-heals at the next NewRound
after deploy.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HPZTFAkTkMusMUe2nQBTUD
@mikezupper
mikezupper merged commit 32c6d68 into main Aug 31, 2026
4 checks passed
@mikezupper
mikezupper deleted the fix/orch-profile-discovery-404 branch August 31, 2026 21:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant