Skip to content

Add per-client bid seen tracking - #824

Merged
pk910 merged 1 commit into
masterfrom
pk910/bid-seen-tracking
Aug 6, 2026
Merged

Add per-client bid seen tracking#824
pk910 merged 1 commit into
masterfrom
pk910/bid-seen-tracking

Conversation

@pk910

@pk910 pk910 commented Aug 6, 2026

Copy link
Copy Markdown
Member

Overview

Dora tracks and deduplicates execution payload bids (ePBS), but discarded which clients observed each bid on gossip — a major visibility gap. This PR records per-client bid observations (including first-seen timing) with minimal overhead and surfaces them in the slot page's bid list.

image

How it works

Ingestion: The bid cache (blockBidCache) interns observer client names and keeps a per-bid map of client → first-seen offset (ms from slot start, negative = gossiped ahead of the slot). Duplicate SSE bid events — previously discarded — now record the observation: the hot path goes from "map lookup, drop" to "map lookup, record one entry". Bids extracted from block bodies record no observer, so 0/N genuinely means "nobody saw it on gossip".

Storage (two tiers):

  • block_bids (SQL) only gains two int columns: seen_count / seen_total, stamped at flush time. 0/0 marks pre-feature rows.
  • The full data goes to the blockdb as a self-sufficient per-slot BIDS object (<slot>_bids in S3, namespace 7 in pebble): client-name table + full bid fields + seen bitmask + per-client first-seen offsets (~5 KB/slot). Since entries carry the complete bid, historic bid data could later be served from the blockdb alone, without the relational table. Flushes do read-merge-write so restarts and late-arriving bids union instead of clobber; observations are restored into the cache on startup. Persistence is optional — without a blockdb, counts still work for all slots and the detail view covers cached slots.

UI: The bids table gets a "Seen by" column with a colored 42/50 badge (green = all clients, yellow = majority, red = minority) rendered purely from the SQL columns — no blockdb access on page load. Expanding the callout row lazily fetches /slot/{slot}/bidseen once per page (frontend-cached), showing "Seen by" chips with per-client delays (+380ms) and a "Not seen by" group.

API: seen_count / seen_total added to /v1/slot/{slotOrHash}/bids; docs regenerated.

Config

  • blockDb.pebble.bidsRetention — optional retention for bids objects (pebble mode; tiered mode stores them in S3 only, uncached).

Tooling

  • dora-utils blockdb-copy gains a --no-bids flag and a dedicated raw-bytes bids pass (identical encoding on both backends); the block passes explicitly skip _bids objects / namespace 7.

Overhead

Hot path: one bitset/map insert per duplicate event. Memory: <200 KB for the cache window. Storage: ~5 KB/slot, one blockdb write per slot at flush. Historic slot pages only touch the blockdb when the seen callout is expanded.

@barnabasbusa

Copy link
Copy Markdown
Collaborator

wow this is huge !

@pk910
pk910 merged commit f4ac113 into master Aug 6, 2026
2 checks passed
@pk910
pk910 deleted the pk910/bid-seen-tracking branch August 6, 2026 15:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants