Sinker

Sinker watches every commitment stage of a Solana transaction in real time — submitted, processed, confirmed, finalized, and every failure in between. It routes through Jito, but the problem it solves is bigger than Jito: most tools give you one bit of information. Sinker gives you the full picture, and an AI agent that acts on it.

Fair warning — The README it's a long one. But every word is mine and every number came from a real mainnet run. Not AI slop, promise. Route through the Table of Contents for better navigation. Enjoy the read.

Research paper · Demo playlist

---

Contents

• What actually happens to a bundle
• Architecture
• Backpressure & stream resilience
• WebSocket slot source (experimental)
• Vigil
• Synapse — a two-speed nervous system
• The three questions
• Decisions & tradeoffs
• What we got wrong
• Failure handling
• Verify it yourself(lifecycle log)
• SDK
• Setup
• Known gaps
• Architecture document
• Dashboard

---

Sinker watches all seven from a live Yellowstone gRPC stream, classifies every failure at the source, and hands every operational decision to an embedded AI agent. Below is that agent working on mainnet — reading a live auction floor and bidding its own tip up across four real rejections, each escalation its own reasoned call:

When the floor sits calm the agent clears in the 1–10K-lamport range; when it spiked here, the agent climbed with it — autonomously, no human in the loop. It doesn't always clear on the first bid; it adapts until it does. And because most submissions need no judgment at all, it runs at two speeds — a reflex that skips the model entirely, the full reasoning tier only when something breaks — so the LLM's latency is paid only where it earns a decision. Across three weeks on mainnet-beta (May 31 – June 22) it landed 49 bundles to finalized — a sample (with the failures) is listed below, the full run in the lifecycle log; every claim here is measured, every slot one you can go check.

---

What actually happens to a bundle

Almost all the danger lives between that HTTP 200 and finalized — and the lie we opened with is only the first of six ways a bundle dies.

A bundle crosses three independent network components and five commitment stages before it's irreversible, and each stage has its own failure mode that demands its own recovery — a fee_too_low at the auction is a different problem from a slot_skip at the leader, which is different again from a compute_exceeded on-chain. The silent drop is just the loudest: that HTTP 200 for a bundle that was never forwarded, exposed ~750 ms later as Invalid. Sinker classifies each one at the source — where a failure is born is what dictates the fix — before the agent decides what to do.

---

Architecture

Two layers, one hard boundary: the Rust core never calls an LLM, and the agent never touches a keypair.

Rust sidecar (localhost:7777) owns everything that touches the chain — Yellowstone slot streaming, Jito bundle construction, commitment tracking, lifecycle persistence. No LLM calls, no policy.

TypeScript agent (Synapse) owns everything that needs judgment — tip sizing, retry strategy, failure diagnosis — across two tiers, a deterministic reflex and the reasoning cortex. No keypair, no direct Jito calls.

That boundary isn't a runtime flag you could flip — it's structural. The submission path in Rust has no entry point a decision could bypass; everything flows through one agent-authenticated endpoint. The stream senses and never submits; every bundle that reached Jito was explicitly authorized by the agent after reading live tip-floor data, queue state, and prior outcomes. Swap MODEL_PROVIDER and the execution layer doesn't change — Anthropic, OpenAI, Groq-Llama, xAI, or a local Ollama model for air-gapped operation.

---

Backpressure & stream resilience

A long-lived gRPC consumer has two ways to die under load — a slow disk and a dropped connection. Sinker is built against both.

No backpressure onto the stream. Every lifecycle write is an append to a JSONL log — and disk is the one thing on the hot path that can stall. If the read loop wrote inline, a slow fsync would block it from pulling the next slot event: the disk would backpressure the network read, and Sinker would start measuring its own I/O latency instead of the chain's. Instead, the read loop hands each write to a dedicated writer task over an unbounded channel and returns instantly. Disk latency stays on the writer's side of the channel; if storage stalls, the queue absorbs it while the gRPC stream keeps draining at full rate. The unbounded buffer is the deliberate trade — a little memory under a stall, in exchange for never throttling the reader (a throttled reader would miss the very slot timing the system exists to capture).

Reconnection. The connect → subscribe → read cycle runs inside an outer loop with exponential backoff (500 ms → 30 s, reset on the first healthy message). A dropped or unreachable stream re-sends the same two-filter subscription and resumes, rather than taking the sidecar down with it.

State survives the gap. The in-flight tracking maps (slot_watch, tip_watch) are held across reconnects — so a bundle whose tip already landed during an outage still receives its confirmed and finalized stamps once the stream comes back. The reconnect is invisible to the lifecycle record.

---

WebSocket slot source (experimental)

Branch: feature/ws-slot-source

Sinker's slot stream normally comes from Yellowstone gRPC — sub-50 ms first-shred latency, native leader identity on every event. But gRPC is a managed service with real outages (the June 2026 incident that motivated this work is in the engineering log). A second, independent source is available via --slot-source:

Flag	Behavior
--slot-source yellowstone	Yellowstone gRPC only. Default — no behavior change.
--slot-source websocket	WebSocket only via vigil. No Yellowstone subscription required.
--slot-source dual	Both simultaneously. Yellowstone primary; vigil runs silently and takes over on disconnect.

All three modes emit the same SidecarEvent::SlotUpdate on the internal event bus — the agent, bypass mode, and dashboard see no difference in the signal.

websocket uses vigil, a standalone Rust crate built for this project that subscribes to slotsUpdatesSubscribe on any standard Solana RPC endpoint. Leader identity comes from a local ChaCha20 schedule — the same deterministic algorithm every validator runs — bootstrapped from one getVoteAccounts call per epoch, zero hot-path RPC after that. Jito validator detection is pulled from the Kobe API at startup. No Geyser subscription, no extra credentials.

dual is the resilience mode. Yellowstone runs as a background task; vigil keeps the foreground alive. A dedup guard ensures only the faster source wins each slot — when Yellowstone is healthy it wins every race; when it stalls, vigil takes over automatically with no config change and no restart.

---

Vigil

vigil is a standalone Rust crate written specifically for this project — a full slot-sensing stack built against Solana's native WebSocket pubsub, not a wrapper around existing tooling.

→ vigil crate README

---

Synapse — a two-speed nervous system

The bounty asks for an AI operator. Ours runs at two speeds — a reflex that skips the model entirely on the calm majority, and a cortex that reasons when something actually breaks — so the LLM's latency is paid only where it buys a decision.

What it decides — and what it can't

Synapse owns every operational call: how much to tip, when to submit, how to read a failure, whether to retry. It owns none of the execution — it never holds the keypair, never signs, never calls Jito directly; it issues commands over a localhost API and the Rust core does the dangerous part. Because that boundary is a network contract (see Architecture), the model is swappable: one env var moves you across Anthropic, OpenAI, Groq, xAI, or a local Ollama with zero change to the execution layer.

One signal, two speeds

Most submissions need no judgment at all. A stable floor with a fresh transaction has exactly one correct move — bid the known clearing floor — and paying an LLM to "decide" it is pure latency for nothing. So Synapse is modelled as a nervous system. Every cycle hits a gate (decideCycle) that routes it to the cheapest tier that can decide it correctly:

• Reflex (System 1, ~0 ms, no model) — like the spinal reflex arc that bypasses the brain: when every trivial-case check holds, it bids the recent clearing floor deterministically.
• Cortex (System 2, the LLM) — where deliberation lives. Any ambiguity — an unclassified failure, a moving floor, a near-deadline tx, a cold start — escalates here.

Same logic as biology: skip the cortex's latency on the common case, spend it only where a decision needs a brain — so in steady state the LLM isn't on the path at all. It's a runtime toggle, not a fork (POST /internal/mode; Cortex is the default).

Reflex — fast, and free to be wrong

The gate is deliberately conservative: any doubt routes to the Cortex, because a false "trivial" is cheap to be wrong about. A misjudged reflex bid simply fails — and a failed Jito bundle pays no tip (Invalid), then fires bundle_settled(failed), which re-enters as a settled-failure signal and routes its own retry to the Cortex. So the reflex can be optimistic by construction; the cortex is its backstop, at zero lamport cost. That self-correcting loop is the amber path in the diagram.

Cortex — woken by work, never by a clock

The Cortex is an event-driven ReAct loop — one model call per cycle — woken by exactly three things: a new transaction, a settled failure, and a 30 s watchdog for anything missed during a reconnect. Not by slot ticks: an earlier design treated each ~400 ms tick as work and burned tokens doing nothing. The loop pre-fetches state, tip floor, history, and queue in one parallel batch (~50 ms), so a cycle is two judgment calls (submit + writeTrace), not six.

The tip ladder

The intelligence is in the bidding. The agent opens low — at the lowest tip recent history shows landing (often the observed minimum, ≥ Jito's 1,000 L floor) — because a fee_too_low rejection costs nothing but information. It escalates only on that free failure, climbing one tier per consecutive fee_too_low (p50 → p75 → p95 → p99). It isn't hardcoded: Rust exposes the raw percentiles {p50 … p99, ema} and executes whatever tip comes back — the policy lives in the prompt as judgment. The escalation hero at the top of this README is that ladder firing under a spiking floor; the discipline is opening cheap when the floor is calm. The finalized tip range (1,191 → 216,445 L) is the market talking, not a fixed setting. (It wasn't always this disciplined — What we got wrong has the entry where it overpaid.)

The tax, measured

The reason the two tiers exist at all: we measured the LLM's cost on the critical path, decomposed into decision time (the agent thinking) and settle time (the chain, identical for everyone). The settle is the same whether a human, a script, or a model submitted — so the only thing the AI layer is responsible for is decision_ms:

Reflex's decision time matches a no-AI fixed-tip submit (374 ms vs naive's 346 ms — noise) and ties its cost (1,000 L); Cortex's entire ~4 s penalty is LLM think-time, nothing on-chain. The fast-path doesn't approximate no-AI speed — it is no-AI speed, with the model held in reserve.

And here it is over a live run — 15 submissions in Reflex mode with three fee_too_low faults injected, so you can watch System 1 hold a flat baseline and System 2 fire only on the exceptions:

Twelve fast-path submissions sit at ~zero decision time; each injected fault escalates to the LLM, which lifts the tip above the floor and lands — and the very next transaction is back on the fast-path. (The escalation bars are full recovery time — the failed reflex attempt plus the LLM reclassify-and-resubmit — not the bare LLM cost.)

Auditable by construction

The reasoning isn't a log line added for show — it's the agent's own reason field, written verbatim beside the slot number on every bundle:

"Bundle reached confirmed stage (872ms to processed, 277ms to confirmed). No failure. p75=6388L worked as expected given stable history. No change to strategy needed for next cycle."

"Retry submitted at p99=100000 after fee_too_low at p95=100000. Requeued tx ba2bdabb… Escalation should clear the fee floor."

"First-cycle submission at p75=3996. Two txs bundled. Leader changed mid-cycle. Stage remains submitted; awaiting bundle_settled SSE. Will escalate to p95 on next Invalid(fee_too_low)."

Synapse wasn't always this lean — it started as a three-agent graph we cut down, the retry path hid a real race, and the bidder once overpaid until we measured it; all three are in What we got wrong. The per-failure-class recovery rules live in Failure handling.

---

The three questions

The bounty asks these specifically. Each answer leads with the bottom line, then the evidence — and every number is something we measured on mainnet, not a textbook constant.

What does the delta between processed_at and confirmed_at tell you about network health at the time of submission?

▶ The 16 bundles behind this graph — view & verify on Solana + Jito explorers » · raw

It's the wall-clock cost of consensus — from a block existing (processed) to two-thirds of stake voting it confirmed. Your transaction doesn't move it; it's a live read on the validator set's health.

Across these 16 bundles it was tight — median 367 ms, p95 506 ms — about 0.92 of Solana's ~400 ms slot, i.e. two-thirds of stake voted before the next leader finished its slot: a healthy, uncongested network. But the diagnostic is the spread, not the number: the delta is gossip + replay + vote-landing across the whole set, so it widens the instant any of those degrade — before you see missed slots or failed landings. A sustained drift past ~2 slots is real trouble (partition, validator overload, a fork splitting stake) — exactly where the agent's TTL-aware escalation earns its keep.

And it can't be measured after the fact — the chain never records when a block crossed processed→confirmed; it exists only at the instant of observation, which is why Sinker reads it from a Geyser stream, not RPC polling.

Why should you never use finalized commitment when fetching a blockhash for a time-sensitive transaction?

Short answer: finalized runs ~32 slots (~12 s) behind the chain tip, so a blockhash fetched there arrives with ~32 of its 150 validity slots already spent. A time-sensitive transaction — racing to land in a tight window, often with retries — is exactly the case that can't spare that runway. So we fetch at confirmed (~2 slots behind) instead.

The bar above is that budget. A blockhash lives for 150 slots (~60 s); fetch at finalized and ~32 are already spent before you submit, leaving ~118 — while confirmed lags only ~2 and keeps ~148, roughly 30 more slots of runway. For a transaction with time to spare that gap is harmless; for a time-sensitive one it's the margin you're racing against.

That ~12-second finalized lag isn't a textbook constant — we measured it ourselves: a median of 12.1 s (≈30 slots) across 16 mainnet bundles ». And the budget drains faster than the bar suggests: a bundle that retries and refreshes its blockhash burns more slots each cycle, so the thinner finalized runway is the first to run out.

So we fetch at confirmed. The only price is a confirmed-but-not-finalized reorg — negligible on mainnet (<0.1% of slots) — a trade we take for the runway every time.

What happens to your bundle if the Jito leader skips their slot?

Short answer: nothing executes. The bundle was never included in a block — that's Invalid, not Failed — so Sinker labels it slot_skip and re-sends it unchanged at the same tip, and the block engine carries it to the next Jito leader. Why that's the right move, below.

The subtlety is why. A skipped slot produces no block, so the bundle didn't fail — it never ran, and those demand opposite fixes. getInflightBundleStatuses tells you which world you're in: a skip reads Invalid (never included), not Failed (landed, then reverted). That's the gap between safe to resend untouched and do not retry — you'll double-execute or burn another tip. Invalid is noisy at first, so only a persistent one with nothing on-chain earns the slot_skip label. (And it had to be a Jito leader that skipped — Sinker only submits into an open Jito window; otherwise it holds.)

The fix is the easiest in the policy: resend at the same tip. No requeue (the bytes never left the queue), no higher tip (the auction didn't reject you) — just re-send, and the block engine carries it to the next Jito leader, a slot or two away. We don't drop to a normal transaction: that throws away the atomicity and MEV protection that were the reason to bundle.

Honest caveat: we infer the skip (a persistent Invalid that never lands) rather than confirm it. Yellowstone emits SLOT_DEAD for real skips; correlating those with our submission slots would make it proof — scoped, not yet wired.

---

Decisions & tradeoffs

Most of these weren't chosen on a whiteboard — they were forced by how Solana and our providers actually behave, and each cost us something we decided was worth paying.

Confirm the tip, not the bundle

We ran on Jito's getInflightBundleStatuses for everything. It answers one question well — did the bundle land? — but it's the wrong instrument for when did it cross each commitment level: it never exposes processed at all, and as a pull API its timestamps measure our poll cadence, not the network's propagation. The damage was quiet but total — processed_at was null across all 263 logged bundles, and confirmed_at got overwritten on every poll, collapsing a real ~12 s confirmed→finalized interval into a phantom ~2.2 s.

The fix came from atomicity. A Jito bundle lands all-or-nothing, so if the tip transaction reaches a commitment level, every tx in the bundle did too — and we already sign that tip tx on every submit. So we stopped tracking the bundle and started watching one signature on the Yellowstone stream (a static account_include = [sidecar_pubkey]), slot-joined to the slot lane for processed / confirmed / finalized with zero added RPC. It also sidestepped the rate limit — polling burned up to ~50 HTTP calls per bundle against Jito's 2 req/s quota; the stream costs none. We don't confirm the bundle; we confirm the tip, and the tip speaks for the rest. Cost: it leans on Jito atomicity (which holds by construction) and on the stream being up — if it drops, we fall back to polling.

A rolling leader window, not the whole epoch

The leader schedule looked like a one-line getLeaderSchedule call — until that call, a ~3 MB dump of all 432,000 epoch slots, got throttled and severed mid-body by our provider on every attempt, even at a 120 s timeout. We switched to getSlotLeaders(start, 5000): a window where array position is the slot offset, so a lookup is queue[slot − head_slot] — O(1), no inversion, no epoch-boundary special case, refetched in the background ~500 slots before exhaustion so there's never a gap. The honest part: getLeaderSchedule actually moves 5× less data over a full epoch (one call vs ~96). Cost: more calls — bought with 87× less RAM, no single point of failure, and epoch boundaries crossed transparently. Forced by the timeout; better architecture regardless.

An in-memory queue — durability would be a bug

The intake queue holds pre-signed transactions waiting to be bundled, and the reflex for anything "production-grade" is to back it with a database so a restart loses nothing. For a Solana transaction queue that reflex is a correctness bug wearing a safety blanket. A blockhash dies after ~150 slots (~60 s), so any transaction that survived a restart would come back already past expiry — rejected by the runtime on arrival. The caller has to re-sign with a fresh blockhash regardless, so a durable queue would faithfully hand back transactions guaranteed to fail, dressed up as recovered work. So the queue stays in memory and evicts each entry at 50 slots — well short of the 150-slot cliff — so a tx is always bundled with a blockhash that has ample life left, and one that ages out fires a TxExpired event instead of rotting silently. Cost: a restart drops the queue — which is exactly right, because those transactions were already dead; the caller re-submits a freshly signed one, as it would have had to anyway.

Wait for the next Jito leader, not a fallback transaction

About 2.5% of slots are led by validators not running jito-solana, where a bundle simply can't land — and the textbook fix is to drop to a regular sendTransaction for those slots. We measured the case and chose not to. Across two epochs, Jito coverage held at 97.4–97.5% (707 of 735 validators, then 705 of 731), so a Jito leader is almost always within a 4-slot lookahead — for a non-Jito window, waiting for the next Jito leader costs ~1 second at most. A sendTransaction fallback, by contrast, is a whole second submission path — its own retry loop, its own priority-fee tuning — and it throws away the atomicity and MEV protection that were the reason to bundle. Paying that to save ~1 second on 2.5% of slots is a bad trade; the gap is a wait, not a fork. Cost: leader detection is still a coverage proxy today, so those rare slots currently surface as a dropped bundle rather than a clean hold — precise detection (and the fallback itself, if a latency-critical caller ever needs it) is documented, not built.

Two speeds, not one — pay the model only when it pays off

An LLM on every submission is the obvious design, and it's wrong: we measured the tax at ~4 s of decision latency per cycle on a slot-sensitive path, where most cycles have one obvious correct move. So Synapse runs a deterministic Reflex on the calm majority and reserves the Cortex (the model) for ambiguity — see the agent layer. The objection writes itself: a hardcoded fast-path will misjudge and lose a transaction. It can't, cheaply — a wrong reflex bid fails at the auction, pays no tip (Invalid), and the resulting bundle_settled(failed) routes its own retry to the Cortex. The cost of optimism is bounded at one free failure, and the gate is conservative enough (seven trivial-case conditions, any one routing to the model) that it rarely pays even that. Cost: the fast-path can be wrong — but only ever for free, and the model is one POST /internal/mode away as a runtime backstop, not a redeploy.

And none are round numbers we liked — each falls out of a measurable property of the network:

Parameter	Value	Why it's that number
Leader lookup	O(1) · 562× refetch margin	queue[slot − head_slot] on the rolling window; at ~450 ms/slot the buffer holds ~562 fetch-lengths of warning before it can run dry.
Blockhash commitment	confirmed · +~30 slots runway	150-slot validity; finalized is born ~32 slots stale, confirmed only ~2.
Transport	SSE · ~500 ms saved/event	Polling at interval P costs P/2 expected wait; loopback SSE is sub-millisecond — ~500 ms removed at P = 1 s.

---

What we got wrong (and measured our way out of)

The most instructive parts of building Sinker were where a clean idea met an uncooperative network and lost. Each story is the same arc — symptom → measurement → root cause → fix — and more than one falsified a hypothesis we'd already shipped.

The Invalid red herring

For weeks, every bundle came back Invalid within ~750 ms — 100% of them, across tips from 1K to 100K, multiple engines, multiple shapes. We proved the transaction innocent four ways (60 s of polling: always Invalid; getSignatureStatuses: null; simulateBundle: "succeeded"; plain sendTransaction: landed in 2 s). It was always the routing layer.

So we shipped a hypothesis — the leader-lookahead was too tight to catch a Jito leader, so we gated submission to imminent ones — then measured what it depended on and watched it collapse: Jito covers 97.4–97.5% of slots, so a leader is almost always inside a 4-slot window. A window that wide can't produce a 100% failure.

The real cause was humbler: the public block engine deprioritizes anonymous searchers before it ever checks the leader schedule. One credential fixed it — a Jito UUID for authenticated 2 req/s access; the first authenticated bundle finalized in ~32 s (slot 422,977,318, still on the explorers). The leader-gate survived only as a diagnostic the agent still reads — the lesson outliving the bug: write every hypothesis into the same log as the outcomes, so a wrong one stays falsifiable instead of quietly load-bearing.

From a three-agent graph to one ReAct loop

The first agent design was a multi-agent graph — a tip oracle, a slot watcher, a failure reasoner — gated by a hand-tuned "network health score." The orchestration added moving parts without improving decisions; the sub-agents mostly shuffled the same context between prompts, and the score was an opaque scalar standing in for reasoning we could have just read. We collapsed it to a single event-driven ReAct loop whose system prompt carries the tip policy, failure taxonomy, and retry rules, and deleted the health score entirely. The honest version of this project documents the simplification — not the architecture diagram we didn't keep.

The retry that silently dropped

A genuine concurrency hazard: bundle_settled(failed) can fire while the original submit cycle is still running. A naive handler hits if (runningCycle) return and silently discards the retry context — the failure is seen but never acted on, the worst kind of bug because nothing errors.

} finally {
  state.runningCycle = false;
  if (state.pendingRetry) {
    const ctx = state.pendingRetry;
    state.pendingRetry = null;
    await maybeRunCycle(ctx.slot, state, ctx);   // drain immediately, zero added delay
  }
}

The lesson: event-driven retry loops need explicit queue semantics. "Return early if busy" quietly means "drop the most important event."

The agent that overpaid

The agent landed everything — and quietly overpaid doing it: it opened most bids at p75 (~8K L) when 2,000 L cleared 100% of the time, ~4× the going rate, invisibly. The bug was in the prompt, not the model — its tip policy defaulted to p75 and read the paid distribution (what others bid) rather than what actually lands; overpayment is silent where a fee_too_low is loud, so nothing ever flagged the waste.

Because the tip policy lives in the prompt as judgment (Rust just executes the number it returns), the fix was a prompt rewrite, not a code change — fix(agent): rewrite tip strategy: open at the lowest tip recent history shows landing and escalate only on a fee_too_low, which pays no tip — so a rejection is free information, not a loss. Result: 8,058 → 1,073 L per landing, same landing rate.

The benchmark that flattered us

The benchmark reported the fast-path beating a no-AI submit by ~600 ms — impossible, since Reflex has more hops than the naive arm. The wall-clock land_ms hid two bugs (above): a 1,500 ms poll quantized the landing to whichever poll caught it, and the naive arm awaited its own blocking submit inside the timer while the agent arm submitted async, off it.

The fix: stop trusting the script's stopwatch and read the sidecar's own timestamps (submitted_at, submitted_to_processed_ms, …), independent of poll cadence. The phantom gap vanished — Reflex ≈ naive, as it must be — and the real signal, the LLM's ~4 s decision_ms, stood out. A benchmark is a hypothesis too, and an unfalsifiable one flatters whoever wrote it.

---

Failure handling

Sinker classifies every non-confirmation at the source — where a failure is born, not just that one happened — because each class demands a different recovery, and guessing wrong wastes a tip or double-executes.

There were 41 classified non-confirmations across the run — but the raw number lies, so here's the honest cut. Only 10 were a transaction that genuinely reverted on-chain (bundle_failed). 14 were fee_too_low — the auction telling us to bid higher, which is the system working: the agent re-bid a tier up. 6 were deliberate fault-injection (synthetic, never real submissions). And 10 were our own post-send false-failures — the known defect where the sidecar stamps Failed after the bundle was already propagated, so it may actually have landed. We surface those rather than filter them out; an audit log that hides its own defects isn't an audit log.

Each class maps to a distinct recovery, encoded in the agent's prompt. The three that actually fired on this run are marked ✓; the other two are playbook the run never needed:

Failure	Fired	Agent action
fee_too_low	✓	getRawTx → requeue → submit one percentile tier higher
bundle_failed / AlreadyProcessed	✓	inspect raw_error — AlreadyProcessed = already landed → trace as success; InstructionError = don't retry
slot_skip	✓	resend at the same tip — never ran, no escalation needed
expired_blockhash	—	getRawTx → requeue → submit at the same tip (Rust attaches a fresh blockhash every submit)
compute_exceeded	—	write trace, hold — the instruction is broken, retrying won't help

The money side tells the same story. Finalized bundles cluster low — median ~5,800 lamports, almost all under 11K — because the agent bids the floor, not the ceiling, when the market is calm. The points that climb into the 100K–1M range aren't the system failing: they're a genuinely spiking floor (the organic escalations), the fault-injection tests, and the false-failures. Lost transactions live in exactly one column, and it's a small one.

---

Verify it yourself

The full lifecycle log is lifecycle.jsonl, in the repo root — every bundle's slot, commitment progression, timestamps, tip, and failure class, append-only. The bounty asks for a log of at least 10 real submissions including 2 failures; this one holds far more. Below is a representative 20 — 15 that landed and 5 that didn't. Every slot links to its real mainnet-beta block (how the bounty says judges confirm the stack ran on real infrastructure); landed bundles add Jito + Solscan links, and the five failures are recorded, with their class, in the log.

#	Slot	Submitted (UTC)	Tip (L)	→proc	→conf	→final	Outcome	Verify
1	427,952,638	2026-06-21 13:44	2,253	361ms	544ms	11.8s	finalized	◇ Jito · ◎ tx
2	427,956,339	2026-06-21 14:09	1,789	281ms	264ms	13.0s	finalized	◇ Jito · ◎ tx
3	427,956,733	2026-06-21 14:12	2,160	231ms	394ms	12.0s	finalized	◇ Jito · ◎ tx
4	427,957,022	2026-06-21 14:14	2,710	372ms	185ms	11.7s	finalized	◇ Jito · ◎ tx
5	428,137,602	2026-06-22 10:10	10,000	421ms	320ms	12.8s	finalized	◇ Jito · ◎ tx
6	428,137,658	2026-06-22 10:10	10,000	374ms	302ms	12.6s	finalized	◇ Jito · ◎ tx
7	428,137,713	2026-06-22 10:11	5,000	413ms	385ms	12.3s	finalized	◇ Jito · ◎ tx
8	428,143,857	2026-06-22 10:52	9,069	397ms	317ms	12.1s	finalized	◇ Jito · ◎ tx
9	428,143,880	2026-06-22 10:52	9,069	604ms	480ms	12.6s	finalized	◇ Jito · ◎ tx
10	428,144,755	2026-06-22 10:58	2,957	382ms	558ms	12.8s	finalized	◇ Jito · ◎ tx
11	428,144,808	2026-06-22 10:58	6,000	498ms	312ms	12.7s	finalized	◇ Jito · ◎ tx
12	428,144,890	2026-06-22 10:58	6,000	2,049ms	421ms	12.1s	finalized	◇ Jito · ◎ tx
13	428,144,945	2026-06-22 10:59	2,483	761ms	379ms	13.7s	finalized	◇ Jito · ◎ tx
14	428,144,995	2026-06-22 10:59	4,291	560ms	276ms	11.8s	finalized	◇ Jito · ◎ tx
15	428,145,134	2026-06-22 11:00	2,400	514ms	255ms	11.9s	finalized	◇ Jito · ◎ tx
16	428,145,187	2026-06-22 11:00	7,483	570ms	427ms	12.1s	finalized	◇ Jito · ◎ tx
17	428,145,236	2026-06-22 11:01	6,388	863ms	367ms	11.6s	finalized	◇ Jito · ◎ tx
18	428,433,144	2026-06-23 18:56	2,000	1,355ms	—	—	slot_skip	lifecycle.jsonl
19	428,477,984	2026-06-23 23:55	1,000	487ms	—	—	slot_skip	lifecycle.jsonl
20	424,988,734	2026-06-07 22:58	2,228	—	—	—	fee_too_low	lifecycle.jsonl

---

SDK

The sidecar exposes a TypeScript SDK so any app can submit bundles without touching Rust:

import { Sinker } from 'sinker-sdk';

const sinker = new Sinker({ url: 'http://localhost:7777' });
const tx = await sinker.enqueue(txBase64);       // pre-signed transaction bytes
const status = await tx.waitFor('confirmed');    // block until 2/3 stake has voted
console.log(status.explorer_links);
// → { jito: "https://explorer.jito.wtf/bundle/...", solana: "https://explorer.solana.com/tx/..." }

Supports Anthropic, OpenAI, Groq, and xAI as the agent model provider — swap MODEL_PROVIDER in .env and nothing else changes.

---

Setup

Prerequisites: Rust (stable) · Node.js 20+ · a funded keypair · a Yellowstone gRPC endpoint · a Jito UUID · an LLM API key

.env at the repo root:

# chain
RPC_URL=https://mainnet.helius-rpc.com/?api-key=YOUR_KEY
KEYPAIR_PATH=/path/to/keypair.json
YELLOWSTONE_ENDPOINT=https://your-grpc:10000
YELLOWSTONE_X_TOKEN=your-token

# jito
JITO_BLOCK_ENGINE_URL=https://mainnet.block-engine.jito.wtf
JITO_UUID=your-jito-uuid

# agent
MODEL_PROVIDER=xai
XAI_API_KEY=your-key

# optional
# API_PORT=7777
# BYPASS_AGENT=false
# AGENT_DECISION_MODE=cortex   # cortex (LLM every cycle, default) | reflex (fast-path + LLM on exception)
# SLOT_SOURCE=yellowstone       # yellowstone (default) | websocket | dual
# WS_ENDPOINT=                  # derived from RPC_URL (https→wss) if not set

MODEL_PROVIDER takes xai · anthropic · openai · groq (set the matching API key). JITO_UUID is what grants authenticated 2 req/s access — without it the public block engine deprioritizes anonymous searchers (every bundle returns Invalid); see What we got wrong.

# install
cargo build                                          # Rust sidecar
cd agents && npm install && cd ..                    # Synapse agent

# run — three terminals
cargo run                                            # 1 · Rust sidecar (localhost:7777, Yellowstone)
cargo run -- start --slot-source websocket           #   · or: WebSocket only (vigil, no Yellowstone needed)
cargo run -- start --slot-source dual                #   · or: both — vigil as silent fallback
cd agents && npm run dev                             # 2 · Synapse agent
cd agents && npx tsx --env-file=../.env scripts/enqueue_test_tx.ts   # 3 · submit a test tx

Decision mode — Cortex or Reflex. Cortex (the LLM reasons every cycle) is the default, so the commands above already run it. To boot the fast-path instead, seed the sidecar at start: AGENT_DECISION_MODE=reflex cargo run. Either way the mode flips live, no restart — the agent reads it each cycle:

curl -sX POST localhost:7777/internal/mode -d '{"decision_mode":"reflex"}'   # → fast-path (System 1)
curl -sX POST localhost:7777/internal/mode -d '{"decision_mode":"cortex"}'   # → full reason (System 2)

Fault-injection demo — exercises the full retry loop without waiting on a real failure:

cd agents && npx tsx --env-file=../.env scripts/fault_demo.ts --failure fee_too_low

Injects a synthetic fee_too_low, lets the agent detect it, classify it, requeue the raw bytes, and resubmit at an escalated tip — printing the full reasoning trace and ✅ PASS / ❌ FAIL. Swap --failure for expired_blockhash, bundle_failed, compute_exceeded, or slot_skip.

---

Known gaps

Post-send RPC error ambiguity — when Jito's response body carries an error after the bundle was already forwarded, the sidecar stamps Failed prematurely; the bundle may have landed. The agent handles it correctly (recognizes AlreadyProcessed on retry as success), but the client-facing status is wrong until the next poll resolves it. The clean fix is an intermediate Uncertain state resolved by polling the signature directly — specified, not yet wired.

Non-Jito slot detection — ~2.5% of slots are led by validators not running jito-solana; because leader detection is still a coverage proxy, those bundles are currently submitted anyway and surface via the Invalid path. Planned: precise detection so those windows are held for the next Jito leader instead of dropped (cheap at 97.5% coverage — see Decisions). A sendTransaction fallback is documented but deliberately deprioritized, for the reasons in that section.

Model dependency — the agent is only as sharp as the model behind it. Failure classification, retry sequencing, and tip judgment all ride on the configured model's reasoning; a smaller or cheaper model may misclassify a cause or fumble the multi-step recovery under pressure. The provider-agnostic boundary makes a cross-model comparison cheap to run — we just haven't run it yet.

Throughput ceiling (2 req/s) — submission shares Jito's per-UUID quota. That's ample for sequential and low-volume operation, and moving commitment tracking onto the stream freed the quota for actual submits — but under heavy parallel load it becomes the bottleneck. Horizontal scaling (multiple UUIDs / agent workers) is the path, deliberately unbuilt for the current single-sidecar topology.

---

Architecture document

→ Public architecture & systems paper

---

Dashboard

The dashboard lives in a separate repo — github.com/ubadineke/sinker-web. It's a Next.js app that connects to the sidecar and gives you a live view of everything: bundle status, commitment stages, the agent's decision trace, tip floor, and explorer links. If you want to self-host it or poke around the UI code, that's the place.

---

MIT License