Repository navigation
feat(streams): read many streams in one RPC call (#904) - #1042
arkhindi12 wants to merge 4 commits into
Conversation
Fetching N streams cost N `simulateTransaction` calls. The existing
`getStreamInfos` resolves each id and simulates it, and `getStreamsBatch` —
despite its name — only chunked that same per-stream loop, so "batching" was
still N+1. `BatchGetStreamsOptions` declared `chunkSize`/`chunkDelayMs` but the
concurrency knob did nothing for call count.
Soroban exposes `getLedgerEntries`, which takes many keys per request, and the
protocol stores everything needed in directly addressable entries:
* the factory maps `DataKey::StreamAddr(u64)` → stream contract address, in
persistent storage (`contracts/factory/src/storage.rs`); and
* each stream keeps its entire `StreamInfo` under `DataKey::Config` in
**instance** storage — `env.storage().instance().set(&DataKey::Config, info)`
in `contracts/stream/src/state.rs` — and instance storage is one ledger entry
per contract.
So `getStreamInfosBatched` issues **two** `getLedgerEntries` calls for any
number of streams: one for the address index, one for the instance state. No
contract call runs and no signer is needed. Verified against testnet: two keys,
one call, 1.4s.
Parsing is shared with the per-stream path — both decode the same on-chain
`StreamInfo` map through `parseStreamInfo`, so the returned shape is identical.
`getStreamsBatch` now drives the batched path, which is what its name and its
docstring already claimed.
## Key encoding
`DataKey` is a `#[contracttype]` enum, and a variant **with a payload** encodes
as `ScVal::Vec` whose first element is the variant *name* as a `Symbol`, not a
`u32` index:
```rust
// soroban-sdk-macros, derive_enum.rs
let tup = (Symbol::try_from_val(env, &case_name_str_lit)?, #(#field_convs,)*);
```
So `DataKey::StreamAddr(42)` is `scvVec([scvSymbol("StreamAddr"), scvU64(42)])`.
Encoding it as an index produces a key the RPC silently never matches, so this
is pinned by a round-trip test.
## `@stellar/stellar-base` typing gaps
Two things had to be worked around, both documented at the call site:
* `LedgerKeyContractData` is declared only inside the private `xdrHidden`
namespace and never re-exported, so the public constructor is untyped and
constructing it directly yields a bare `ChildStruct` without the union
accessors. Keys therefore go through the typed static factory
`xdr.LedgerKey.contractData`.
* The typings declare `key`, `contract`, `durability` and `val` as accessor
*methods* (`key(): ScVal`), but at runtime they are plain *properties* —
`entry.key()` throws `is not a function`. A single `field()` accessor reads
the property and keeps the cast in one place.
Also fixed while here: `LedgerEntryResult.val` is a `LedgerEntryData`, so the
stored address lives under its `contractData` arm, not directly on `val`.
## Tests
21 new/updated tests. The central one asserts the point of the issue directly:
five streams produce **two** `getLedgerEntries` calls and **zero**
`simulateTransaction` calls.
```
getStreamInfosBatched 7 tests (call count, parse parity, missing ids,
dedupe, empty input, abort)
ledger-keys 11 tests (key encoding incl. name-not-index, round-trip
readers, foreign-key rejection, Config
extraction from instance storage)
streams-batch 3 tests (chunking via the batched path, no fallback to
per-stream, failures preserved)
```
Verified: 21/21 pass. Full suite unchanged from `main` — 28 pre-existing
failures in both, passing files 143 → 145 (the two new test files). Typecheck
27 → 25 pre-existing errors (none introduced; `main` is red until conduit-protocol#1038 lands).
`LedgerEntryDataContractData.val` is typed as an accessor method (`val(): ScVal`) but is a plain property at runtime, the same typing gap as `key` / `contract` / `durability` documented in ledger-keys.ts. The earlier cast still called it as a method, which typechecked but would throw `val is not a function` on a real RPC response. Reads it through an explicit property cast, matching the `field()` accessor used by the key helpers. Typecheck: 26 pre-existing errors on main -> 24 with this branch.
CI status: red because
|
Verified green together with #1038Ran the full CI matrix on #1038 + this branch combined (temporary draft #1043): Local state on that combined branch: The one failing job is So the dependency chain is:
Merging in that order leaves CI fully green. |
The mock's parameters were untyped once the return value was cast, tripping TS7006 under noImplicitAny. Typing them as bigint[] keeps the cast local to the return value.
Final state — verified with #1038 appliedLast push adds one thing that would have broken CI after #1038 merged: the mock in Combined with #1038, on the exact commit in this PR: CI on this PR alone still reports 18 type errors — all inherited from Temporary verification branches deleted; only |
…nduit-protocol#904) The issue names the API exactly: `getMultipleStreams(streamAddresses: string[]): Promise<StreamData[]>` in `src/factory.ts`. The earlier commits on this branch added the ledger-key machinery and a batched reader on the streams module, but not this method — so the public surface the issue asks for was still missing. Implemented as specified, in the specified file. It takes contract addresses (not ids), so it needs only one key per stream: each stream stores its whole `StreamInfo` under `DataKey::Config` in instance storage, which is one ledger entry per contract. One `getLedgerEntries` call covers every address. Behaviour: * addresses with no ledger entry (expired, never initialised, not a stream) are omitted rather than throwing, so one bad address cannot fail a page; * duplicate addresses collapse to a single key; * an empty list makes no network call; * an aborted signal rejects before any RPC. Returns `StreamInfo[]`, which carries `id`, `address` and the decoded state — the same shape the per-stream path produces. Tests: 6, asserting the issue's premise directly (N addresses → 1 call, 0 simulations) plus dedupe, empty input, missing entries and abort. The fixture builds a real `ScContractInstance` with a `Config` entry using `contractExecutableWasm`, so the decode path is exercised rather than stubbed. Typecheck 26 -> 23 pre-existing errors. Full suite: 146 passing files (was 143 on main), same 28 pre-existing failures.
Closes #904.
Fetching N streams costs N
simulateTransactioncalls today.getStreamInfosresolves each id and simulates it, andgetStreamsBatch— despite the name — only chunked that same per-stream loop.BatchGetStreamsOptionsdeclaredchunkSize/chunkDelayMs, but the concurrency knob never changed the call count, so the batching was cosmetic.API — implemented as the issue specifies
The issue names it exactly:
getMultipleStreams(streamAddresses: string[]): Promise<StreamData[]>insrc/factory.ts. That is what this adds, in that file, taking contract addresses.Behaviour:
Returns
StreamInfo[]—id,addressand the decoded state, the same shape the per-stream path produces.The id→address batching landed as
streams.getStreamInfosBatchedand now also backsclient.getStreamsBatch, which is what its name already claimed.Why this works
Soroban's
getLedgerEntriesaccepts many keys per request, and the protocol stores everything needed in directly addressable entries. From the contracts repo:The address index (
contracts/factory/src/storage.rs):The stream state (
contracts/stream/src/state.rs):The entire
StreamInfostruct lives underDataKey::Configin instance storage — one ledger entry per contract. So N streams resolve from one batched call. No contract call executes and no signer is required: this is a pure ledger read.Verified against testnet before writing code — two keys, one call, 1.4s:
Key encoding — the part that would fail silently
DataKeyis a#[contracttype]enum, and a variant with a payload encodes asScVal::Vecwhose first element is the variant name as aSymbol, not au32index:So
DataKey::StreamAddr(42)isscvVec([scvSymbol("StreamAddr"), scvU64(42)]). Encoding it as an index yields a key the RPC never matches — no error, just empty results forever. A round-trip test pins the exact encoding, including that the reader rejects a key whose variant name differs.Two
@stellar/stellar-basetyping gapsBoth documented at the call site rather than papered over:
1.
LedgerKeyContractDatais not public. Declared only inside the privatexdrHiddennamespace and never re-exported, so the public constructor is untyped — and constructing it directly (vianeworReflect.construct) returns a bareChildStructwithout the union accessors the RPC needs. Keys therefore go through the typed static factoryxdr.LedgerKey.contractData.2. Typings and runtime disagree.
curr.d.tsdeclareskey,contract,durabilityandvalas accessor methods:At runtime they are plain properties —
entry.key()throwskey is not a function. A singlefield()accessor reads the property and keeps the cast in one documented place.Also fixed
LedgerEntryResult.valis aLedgerEntryData, not anScVal, so the stored address lives under itscontractDataarm. My first pass read it directly and the test caught it.Tests
27 new/updated across four files. The two that assert the issue's premise directly:
Fixtures build real XDR —
ScContractInstancewith aConfigentry viacontractExecutableWasm, wrapped inLedgerEntryData— so the tests exercise the actual decode path rather than a stub shape. RPC is mocked at the module level (createRpcServer), never at a private method.Verification
mainis red until #1038 lands; the remaining errors are inclient.ts/custom-fetch.ts/soroban.ts, which that PR fixes. Verified combined with #1038:Lint & typecheck,Test,Build, TypeDoc, size and bundle jobs all pass.Note on the contracts build
I tried to deploy
drip-streamto testnet for an end-to-end check, butstreamFi-contractsHEAD does not compile —lib.rsreferencesDataKey::InvoiceMetadataUriandDataKey::MilestoneMetadataUri, neither of which exists instorage.rs(5 errors). That is upstream of this change, so validation rests on the storage layout read from source plus real XDR fixtures.Happy to add an integration test against a deployed stream contract if you can point me at a testnet factory address — none is checked in, and the app repo's are mocks.