fix: resolve fromActionState/endActionState by chain position, not zkapp_field interning key

[dkijania]

Fixes a defect in the actions query that silently removes real actions from a
fromActionState response, which makes any client folding an action state from that
checkpoint compute a value that can never match the chain.

Three commits, reviewable independently:

	commit	contents
1	05d0662	the failing tests, no production code
2	7514bed	the fix
3	5f686ae	a nightly invariant suite on real data, and the workflow to run it

The defect

getActionsQuery resolved the checkpoint like this
(src/db/sql/events-actions/queries.ts, emittedActionStateCTE):

AND zkf0.id >= (SELECT id FROM zkapp_field WHERE field = ${fromActionState})

zkapp_field is the interning table for every field value in the archive. Its id records
when a value was first written, not where it sits on the chain. The filter assumed those
two orders are the same. They are not: any archive filled out of chain order — a bulk import
through mina-archive-blocks, a hard-fork migration, a bootstrap — writes a de-duplicated set
of new field values and loses their block order.

When the orders disagree the filter drops real actions, or returns actions from before the
requested checkpoint. endActionState had the same defect.

A client that folds from a checkpoint (o1js Reducer.getActions({ fromActionState })) then
computes a value that never matches, and every transaction it builds fails with
Account_action_state_precondition_unsatisfied — deterministically, for that checkpoint, on
every call. Not a race, not archive lag.

Evidence

On a dump of the mesa-rc-1 archive (2026-08-11), the two interning keys for one affected
account are adjacent and swapped:

   id   |          field           |            meaning
--------+--------------------------+-------------------------------
 491051 | 27124406866644488233811… | the action at block 303203
 491052 | 27078409473756655551477… | the checkpoint at block 303199

The checkpoint sits on the earlier block but holds the larger id, so zkf0.id >= 491052
removes the action at block 303203 — a real, canonical, confirmed action.

Scale on that archive: 1 271 damaged checkpoints across 1 003 of 2 778 zkApp accounts. On
the account that failed in production the rate is 1 in 78, which is why this went unnoticed:
almost every checkpoint works, and the damaged one fails every time.

Running this repository's own ActionsService at e353bbc, unmodified, against that dump:

=== unfiltered ===                        33 entries, chain-link breaks: 0
=== fromActionState = checkpoint ===      32 entries, chain-link breaks: 1 at 303199 -> 303207
=== fromActionState = state of 303203 === 33 entries, first entry at height 303199
MISSING BLOCKS: 303203

The third line is the proof that this is key order and not an off-by-one: asking for the state
of block 303203 returns an entry from block 303199, before the checkpoint. Two opposite
errors from one filter.

The fix

Filter on the block height of the checkpoint, resolved for the requested account before the
main query runs. Greater-or-equal keeps the checkpoint's own block in the answer, because o1js
strips it in createActionsList by comparing actionStateOne with the requested value —
excluding it here would change what existing clients receive.

checkActionState is replaced by resolveActionStateBoundary, scoped to the account and to
the chain. The old check only asked whether the value existed anywhere in zkapp_field, so it
accepted action data, app state, or another account's action state, and the server answered
with unrelated data instead of raising an error. That is the second reason this was invisible.

The one design decision worth your attention

The boundary lookup is deliberately not limited to the queried block range. A checkpoint is
a position, not data, so resolving it does not need its block inside the window. Scoped to the
window, an out-of-range checkpoint would resolve to NULL, and height >= NULL selects no rows
— the server would answer with an empty list, which a reducer reads as "no actions to
fold". That would be worse than the current behaviour.

Being below the window does not by itself make the answer wrong:

• if the account emitted no action between the checkpoint and the start of the window, every
  action in the window is still the complete answer, and that is what clients get today. This
  is the common case for a quiet zkApp folding from its genesis action state — the default
  o1js path, because o1js sends no from/to;
• if it did emit actions in that gap, any list we return is missing them.

So the query is answered when provably complete, and rejected with a new
ACTION_STATE_OUT_OF_RANGE error naming the from to use when it is not. endActionState
gets no such exception: it bounds the request from above, so a checkpoint below the window puts
the whole requested span below it, and an empty answer would be silently wrong.

Happy to change this to "always reject" — it is a small change, and tests for both behaviours
are already written.

Tests

A local network writes blocks in chain order and therefore cannot reproduce this. The
fixture inverts the interning order on purpose. Three accounts dispatch identical actions in
identical blocks and differ only in that order:

Account	Interning order	Purpose
inverted	id(S3) < id(S2) < id(S4) < id(S1)	strong inversion
control	id(S1) < id(S2) < id(S3) < id(S4)	natural order
adjacent	id(S1) < id(S3) < id(S2) < id(S4)	the shape measured on mesa-rc-1

control passing is what makes the suite trustworthy: any change that makes the other two pass
by weakening the expectations breaks control too. The failing set was derived by hand from
the interning ids before the tests were first run, and all 34 outcomes matched.

At commit 1 the suite is 18 pass / 15 fail / 1 todo. At commit 2 it is 36 pass / 0 fail / 0
todo — the todo is resolved and replaced by three explicit cases for the decision above.

Commit 3 adds the real-data counterpart under tests/devnet-dump, reusing the existing dump
download so no new data infrastructure is introduced, plus the nightly workflow (nothing runs
test:devnet-dump automatically today). Because the dump rotates hourly it asserts invariants
only — no expected values, subjects discovered rather than named, deterministic ordering, and
finding nothing to test is a failure rather than a silent pass. Verified to have teeth: with
the old ActionsService injected it fails and names the missing block; with the fixed one it
passes over 12 accounts and 40 checkpoints.

Validation on real data

Every action state of 20 accounts with a known inversion, used as a checkpoint and compared
against the unfiltered list:

	checkpoints checked	wrong suffix returned
before (e353bbc)	61	48
after	61	0

For the account behind the original incident: 32 entries with a chain-link break and block
303203 missing becomes 33 entries with no break and nothing missing.

Existing suites unchanged: 83 unit tests, 25 integration tests. npm run lint clean. All CI
checks pass.

Note for reviewers

The postgres service in the new nightly workflow is pinned to 17 to match the major version
the production dumps are taken from — an older server rejects statements the newer pg_dump
emits. This is the first time the devnet-dump suite will run in CI at all, so the first
scheduled run may need attention; disk space for the dump is the other likely snag, and the job
frees a few GB up front.

Related

o1-labs/o1js#2905 — createActionsList never compares actionStateOne[i] with
actionStateTwo[i+1], so it re-seeds its fold from each block and cannot detect a dropped
block. That is why this fault was silent on the client side. Worth fixing independently of this
PR: it would make any archive fault of this class loud, on any endpoint, for every zkApp.

[dkijania]

Update: the fix is now in this PR (commit 7514bed)

The first commit (05d0662) is unchanged — tests only, red. The second commit is the fix,
so the two can be reviewed, or reverted, independently.

What changed

Filter on the block height of the checkpoint instead of on zkapp_field.id, resolved
for the requested account before the main query runs. >= keeps the checkpoint's own block
in the answer, since o1js strips it in createActionsList by comparing actionStateOne
with the requested value.

checkActionState is replaced by resolveActionStateBoundary, scoped to the account and to
the chain. The old check accepted any value present anywhere in zkapp_field — action data,
app state, another account's action state — which is why the server answered with unrelated
data instead of raising an error.

The one design decision worth your attention

That boundary lookup is deliberately not limited to the queried block range. If it were,
an out-of-range checkpoint would resolve to NULL, and height >= NULL selects no rows — the
server would return an empty list, which a reducer reads as "no actions to fold". That
would be worse than the current behaviour, not better.

Being below the window does not by itself make the answer wrong:

• If the account emitted no action between the checkpoint and the start of the window, every
  action in the window is still the complete answer — and that is what clients get today.
  This is the common case for a quiet zkApp folding from its genesis action state, which is
  the default o1js path, because o1js sends no from/to.
• If it did emit actions in that gap, any list we return is missing them.

So the query is answered when it is provably complete, and rejected with a new
ACTION_STATE_OUT_OF_RANGE error naming the from to use when it is not. endActionState
gets no such exception: it bounds the request from above, so a checkpoint below the window
puts the whole requested span below it, and an empty answer would be silently wrong.

Happy to change this if you would rather always reject — it is a one-line change and the
tests for both behaviours are already written.

Validation on real data

A mesa-rc-1 archive dump (2026-08-11). For 20 accounts known to have an interning-order
inversion, every action state in the unfiltered list was used as a fromActionState and the
answer compared with the expected suffix:

	checkpoints checked	wrong suffix returned
before (e353bbc)	61	48
after (this branch)	61	0

For the account behind the original incident:

before   fromActionState = checkpoint -> 32 entries, 1 chain-link break, block 303203 missing
after    fromActionState = checkpoint -> 33 entries, 0 breaks, nothing missing

and querying the formerly-dropped state now correctly starts at block 303203 rather than
returning block 303199, which is before the checkpoint.

Tests

action-state-ordering: 36 pass, 0 fail, 0 todo (was 18 pass / 15 fail / 1 todo). The
todo is resolved and replaced by three explicit cases: rejected when actions were missed,
answered normally when nothing was missed, and endActionState below the range rejected with
no exception.

Existing suites unchanged: 83 unit, 25 integration. npm run lint clean.

Related

o1-labs/o1js#2905 — createActionsList cannot detect a dropped block, because it never
compares actionStateOne[i] with actionStateTwo[i+1]. That is why this fault was silent on
the client. Worth fixing independently of this PR: it would make any archive fault of this
class loud, on any endpoint, for every zkApp.

[dkijania]

Update: nightly invariant suite on real data (commit 5f686ae)

Third commit. Adds the real-dump counterpart to the fixture suite, plus a nightly workflow —
nothing runs test:devnet-dump automatically today.

It reuses the existing tests/devnet-dump download and load, so no new data
infrastructure. It is registered from devnet-dump.test.ts rather than living in its own
file, because node:test gives each file its own process and therefore its own multi-minute
dump load.

The two suites do different jobs

	Fixture suite (per PR)	Nightly suite
Data	generated, deterministic	a real dump, different every run
Question	does the known defect stay fixed?	do the rules hold on real data shapes?
Builds its own adversarial case	yes — inverts the interning order on purpose	no — takes data as it comes

Designed to need no upkeep

The dump rotates hourly, so anything pinned to its contents rots within a day. Four rules:

1. No expected values. Every assertion is an invariant the data implies about itself,
   with the unfiltered list as the reference: fromActionState returns exactly the suffix,
   endActionState exactly the prefix, consecutive entries link, another account's action
   state is rejected. Nothing to update when the data changes.
2. Subjects are discovered, never named, in a deterministic order, accounts with an
   interning-order inversion first. No ORDER BY random() — a nightly failure that cannot be
   reproduced on the same dump is one nobody can act on.
3. Finding nothing to test is a failure. A suite that silently tests nothing reports
   success, which is worse than no suite. The last test asserts the run covered something.
4. Failures are reproducible without the dump.

It has teeth — verified both ways

Run against a dump with known interning-order inversions with the old ActionsService
injected, it fails and points straight at the incident:

fromActionState returned the wrong set of blocks.
  account    : B62qmMvzQNSCnZ4qH1N9uJByov9EyGushzv1jV7Cqg8UwS6BrRgHGzk
  checkpoint : 27078409473756655551477733610137394255970846757871033008217988809531431111245 (block 303199)
  entries    : expected 133, got 132
  diverges at: index 1 (expected block 303203, got 303207)
  missing    : [303203]
  unexpected : none
  reproduce  : { actions(input: { address: "B62q…", fromActionState: "27078…" }) { … } }

With the fixed service: passes over 12 accounts and 40 checkpoints. A suite that cannot fail
proves nothing, so that check is documented in tests/devnet-dump/README.md as the way to
re-verify it later.

Real accounts have hundreds of action states, so the message reports the first divergence and
the missing blocks rather than printing both lists in full.

Note on the workflow

The postgres service is pinned to 17 to match the major version the production dumps are
taken from — an older server rejects statements the newer pg_dump emits. This is the first
time this suite will run in CI at all, so the first nightly may need a nudge (disk space for
the dump is the other likely snag; the job frees a few GB up front).

PR-gated suites are unaffected: action-state 36/36, integration 25/25, unit 83, lint clean.

[SanabriaRusso]

Verdict: MERGEABLE ✅

Second pass, focused on downstream compatibility. I verified the premise, both boundary semantics, ordering, and the new error path against the code at 5f686ae.

What I checked

• Neither browser/API consumer touches actions or events. mina-explorer sends exactly two archive root fields — blocks(...) (src/services/api/blocks.ts:111,183, src/services/api/analytics.ts:53,76,98) and networkState (src/services/api/blocks.ts:33,194). mina-explorer-api is the same: blocks(...) (app/upstream/archive.py:244, app/indexer/queries.py:117) and networkState (app/upstream/archive.py:172,254, readiness probe app/observability.py:306). Every action* identifier in either repo (actionState, editActionState, e.g. mina-explorer-api/app/upstream/daemon.py:127) is a daemon account(publicKey:) field, not this schema. So the blast radius of this PR is o1js/zkApp clients only — the explorer stack cannot regress from it.
• Error text (HARD CONSTRAINT Add Actions resolver support #1) is untouched. src/errors/error.ts at head only adds throwActionStateOutOfRangeError, still via new GraphQLError(...) with an extensions.code. No masking widened, Cannot query field / Unknown argument / inBestChain wording unaffected.
• The premise is real. main src/db/sql/events-actions/queries.ts:313 and :318 filter zkf0.id >= (SELECT id FROM zkapp_field WHERE field = ...). zkapp_field is interned with zkapp_field_field_key UNIQUE (field) (tests/integration/fixtures/archive_db.sql:5576) and id is assignment order, not chain order. Confirmed the two orders are independent — nothing in the schema ties zkapp_field.id to blocks.height.
• Both inclusivity semantics are preserved exactly, and I convinced myself the change is a strict bug fix in the well-ordered case. zkf0 is zkapp_action_states.element0 reached via zkapp_accounts ON id = emitted_actions.zkapp_id, i.e. the account's post-block action state — and accounts_accessed has one row per (block, account), so every row of a block already shared one zkf0. The old filter was therefore already block-granular, and height >= min(height where state = target) selects the identical set whenever interning order agrees with chain order. >= keeps the checkpoint's own block, matching what clients receive today; changing it to > would double-strip against o1js's createActionsList.
• Tests pin the boundary, and they really ran. tests/integration/action-state-ordering.test.ts:118 asserts heightsOf(actions) === actionBlockHeights.slice(index) plus actions[0].actionState.actionStateOne === state.value — an off-by-one to > returns slice(index+1) and fails. endActionState = S3 is pinned to slice(0,3), so the upper boundary is inclusive-pinned too. CI log for run 31537825690 shows Running ./build/tests/integration/action-state-ordering.test.js and e.g. ok 2 - inverted: fromActionState = S2 (block 28) returns blocks [28,29,30]. The suite is picked up by run-tests.sh's ./build/**/*test.js glob (it is not in the devnet-dump/live-network skip list), and action-state-setup.ts throws rather than skipping when Postgres is absent — no green-when-it-should-be-red.
• Ordering is unchanged and still total. Neither getActionsQuery on main nor at head has an ORDER BY; the total order is imposed in TS and this PR does not touch it — sortAndFilterBlocks (src/services/data-adapters/database-row-adapters.ts:242-251, height then timestamp, then filterBestTip which collapses same-height forks so at most one block per height survives) and sortActions (actions-service.ts, sequenceNumber then index into zkappAccountUpdateIds). Multiple actions in one transaction tie-break on account-update position, which is deterministic. Nothing here regressed.
• Performance is fine, and the index the new predicate needs already exists. resolveActionStateBoundary's hot predicate is aa.account_identifier_id = (...), covered by idx_accounts_accessed_block_account_identifier_id (tests/integration/fixtures/archive_db.sql:5727); zkapp_field.field is covered by the unique constraint at :5576. pending_chain is RECURSIVE and account_action_states is referenced three times, so Postgres materialises both once. The call count is unchanged from main (up to two boundary lookups, previously two checkActionState calls), just heavier per call — an index scan of one account's accounts_accessed history with PK nested loops. Nowhere near the 30s statement_timeout P0: Configure Postgres pool limits + statement_timeout (#165) #182 proposes; the 2000-block analytics blocks query remains the worst case.
• Size is honest. Production code is +233/-38 across three files (queries.ts +129/-13, actions-service.ts +87/-24, error.ts +17/-1). The other ~1741 lines are fixtures, tests, READMEs and the nightly workflow, which is schedule + workflow_dispatch only and does not gate PRs.
• CI/merge state: all 7 checks SUCCESS; mergeable: MERGEABLE, mergeStateStatus: BLOCKED — that is the required-approval gate, not a conflict.

The one behaviour change callers will notice

A previously-answered query can now 400 with ACTION_STATE_OUT_OF_RANGE: fromActionState resolving below window_start and the account emitted actions in the gap, or any endActionState below window_start. I traced what those callers got before — the old filter kept every in-window row (all in-window zkf0.ids exceed the checkpoint's in the well-ordered case), so they received a list silently truncated at window_start, which folds to a state that can never match the chain. Turning that into a loud, self-describing error is the right trade, and state_entering_window correctly rescues the case that matters most: the quiet zkApp folding from an old genesis action state with no from/to, which is the default o1js path. That path returns [] when there is nothing after the checkpoint, which folds to the checkpoint itself — correct. Not a blocker, but worth a line in the release notes, because for a busy long-lived zkApp this flips a silent wrong answer into a hard failure and someone will file it as a regression.

Non-blocking nits

1. from: 0 bypasses the new out-of-range guard. The two window computations disagree on falsy zero. fullChainCTE (queries.ts:8) does if (fromAsNum) — 0 is falsy, so from: 0 falls through to the default height >= MAX(height) - BLOCK_RANGE_SIZE window. resolveActionStateBoundary uses fromAsNum !== null, and since from?.toString() yields the truthy string "0", it computes window_start = 0. Result: with from: 0, fromActionState: <old state>, the guard never fires but the real window is the last 10 000 blocks — the exact silent truncation this PR removes, reintroduced on one input. It is a pre-existing fullChainCTE quirk rather than something this PR causes, hence not blocking. One-line fix that mirrors fullChainCTE's truthiness exactly:

// src/db/sql/events-actions/queries.ts, in resolveActionStateBoundary
-  const fromAsNum = from ? Number(from) : null;
-  const toAsNum = to ? Number(to) : null;
+  // `|| null` so "0" collapses to null, matching fullChainCTE's `if (fromAsNum)`
+  // truthiness check. Number(undefined) is NaN, which is also falsy.
+  const fromAsNum = Number(from) || null;
+  const toAsNum = Number(to) || null;

and the test that pins it, alongside the existing out-of-range cases:

test('from: 0 does not bypass the out-of-range guard', async () => {
  const account = fixture.accounts.control;
  const actions = await actionsService.getActions(
    { address: account.address, fromActionState: account.states[0].value, from: 0 },
    nullOptions
  );
  // window_start must be derived the same way as fullChainCTE, so `from: 0`
  // behaves exactly like sending no `from` at all.
  const noFrom = await actionsService.getActions(
    { address: account.address, fromActionState: account.states[0].value },
    nullOptions
  );
  assert.deepStrictEqual(heightsOf(actions), heightsOf(noFrom));
});

Cleanest long-term shape is to export one resolveWindow(from, to) helper and have fullChainCTE and resolveActionStateBoundary both call it, so they cannot drift again.

2. status is not passed to the boundary lookup. account_action_states accepts chain_status = 'canonical' OR id IN pending_chain regardless of the requested status, while the main query filters blocks_accessed by it (blocksAccessedCTE). For { status: PENDING, fromActionState: <canonical state> } the boundary resolves on the canonical chain and can now trip ACTION_STATE_OUT_OF_RANGE where the old code returned pending rows. o1js does not send status, so this is theoretical — a sentence in the resolveActionStateBoundary docblock saying the boundary is deliberately resolved over the full chain would close it.

3. min(height) and re-seeded action states. boundary_height is min(height) over the account's whole history. The hash chain makes value recurrence impossible in normal operation, with one exception: the empty/initial action state, which reappears for an account if a hard-fork migration re-seeds it inside the same database. min() then resolves to the pre-fork creation height, state_entering_window compares equal, and the answer silently omits the pre-fork actions. Given that hard-fork migration is one of the scenarios in the PR description, a follow-up either scoping the lookup at or above the fork height, or a nightly invariant asserting boundary_height is unique per (account, state), would be worth having. assertActionChainIsLinked would catch the symptom if a dump ever exhibits it.

Automated second-pass review — focus: downstream compatibility with mina-explorer / mina-explorer-api.

[SanabriaRusso]

Approving on the basis of the second-pass review comment above: no mid-to-high severity security, compatibility, or degradation issue found, and the downstream contract with mina-explorer / mina-explorer-api holds — GraphQL validation error text reaches errors[].message verbatim, the browser SPA's cross-origin access is preserved, and the real consumer query shapes (including the 2000-block analytics query and the 500-row page crawl) still pass.

Two things this approval does not mean:

• It does not close the non-blocking items in the review comment. Several are worth fixing before or shortly after merge; they are written up there with patches.
• It does not by itself mean the branch is ready to merge. main requires branches to be up to date, so this needs an update-branch (or a rebase, if the branch is conflicting) first, and a few PRs in this series have cross-PR ordering constraints called out in their review comments.

Automated second-pass review — focus: downstream compatibility with mina-explorer / mina-explorer-api.