Severity: medium · Component: fluree-db-server transact route, fluree-db-consensus CachingCommitter / raft queued transactor
Summary
The write path has a concurrency cap but no time bound. CachingCommitter already load-sheds via a two-tier admission semaphore — global DEFAULT_PENDING_LIMIT=1024 (fluree-db-consensus/src/caching.rs:59), per-ledger 128 (caching.rs:68) — acquired before staging and refusing overflow with Overloaded→503 (fluree-db-consensus/src/caching.rs:719). But transact_via_consensus awaits committer.transact() inline with no timeout of any kind (fluree-db-server/src/routes/transact.rs:304), and the admission caps are hardcoded (never with_pending_limit-wired at fluree-db-server/src/state.rs:217). A write that stalls on slow storage, lock contention, or a stuck peer holds its admission permit and the per-ledger write lock indefinitely, which wedges the ledger (other writers block on the lock, then 503 as admission fills) and pins memory.
Two distinct timeouts are needed (they are not the same knob)
1. Ingress deadline (request-level). Measured from when the request is received: "if this can't be completed within N seconds, don't even start it." This bounds total client-perceived latency and prevents doing work whose result will be discarded. It must include time spent waiting — for the per-ledger write lock, and (in raft) sitting in the proposal queue. The check is cheap and singular: at the moment we are about to begin processing (acquire the write lock / drain from the raft queue), compare now against the deadline and drop early if already past. No loop involvement.
Raft implication: in raft mode the transaction is enqueued, drained by the leader, and applied later (possibly across a leader change). For the ingress deadline to be honored at drain/apply time, the ingress timestamp (or absolute deadline) must travel with the transaction — TransactionRequest currently carries no such field, and neither does the raft queue envelope, so this needs to be added. Design call to make: record an absolute ingress time vs. a relative budget, and decide which node's clock owns the deadline evaluation (cross-node skew matters). Optionally also persist it in durable txn metadata for audit.
2. Processing timeout (execution budget). Measured from when processing actually starts (after dequeue / lock acquisition), bounding the staging+commit execution itself — the "stuck on slow storage / pathological WHERE" guard. Implemented as tokio::time::timeout around the execution portion only, not the queue/lock wait. On elapse, drop the future → the held LedgerWriteGuard and admission permit RAII-release, so the timeout unwedges rather than deepens the problem. Return a distinct status (504) so it is separable from admission's 503 in metrics.
Admission caps: config-wire and right-size
clone_state() is cheap (every heavy LedgerState field is Arc, fluree-db-ledger/src/lib.rs:89-102), so the memory driver under load is the concurrent staging working set — the global cap is the right lever. Wire with_pending_limit / with_per_ledger_pending_limit (caching.rs:324/:337) from config at state.rs:217 so small nodes can dial them down; defaults stay 1024/128 (backward-compatible).
Where to hook the cancellation check (without slowing txns)
The coarse tokio::time::timeout already gives drop-based cancellation at await points with zero per-row cost — that is the primary mechanism. If finer cancellation is wanted (so a long sync staging segment can bail mid-flight), the staging WHERE runs the same query engine as reads (execute_where_streaming), which already has cooperative checkpoints at I/O boundaries (fast_path_common::bail_if_cancelled — leaflet refill, partition start). Reuse those by threading a cancellation handle through staging; do not add a check to the fused per-row/per-group merge loops — that path measured +5-15% end-to-end from codegen perturbation even when strided (see the hot-loop-purity note in CLAUDE.md). Cancellation belongs on I/O boundaries only.
Safety
The commit point is the nameservice CAS publish. Dropping before it leaves only a harmless orphan blob; dropping after it but before finalize_commit leaves the cache one step behind the durable head, which self-heals on the next write via CommitConflict refresh (fluree-db-transact/src/commit.rs:1050). No torn state, and permit/lock release on drop.
Tests
- Overflow → 503 + bounded peak in-flight.
- Stuck/slow commit → 504 within the processing budget and a subsequent write to the same ledger succeeds (proves no wedge).
- Ingress deadline → a request that waits past its deadline (lock held by another writer, or raft queue backlog) is dropped before processing starts, not after.
- Per-ledger fairness — a burst on ledger A doesn't 503 writes to ledger B.
Config knobs
Mirror query_min_t_timeout_ms (present in both config.rs and config_file.rs): write_ingress_deadline_ms, write_processing_timeout_ms (0 = off), max_pending_writes, max_pending_writes_per_ledger.
Optional follow-ups for consideration
- Persist the ingress deadline / timing in durable txn metadata (audit + post-hoc analysis of dropped txns).
- Explicit
TxnCancellation token threaded through TxnOpts (deterministic disconnect signal + sync-segment checkpoints), if drop-based proves insufficient under profiling.
Retry-After header on the 503/504 responses to guide client backoff.- Confirm bulk
/import does not route through transact_via_consensus before enabling a default processing timeout; very large single ingests should use the import path.
cc @zonotope — the ingress-deadline propagation through the raft queue envelope overlaps with the raft work.
Severity: medium · Component:
fluree-db-servertransact route,fluree-db-consensusCachingCommitter/ raft queued transactorSummary
The write path has a concurrency cap but no time bound.
CachingCommitteralready load-sheds via a two-tier admission semaphore — globalDEFAULT_PENDING_LIMIT=1024(fluree-db-consensus/src/caching.rs:59), per-ledger 128 (caching.rs:68) — acquired before staging and refusing overflow withOverloaded→503 (fluree-db-consensus/src/caching.rs:719). Buttransact_via_consensusawaitscommitter.transact()inline with no timeout of any kind (fluree-db-server/src/routes/transact.rs:304), and the admission caps are hardcoded (neverwith_pending_limit-wired atfluree-db-server/src/state.rs:217). A write that stalls on slow storage, lock contention, or a stuck peer holds its admission permit and the per-ledger write lock indefinitely, which wedges the ledger (other writers block on the lock, then 503 as admission fills) and pins memory.Two distinct timeouts are needed (they are not the same knob)
1. Ingress deadline (request-level). Measured from when the request is received: "if this can't be completed within N seconds, don't even start it." This bounds total client-perceived latency and prevents doing work whose result will be discarded. It must include time spent waiting — for the per-ledger write lock, and (in raft) sitting in the proposal queue. The check is cheap and singular: at the moment we are about to begin processing (acquire the write lock / drain from the raft queue), compare
nowagainst the deadline and drop early if already past. No loop involvement.Raft implication: in raft mode the transaction is enqueued, drained by the leader, and applied later (possibly across a leader change). For the ingress deadline to be honored at drain/apply time, the ingress timestamp (or absolute deadline) must travel with the transaction —
TransactionRequestcurrently carries no such field, and neither does the raft queue envelope, so this needs to be added. Design call to make: record an absolute ingress time vs. a relative budget, and decide which node's clock owns the deadline evaluation (cross-node skew matters). Optionally also persist it in durable txn metadata for audit.2. Processing timeout (execution budget). Measured from when processing actually starts (after dequeue / lock acquisition), bounding the staging+commit execution itself — the "stuck on slow storage / pathological WHERE" guard. Implemented as
tokio::time::timeoutaround the execution portion only, not the queue/lock wait. On elapse, drop the future → the heldLedgerWriteGuardand admission permit RAII-release, so the timeout unwedges rather than deepens the problem. Return a distinct status (504) so it is separable from admission's 503 in metrics.Admission caps: config-wire and right-size
clone_state()is cheap (every heavyLedgerStatefield isArc,fluree-db-ledger/src/lib.rs:89-102), so the memory driver under load is the concurrent staging working set — the global cap is the right lever. Wirewith_pending_limit/with_per_ledger_pending_limit(caching.rs:324/:337) from config atstate.rs:217so small nodes can dial them down; defaults stay 1024/128 (backward-compatible).Where to hook the cancellation check (without slowing txns)
The coarse
tokio::time::timeoutalready gives drop-based cancellation at await points with zero per-row cost — that is the primary mechanism. If finer cancellation is wanted (so a long sync staging segment can bail mid-flight), the staging WHERE runs the same query engine as reads (execute_where_streaming), which already has cooperative checkpoints at I/O boundaries (fast_path_common::bail_if_cancelled— leaflet refill, partition start). Reuse those by threading a cancellation handle through staging; do not add a check to the fused per-row/per-group merge loops — that path measured +5-15% end-to-end from codegen perturbation even when strided (see the hot-loop-purity note inCLAUDE.md). Cancellation belongs on I/O boundaries only.Safety
The commit point is the nameservice CAS publish. Dropping before it leaves only a harmless orphan blob; dropping after it but before
finalize_commitleaves the cache one step behind the durable head, which self-heals on the next write viaCommitConflictrefresh (fluree-db-transact/src/commit.rs:1050). No torn state, and permit/lock release on drop.Tests
Config knobs
Mirror
query_min_t_timeout_ms(present in bothconfig.rsandconfig_file.rs):write_ingress_deadline_ms,write_processing_timeout_ms(0 = off),max_pending_writes,max_pending_writes_per_ledger.Optional follow-ups for consideration
TxnCancellationtoken threaded throughTxnOpts(deterministic disconnect signal + sync-segment checkpoints), if drop-based proves insufficient under profiling.Retry-Afterheader on the 503/504 responses to guide client backoff./importdoes not route throughtransact_via_consensusbefore enabling a default processing timeout; very large single ingests should use the import path.cc @zonotope — the ingress-deadline propagation through the raft queue envelope overlaps with the raft work.