feat(storage): service-wide, stackable guard layer for API limits (#1630)

[AkramBitar]

Add size limits to the storage service so one huge write or one unlimited query can't overload the database. Fixes the [MED] issue #1630 (denial of service from unlimited resource use, CWE-400 / CWE-770).

What changed since the first revision

The first revision implemented the limits only for the transaction store, via per-constructor WithMax* options + inline checks. Per review feedback (that approach doesn't scale beyond one store), this has been reworked into a single, stackable guard decorator layer applied across the whole storage service. The transaction store was migrated onto it, so there is now one mechanism.

How it works

• New package token/services/storage/db/guard/: a Policy (loaded once from config), payload-size checks, a row-capping LimitIterator, and interface-embedding decorators for the transaction, token, endorser, identity and wallet stores. Each decorator embeds the store interface (delegating everything) and overrides only the methods that need a check, plus the nested write-transactions and returned iterators.
• Applied once at the multiplexed.Driver seam: every NewXxx result is wrapped, so all backing drivers (SQLite, PostgreSQL) and all stores are covered uniformly. Additional layers (metrics/tracing) can stack the same way, with the concrete SQL store at the bottom.

What is guarded

Write payload size — maxPayloadSize (default 4 MiB; 0 disables), rejected before reaching the DB:

• transaction: AddTokenRequest, AddTransaction, AddMovement, AddTransactionEndorsementAck
• token: StoreToken, StorePublicParams, StoreCertifications
• endorser: AddValidationRecord
• identity: StoreIdentityData, StoreSignerInfo, RegisterIdentityDescriptor, AddConfiguration
• wallet: StoreIdentity

Read caps — maxPageSize (default 1000):

• paginated QueryTransactions — nil / pagination.None() / over-max pages are rejected
• streaming iterators — token unspent/spendable/unsupported iterators, QueryValidations, QueryTokenRequests, IteratorConfigurations — capped by a LimitIterator that errors (rather than silently truncating) once the cap is exceeded

Intentionally uncapped: QueryMovements (feeds balance totals; dropping rows would silently return wrong balances).

Also closes two gaps in pagination.ValidateLimited: keyset page size is now capped against maxPageSize, and a typed-nil offset is guarded.

Configuration / upgrade note

Limits are on by default (4 MiB / 1000), so existing deployments get them automatically. Override via token.storage.maxPayloadSize / token.storage.maxPageSize; an explicit 0 disables the respective check. Paging on QueryTransactions cannot be turned off — nil/None are rejected — so any caller that read everything at once now pages (see collectAllTransactions / checks.go).

Not yet addressed (tracked follow-up)

These vectors can't be closed by a wrapper alone, because the SQL has already materialised the full result before the decorator sees it — they need query-level changes and will be handled in a follow-up:

• Materialized slice/map reads: ListUnspentTokens, ListUnspentTokensByWallets, ListHistoryIssuedTokens, ListAuditTokens, QueryTokenDetails, ConfigurationsByID, GetWalletIDs, GetExistingSignerInfo — need a SQL-level LIMIT (with a defined order + documented truncation) or conversion to iterators.
• Variadic input-size caps: DeleteTokens, GetTokens, GetTokenRequests — a huge id list expands into a huge IN (...) clause.
• Keystore.Put: opaque value serialised inside the store, so its size is only known at marshal time.

Closes #1630.

[adecaro]

Hi @AkramBitar , thanks for submitting this.

My concern is the following: The storage service has many queries that would require similar constraints. Right now, this opts-based approach works only for the Transactions db.

I would suggest to perform an analysis of the entire storage service to understand which limits need to be put in place and then design a flexible solution to address the issues. A wrapper-based solution could also be interested so that we can stack different layers on top of each other each perform a different function and the bottom one running the actual SQL query.

What do you think?

Thanks 🙏

[AkramBitar]

Hi @AkramBitar , thanks for submitting this.

My concern is the following: The storage service has many queries that would require similar constraints. Right now, this opts-based approach works only for the Transactions db.

I would suggest to perform an analysis of the entire storage service to understand which limits need to be put in place and then design a flexible solution to address the issues. A wrapper-based solution could also be interested so that we can stack different layers on top of each other each perform a different function and the bottom one running the actual SQL query.

What do you think?

Thanks 🙏

Hi @adecaro,

Thanks a lot for the review. Good idea. I implemented it.

Thanks,
Akram

[adecaro]

shouldn't this be configurable as well?

[AkramBitar]

Good point. Rather than adding a separate config key, I've made the checker page at the storage layer's existing token.storage.maxPageSize ca. So it reads within the same bound the guard layer already enforces, with no new knob to keep in sync. The 100 constant is now just the fallback for when the cap is disabled (maxPageSize = 0), since an unbounded read gets rejected and we still need some batch size to page by.

[adecaro]

I think this function is used by the token selectors and I think these token selectors work on the assumption that they can load all the tokens that belong to a wallet. If this is not the case anymore, we selector might report unexpected errors. No?

[AkramBitar]

Good catch, you're right. LimitIterator errors instead of truncating, so a wallet with more than MaxPageSize tokens would fail selection even though the funds exist. UnspentTokensIteratorBy / SpendableTokensIteratorBy are a legitimate "load all tokens of this wallet" operation the selectors rely on, so I'll drop the cap on them and keep it only on the unbounded, non-wallet-scoped reads. Write payload limits are unaffected.

[adecaro]

if maxPageSize > 0, empty should not be allowed, no?

[AkramBitar]

Empty compiles to LIMIT 0, it returns zero rows, so it can never exceed maxPageSize. The unbounded case is None (no LIMIT), which we already reject. Allowing empty is safe; rejecting it would only break harmless zero-row queries.

[adecaro]

so, in QueryTransactions the caller can extract all the transactions even though in a paginated format. Here instead can only extract up to a certain total number of token requests. I think we need to introduce a standard here. We can say that all queries need to be paginated and we can even say that despite the pagination only a maximum number of rows can be extracted. What do you think?