Skip to content

feat(rate-limit): make tier quotas configurable and return standard rate-limit headers #79

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
produckq-boop wants to merge 1 commit into
base: main
Choose a base branch
from
Open

feat(rate-limit): make tier quotas configurable and return standard rate-limit headers #79

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions .env.example
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,20 @@ INDEXER_BATCH_SIZE=100
INDEXER_CATCHUP_WORKERS=4

# ─── Rate limiting ────────────────────────────────────────────────────────────
# Token-bucket rate limit tiers (per-tier request quota and window in milliseconds).
# Optional RATE_LIMIT_TIER_<TIER>_BURST overrides burst allowance per tier.
RATE_LIMIT_TIER_UNAUTHENTICATED_LIMIT=60
RATE_LIMIT_TIER_UNAUTHENTICATED_WINDOW_MS=60000
RATE_LIMIT_TIER_FREE_LIMIT=1000
RATE_LIMIT_TIER_FREE_WINDOW_MS=60000
RATE_LIMIT_TIER_DEVELOPER_LIMIT=1000
RATE_LIMIT_TIER_DEVELOPER_WINDOW_MS=60000
RATE_LIMIT_TIER_PRO_LIMIT=10000
RATE_LIMIT_TIER_PRO_WINDOW_MS=60000
RATE_LIMIT_TIER_ENTERPRISE_LIMIT=60000
RATE_LIMIT_TIER_ENTERPRISE_WINDOW_MS=60000

# Fallback / adaptive rate limiting
RATE_LIMIT_WINDOW_MS=60000
RATE_LIMIT_MAX=100
RATE_LIMIT_PUBLIC_MAX=100
Expand Down
52 changes: 52 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -171,6 +171,55 @@ Because the server caches keys, a typical rotation involves:

*Note: Never log or commit the `JWT_PRIVATE_KEY` to version control. The server performs a startup validation check and will crash if the keys are missing or malformed when `NODE_ENV=production`.*

## Rate Limiting

Every API request passes through `apiKeyAuth` followed by `tieredRateLimit` ([`src/middleware/tokenBucket.ts`](./src/middleware/tokenBucket.ts), [`src/middleware/rateLimit.ts`](./src/middleware/rateLimit.ts), [`src/middleware/apiKeyAuth.ts`](./src/middleware/apiKeyAuth.ts)). Quotas are enforced with an atomic Redis Lua token bucket when `REDIS_URL` is configured, and automatically fall back to an in-process token bucket when Redis is unavailable.

### Tiers & Configuration

Requests without an `X-Api-Key` header use the `unauthenticated` tier (keyed by client IP). Validated API keys resolve their tier from `DevApiKey.tier` (defaulting to `free`). Each tier's request limit and window duration are configurable via environment variables, falling back to the defaults below when unset or invalid:

| Tier | Default Limit | Default Window | Default Burst | Limit Variable | Window Variable |
| ----------------- | ------------- | -------------- | ------------- | --------------------------------------- | ------------------------------------------- |
| `unauthenticated` | `60` | `60000` ms | `10` | `RATE_LIMIT_TIER_UNAUTHENTICATED_LIMIT` | `RATE_LIMIT_TIER_UNAUTHENTICATED_WINDOW_MS` |
| `free` | `1000` | `60000` ms | `50` | `RATE_LIMIT_TIER_FREE_LIMIT` | `RATE_LIMIT_TIER_FREE_WINDOW_MS` |
| `developer` | `1000` | `60000` ms | `50` | `RATE_LIMIT_TIER_DEVELOPER_LIMIT` | `RATE_LIMIT_TIER_DEVELOPER_WINDOW_MS` |
| `pro` | `10000` | `60000` ms | `200` | `RATE_LIMIT_TIER_PRO_LIMIT` | `RATE_LIMIT_TIER_PRO_WINDOW_MS` |
| `enterprise` | `60000` | `60000` ms | `500` | `RATE_LIMIT_TIER_ENTERPRISE_LIMIT` | `RATE_LIMIT_TIER_ENTERPRISE_WINDOW_MS` |

Optional `RATE_LIMIT_TIER_<TIER>_BURST` variables can also be set to adjust per-tier burst allowances.

### Endpoint Group Multipliers

Resource-intensive route groups scale the effective quota for a request by a multiplier (`Math.ceil(limit * multiplier)`):

| Route Pattern | Group | Multiplier |
| ------------------------------------- | ----------- | ---------- |
| `/api/v1/events*` | `events` | `1.0x` |
| `/api/v1/search*`, `/api/v1/nft/search*` | `search` | `0.5x` |
| `POST /api/v1/contracts*` | `contracts` | `0.17x` |
| `/api/v1/simulate*` | `simulate` | `0.08x` |
| `/ws*` | `websocket` | `0.05x` |
| All other endpoints | default | `1.0x` |

### Per-Key & Admin Overrides

- **API key override (`DevApiKey.rateLimitOverride`)** — Setting a positive integer on `DevApiKey.rateLimitOverride` overrides the key's base tier limit for all requests authenticated with that `X-Api-Key` (endpoint multipliers still apply on top of the custom limit). After updating a key record in PostgreSQL, call `invalidateKeyCache(keyHash)` so the updated limit takes effect immediately rather than after `KEY_CACHE_TTL_MS` (default `10000` ms).
- **Admin endpoint override** — `POST /api/v1/admin/rate-limits` accepts `{ identifier, endpoint, max, windowMs }` to persist targeted overrides in `RateLimitOverride` for the fallback limiter (`X-RateLimit-Policy: user-override`).

### Response Headers

Standard rate-limit headers are returned on all rate-limited routes so clients can track remaining quota and back off cleanly:

| Header | Returned On | Description |
| --------------------- | --------------- | --------------------------------------------------------------------------- |
| `RateLimit-Limit` | Every response | Maximum requests permitted in the window for the active tier/override |
| `RateLimit-Remaining` | Every response | Remaining requests available in the current window (`0` when throttled) |
| `RateLimit-Reset` | Every response | Seconds until the rate-limit bucket replenishes to full capacity |
| `Retry-After` | `429` responses | Seconds (`>= 1`) the client should wait before retrying a throttled request |

Legacy `X-RateLimit-Limit`, `X-RateLimit-Remaining`, `X-RateLimit-Reset` (Unix epoch seconds), and `X-RateLimit-Tier` headers are also emitted for backward compatibility.

## Environment Variables

| Variable | Default | Description |
Expand All @@ -185,6 +234,9 @@ Because the server caches keys, a typical rotation involves:
| `INDEXER_BATCH_SIZE` | `100` | Ledgers per batch |
| `ADMIN_SECRET` | — | Bearer token required by every `/api/admin/*` route (indexer). See [`docs/ADMIN_AUTH.md`](./docs/ADMIN_AUTH.md). |
| `METRICS_TOKEN` | — | Bearer/`X-Metrics-Token` required to scrape the API service's `GET /metrics` from outside the host. Unset restricts it to loopback callers. See [Metrics](#metrics). |
| `REDIS_URL` | _(unset)_ | Shared Redis URL for distributed token-bucket rate limiting and caching (falls back to in-memory when unset). See [Rate Limiting](#rate-limiting). |
| `RATE_LIMIT_TIER_<TIER>_LIMIT` | Tier default | Per-window request quota for `UNAUTHENTICATED`, `FREE`, `DEVELOPER`, `PRO`, or `ENTERPRISE`. See [Rate Limiting](#rate-limiting). |
| `RATE_LIMIT_TIER_<TIER>_WINDOW_MS` | `60000` | Window duration in milliseconds for `UNAUTHENTICATED`, `FREE`, `DEVELOPER`, `PRO`, or `ENTERPRISE`. See [Rate Limiting](#rate-limiting). |
| `MOCK_DATA` | `false` | Gates experimental endpoints that fabricate demo data instead of a real integration. See [Experimental & Mock Data](#experimental--mock-data). `ENABLE_EXPERIMENTAL` is accepted as an alias. |

## Experimental & Mock Data
Expand Down
3 changes: 1 addition & 2 deletions package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,8 +23,7 @@
"repair": "ts-node src/indexer/repair-run.ts",
"archive": "ts-node src/archival/run.ts",
"seed": "ts-node prisma/seed.ts",
"test": "DATABASE_URL=postgresql://test:test@localhost:5432/test TESTNET_DATABASE_URL=postgresql://test:test@localhost:5432/test STELLAR_NETWORK=testnet vitest run tests/reentrancy-fortress.test.ts tests/api/nlq.test.ts tests/arbitrage-engine.test.ts tests/indexer/token-metadata.test.ts tests/error-handling-integration.test.ts tests/build-queue.test.ts tests/indexer/decoder-parity.test.ts tests/tracer.test.ts tests/api/mock-data-gating.test.ts tests/api/reputation-leaderboard.test.ts",
"test": "DATABASE_URL=postgresql://test:test@localhost:5432/test TESTNET_DATABASE_URL=postgresql://test:test@localhost:5432/test STELLAR_NETWORK=testnet vitest run tests/reentrancy-fortress.test.ts tests/api/nlq.test.ts tests/arbitrage-engine.test.ts tests/indexer/token-metadata.test.ts tests/error-handling-integration.test.ts tests/build-queue.test.ts tests/indexer/decoder-parity.test.ts tests/tracer.test.ts tests/webhooks/dispatcher.test.ts",
"test": "DATABASE_URL=postgresql://test:test@localhost:5432/test TESTNET_DATABASE_URL=postgresql://test:test@localhost:5432/test STELLAR_NETWORK=testnet vitest run tests/reentrancy-fortress.test.ts tests/api/nlq.test.ts tests/arbitrage-engine.test.ts tests/indexer/token-metadata.test.ts tests/error-handling-integration.test.ts tests/build-queue.test.ts tests/indexer/decoder-parity.test.ts tests/tracer.test.ts tests/api/mock-data-gating.test.ts tests/api/reputation-leaderboard.test.ts tests/webhooks/dispatcher.test.ts tests/api-key-auth.test.ts tests/rate-limit.test.ts tests/token-bucket.test.ts",
"test:watch": "vitest",
"test:coverage": "vitest run --coverage",
"test:ui": "vitest --ui",
Expand Down
10 changes: 10 additions & 0 deletions src/index.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -130,6 +130,16 @@ app.use(
origin: corsOrigin,
methods: ['GET', 'POST', 'PUT', 'PATCH', 'DELETE', 'OPTIONS'],
allowedHeaders: ['Content-Type', 'Authorization', 'X-Api-Key', 'X-Request-Id'],
exposedHeaders: [
'RateLimit-Limit',
'RateLimit-Remaining',
'RateLimit-Reset',
'Retry-After',
'X-RateLimit-Limit',
'X-RateLimit-Remaining',
'X-RateLimit-Reset',
'X-RateLimit-Tier',
],
credentials: true,
}),
);
Expand Down
28 changes: 26 additions & 2 deletions src/middleware/apiKeyAuth.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,30 @@ declare global {
}
}

const VALID_RATE_LIMIT_TIERS = new Set<RateLimitTier>([
'unauthenticated',
'free',
'developer',
'pro',
'enterprise',
]);

export function normalizeRateLimitTier(
rawTier: string | null | undefined,
fallback: RateLimitTier = 'free',
): RateLimitTier {
if (!rawTier) return fallback;
const candidate = rawTier.trim().toLowerCase() as RateLimitTier;
return VALID_RATE_LIMIT_TIERS.has(candidate) ? candidate : fallback;
}

function normalizeRateLimitOverride(rawOverride: number | null | undefined): number | undefined {
if (typeof rawOverride !== 'number' || !Number.isFinite(rawOverride) || rawOverride <= 0) {
return undefined;
}
return Math.floor(rawOverride);
}

function hashKey(raw: string): string {
return crypto.createHash('sha256').update(raw).digest('hex');
}
Expand Down Expand Up @@ -247,8 +271,8 @@ async function resolveApiKey(raw: string): Promise<ApiKeyContext | null> {
id: record.id,
keyName: record.name,
developerId: record.developerId,
tier: (record.tier as RateLimitTier) ?? 'free',
rateLimitOverride: record.rateLimitOverride ?? undefined,
tier: normalizeRateLimitTier(record.tier, 'free'),
rateLimitOverride: normalizeRateLimitOverride(record.rateLimitOverride),
allowedIps: Array.isArray(record.allowedIps) ? (record.allowedIps as string[]) : undefined,
allowedEndpoints: Array.isArray(record.allowedEndpoints)
? (record.allowedEndpoints as string[])
Expand Down
109 changes: 67 additions & 42 deletions src/middleware/rateLimit.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,9 @@ import { prismaRead } from '../db';
import { logger } from '../logger';
import {
checkTokenBucket,
RateLimitTier,
type RateLimitTier,
setRateLimitRedisClient,
TokenBucketResult,
type TokenBucketResult,
} from './tokenBucket';

const developerKeys = new Set((process.env.API_KEYS_DEVELOPER ?? '').split(',').filter(Boolean));
Expand Down Expand Up @@ -102,13 +102,15 @@ function applyAdaptiveThrottle(
const currentMax = Math.min(throttledMax, tierConfigValue.max);
res.setHeader('X-RateLimit-Warn', 'true');
res.setHeader('X-RateLimit-Predicted', `${currentMax}`);
req.app.locals.rateLimitPredictedMax = currentMax;
if (req.app?.locals) {
req.app.locals.rateLimitPredictedMax = currentMax;
}
return { ...tierConfigValue, max: currentMax };
}

function getRequestBucketKey(req: Request, tier: TierName, userIdentifier?: string): string {
const endpoint = req.path || req.originalUrl || '/';
const keySource = userIdentifier ?? req.ip ?? 'unknown';
const keySource = userIdentifier ?? req.apiKey?.id ?? req.ip ?? 'unknown';
return `${tier}:${keySource}:${endpoint}`;
}

Expand Down Expand Up @@ -159,7 +161,6 @@ function buildLimiters(store?: Store): Limiters {
}

let legacyLimiters: Limiters = buildLimiters();
let useTokenBucket = false;

export async function initRateLimitStore(): Promise<void> {
const redisUrl = process.env.REDIS_URL;
Expand All @@ -174,7 +175,6 @@ export async function initRateLimitStore(): Promise<void> {
await client.connect();

setRateLimitRedisClient(client as any);
useTokenBucket = true;

const store = new RedisStore({
sendCommand: (...args: string[]) => (client as any).sendCommand(args),
Expand All @@ -188,14 +188,35 @@ export async function initRateLimitStore(): Promise<void> {
}
}

function writeRateLimitHeaders(res: Response, result: TokenBucketResult): number {
const nowSec = Math.floor(Date.now() / 1000);
const remaining = Math.max(0, result.remaining);
const resetSeconds = Math.max(0, result.resetAt - nowSec);

res.setHeader('RateLimit-Limit', String(result.limit));
res.setHeader('RateLimit-Remaining', String(remaining));
res.setHeader('RateLimit-Reset', String(resetSeconds));

res.setHeader('X-RateLimit-Limit', String(result.limit));
res.setHeader('X-RateLimit-Remaining', String(remaining));
res.setHeader('X-RateLimit-Reset', String(result.resetAt));
res.setHeader('X-RateLimit-Tier', result.tier);

return resetSeconds;
}

function runLocalRateLimit(req: Request, res: Response, next: NextFunction): void {
void (async () => {
const apiKey = req.headers['x-api-key'] as string | undefined;
const tier = getRateLimitTier(apiKey);
const userIdentifier = req.headers['x-user-id'] as string | undefined;
const endpoint = req.path || req.originalUrl || '/';
const configOverride = userIdentifier ? await getUserOverride(userIdentifier, endpoint) : null;
const baseConfig = configOverride ?? normalizeTierConfig()[tier];
const apiKeyOverride =
req.apiKey?.rateLimitOverride && req.apiKey.rateLimitOverride > 0
? { windowMs: DEFAULT_TIERS.public.windowMs, max: req.apiKey.rateLimitOverride }
: null;
const baseConfig = apiKeyOverride ?? configOverride ?? normalizeTierConfig()[tier];
const effectiveConfig = applyAdaptiveThrottle(req, res, baseConfig);
const now = Date.now();
const bucketKey = getRequestBucketKey(req, tier, userIdentifier);
Expand All @@ -212,15 +233,24 @@ function runLocalRateLimit(req: Request, res: Response, next: NextFunction): voi
resetAt: now + effectiveConfig.windowMs,
};
const remaining = Math.max(0, effectiveConfig.max - bucket.count);
const resetEpochSeconds = Math.ceil(bucket.resetAt / 1000);
const resetSeconds = Math.max(0, resetEpochSeconds - Math.floor(now / 1000));

res.setHeader('RateLimit-Limit', `${effectiveConfig.max}`);
res.setHeader('RateLimit-Remaining', `${remaining}`);
res.setHeader('RateLimit-Reset', `${resetSeconds}`);
res.setHeader('X-RateLimit-Limit', `${effectiveConfig.max}`);
res.setHeader('X-RateLimit-Remaining', `${remaining}`);
res.setHeader('X-RateLimit-Reset', `${Math.ceil(bucket.resetAt / 1000)}`);
res.setHeader('X-RateLimit-Reset', `${resetEpochSeconds}`);

if (configOverride) res.setHeader('X-RateLimit-Policy', 'user-override');
if (configOverride || apiKeyOverride) res.setHeader('X-RateLimit-Policy', 'user-override');

if (bucket.count > effectiveConfig.max) {
const retryAfter = Math.max(1, resetSeconds);
res.setHeader('RateLimit-Remaining', '0');
res.setHeader('X-RateLimit-Remaining', '0');
res.status(429).json({ error: 'Too many requests' });
res.setHeader('Retry-After', `${retryAfter}`);
res.status(429).json({ error: 'Too many requests', retryAfter });
return;
}

Expand All @@ -238,42 +268,37 @@ export async function tieredRateLimit(
const tier: RateLimitTier = keyCtx?.tier ?? getTierFromEnvKey(rawKey);
const clientKey = keyCtx?.id ?? req.ip ?? 'anonymous';

if (useTokenBucket) {
try {
const result: TokenBucketResult = await checkTokenBucket(
clientKey,
tier,
req.method,
req.path,
keyCtx?.rateLimitOverride,
);

res.setHeader('X-RateLimit-Limit', result.limit);
res.setHeader('X-RateLimit-Remaining', result.remaining);
res.setHeader('X-RateLimit-Reset', result.resetAt);
res.setHeader('X-RateLimit-Tier', result.tier);
(req as Request & { rateLimitResult?: TokenBucketResult }).rateLimitResult = result;

if (!result.allowed) {
const retryAfter = Math.max(1, result.resetAt - Math.floor(Date.now() / 1000));
res.setHeader('Retry-After', retryAfter);
res.status(429).json({
error: 'Rate limit exceeded',
tier: result.tier,
retryAfter,
resetAt: new Date(result.resetAt * 1000).toISOString(),
});
return;
}

next();
try {
const result: TokenBucketResult = await checkTokenBucket(
clientKey,
tier,
req.method,
req.path,
keyCtx?.rateLimitOverride,
);

const resetSeconds = writeRateLimitHeaders(res, result);
(req as Request & { rateLimitResult?: TokenBucketResult }).rateLimitResult = result;

if (!result.allowed) {
const retryAfter = Math.max(1, resetSeconds);
res.setHeader('Retry-After', String(retryAfter));
res.status(429).json({
error: 'Rate limit exceeded',
tier: result.tier,
retryAfter,
resetAt: new Date(result.resetAt * 1000).toISOString(),
});
return;
} catch (err) {
logger.warn(`[rate-limit] Token bucket error, falling through: ${String(err)}`);
}

next();
return;
} catch (err) {
logger.warn(`[rate-limit] Token bucket error, falling through: ${String(err)}`);
}

if (keyCtx?.rateLimitOverride) {
if (keyCtx?.rateLimitOverride || !req.app) {
runLocalRateLimit(req, res, next);
return;
}
Expand Down
Loading