Budget Rate Limit Policy

[Tharsanan1]

Budget Rate Limit Policy

---

Summary

Introduce a budget-ratelimit policy that limits API usage based on monetary cost calculated from LLM token consumption. Users define pricing per token type (cost per 1M tokens) and set budget limits. The policy enforces spending limits like "$100/hour" or "$1000/day".

---

Motivation

Problem Statement

Organizations using LLM APIs need to control costs, not just token counts. Different token types (prompt vs completion) have different pricing, and teams need budget controls at various time windows (hourly, daily, monthly).

Currently, there's no simple way to:

• Set dollar-based limits on API usage
• Apply different pricing for prompt vs completion tokens
• Enforce multi-tier budgets (burst + sustained limits)

Goals

1. Enable cost-based rate limiting with configurable pricing
2. Support separate budgets for prompt, completion, and total costs
3. Allow multiple budget windows (e.g., $10/min burst, $100/hour, $1000/day)
4. Simple configuration focused on pricing and budgets

Non-Goals

• Dynamic model-based pricing (in the future we need this but currently the advanced ratelimit does not support this)
• Currency conversion or formatting
• Billing integration

---

Proposed Solution

Architecture

Use the delegation pattern (like basic-ratelimit):

• Expose a budget-focused configuration with pricing
• Calculate cost multipliers: costPer1MTokens / 1,000,000
• Transform to advanced-ratelimit quotas with multipliers
• Reuse existing algorithms and backends

User-Facing Configuration

# Define pricing
pricing:
  promptTokens:
    costPer1MTokens: 10.00      # $10 per 1M prompt tokens
    jsonPath: "$.usage.prompt_tokens"
  completionTokens:
    costPer1MTokens: 30.00      # $30 per 1M completion tokens
    jsonPath: "$.usage.completion_tokens"

# Set budget limits
totalBudget:
  limits:
    - limit: 10.00       # $10/minute burst
      duration: "1m"
    - limit: 100.00      # $100/hour
      duration: "1h"
    - limit: 1000.00     # $1000/day
      duration: "24h"

Cost Calculation

cost = tokens × (costPer1MTokens / 1,000,000)

Example:
- 5000 prompt tokens at $10/1M = 5000 × 0.00001 = $0.05
- 2000 completion tokens at $30/1M = 2000 × 0.00003 = $0.06
- Total cost = $0.11

Key Features

Feature	Description
3 Budget Types	Prompt, completion, and total budgets
Per-Token Pricing	Different costs for prompt vs completion
Multi-window Budgets	Multiple limit/duration pairs per budget
Automatic Total	Sum of prompt + completion costs
Extraction Failure Handling	Configurable: skip, default value, or reject

---

Use Cases

1. Enterprise Cost Control

# Department-level monthly budget
totalBudget:
  limits:
    - limit: 10000.00    # $10,000/month
      duration: "720h"   # 30 days

2. Burst Protection

# Prevent runaway costs from bugs/attacks
totalBudget:
  limits:
    - limit: 50.00       # $50/minute max burst
      duration: "1m"
    - limit: 500.00      # $500/hour sustained
      duration: "1h"

3. Token-Type Specific Budgets

# Control expensive completion tokens separately
completionBudget:
  limits:
    - limit: 80.00
      duration: "1h"

References

• gateway/policies/advanced-ratelimit/ - Core rate limiting engine
• gateway/policies/basic-ratelimit/ - Delegation pattern example