Rate Limit Policy: Enhanced Cost Extraction & Multi-Dimensional Limits #651
Replies: 2 comments 5 replies
-
|
Beta Was this translation helpful? Give feedback.
-
|
Beta Was this translation helpful? Give feedback.
-
|
My initial idea was to create a separate policy for Token-Based Rate Limiting that reuses the core rate-limiting functionality from the existing rate limit policy. Below is the OnRequest function from the current Rate Limiting policy: // OnRequest performs rate limit check
func (p *RateLimitPolicy) OnRequest(
ctx *policy.RequestContext,
params map[string]interface{},
) policy.RequestAction {
// 1. Extract rate limit key
key := p.extractRateLimitKey(ctx)
// 2. Extract cost parameter (defaults to 1 for backwards compatibility)
cost := int64(1)
if costVal, ok := params["cost"].(float64); ok {
cost = int64(costVal)
if cost < 1 {
cost = 1 // Ensure minimum cost of 1
}
}
// 3. Check rate limit with cost (weighted rate limiting)
result, err := p.limiter.AllowN(context.Background(), key, cost)
...For the Token-Based Rate Limiting policy, we can reuse the following call: result, err := p.limiter.AllowN(context.Background(), key, cost)This can be achieved by importing the same Go module. In this context, both For example, in a Token-Based Rate Limiting policy:
This approach allows different policies to share the same core rate-limiting logic while customizing key generation and cost calculation as needed. |
Beta Was this translation helpful? Give feedback.
-
|
With this approach, I would have to copy and paste most of the code, right? I would need to reimplement memory/Redis initialization logic, memory caching, and related components. The token-based rate limit is just an example I used here. Even in a normal use case, we might need support for multidimensional rate limiting. Similarly, cost extraction is not specific to LLM token-based use cases. For example: If the core rate-limit policy provided these functionalities, I would only need to write wrapper policies that use a delegation approach to map the parameters and invoke the onRequest and onResponse methods of the core rate-limit policy. |
Beta Was this translation helpful? Give feedback.
-
|
To be clear, there will be separate policies implemented for token-based rate limiting, subscription-based rate limiting, and so on. However, if we make the core rate-limit policy fully functional and flexible, the wrapper policies won’t need to reimplement most of the logic. |
Beta Was this translation helpful? Give feedback.
-
In the proposed approach, params:
quotas:
# Dimension 2: Prompt tokens
- name: "prompt-tokens"
limit: 50000
duration: "1h"
keyExtraction:
- type: header
key: X-User-ID
costExtraction:
enabled: true
sources:
- type: response_body
jsonPath: "$.usage.prompt_tokens"
multiplier: 1.0
default: 0How can different limits be applied for different durations?
Would I have to duplicate the same quota component? for example "prompt-tokens-per-hour" and "prompt-tokens-per-month"? |
Beta Was this translation helpful? Give feedback.
-
|
We can improve this proposed design to support multiple limit-duration per quota as we can get better performance with this. I will update the proposal. So the approach for your scenario will be like |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
Rate Limit Policy: Enhanced Cost Extraction & Multi-Dimensional Limits
Summary
The current rate limit policy has limitations for advanced use cases like token-based rate limiting for LLM APIs where:
Current Limitations
1. Static Cost Only
2. Single Key Extraction (Global)
Cannot have different keys for different limits (e.g., per-user request limit + per-org token limit).
Proposed Solution
Core Idea: Nested Limits within Quotas
Instead of a single limit and duration per quota, we now support a limits array. This allows one set of extraction logic (Key and Cost) to apply to multiple time buckets.
This naturally enables multi-dimensional rate limiting without adding new top-level concepts.
Cost Extraction Feature
Schema
Supported Extraction Sources
request_bodyresponse_bodyrequest_metadataresponse_metadatarequest_headerresponse_headerCost Calculation
When multiple sources are defined:
If a source extraction fails, it contributes
0. If ALL sources fail,defaultis used.Configuration Examples
Example 1: Simple Response Body Extraction
Example 2: LLM Token-Based with Weighted Multipliers
Calculation:
Example 3: Multi-Dimensional (Requests + Separate Token Limits)
Example 4: Metadata-Based Cost
Example 5: Different Keys per Quota
Schema Changes Summary
Before (Current)
After (Proposed)
Beta Was this translation helpful? Give feedback.
All reactions