Skip to content

Commit 706bcb7

Browse files
beardnickbeardnick
committed
doc: update documentation
1 parent f86b98f commit 706bcb7

2 files changed

Lines changed: 174 additions & 24 deletions

File tree

‎docs/en/latest/plugins/ai-rate-limiting.md‎

Lines changed: 87 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -35,22 +35,36 @@ description: The ai-rate-limiting Plugin enforces token-based rate limiting for
3535

3636
## Description
3737

38-
The `ai-rate-limiting` Plugin enforces token-based rate limiting for requests sent to LLM services. It helps manage API usage by controlling the number of tokens consumed within a specified time frame, ensuring fair resource allocation and preventing excessive load on the service. It is often used with [`ai-proxy`](./ai-proxy.md) or [`ai-proxy-multi`](./ai-proxy-multi.md) plugin.
38+
The `ai-rate-limiting` Plugin enforces token-based rate limiting for requests sent to LLM services. It helps manage API usage by controlling the number of tokens consumed within a specified time frame, ensuring fair resource allocation and preventing excessive load on the service. Token counters can be stored locally on each APISIX node or persisted to Redis/Redis Cluster to coordinate quotas across replicas. It is often used with [`ai-proxy`](./ai-proxy.md) or [`ai-proxy-multi`](./ai-proxy-multi.md) plugin.
3939

4040
## Attributes
4141

42-
| Name | Type | Required | Default | Valid values | Description |
43-
|------------------------------|----------------|----------|----------|---------------------------------------------------------|-------------|
44-
| limit | integer | False | | >0 | The maximum number of tokens allowed within a given time interval. At least one of `limit` and `instances.limit` should be configured. |
45-
| time_window | integer | False | | >0 | The time interval corresponding to the rate limiting `limit` in seconds. At least one of `time_window` and `instances.time_window` should be configured. |
46-
| show_limit_quota_header | boolean | False | true | | If true, includes `X-AI-RateLimit-Limit-*`, `X-AI-RateLimit-Remaining-*`, and `X-AI-RateLimit-Reset-*` headers in the response, where `*` is the instance name. |
42+
| Name | Type | Required | Default | Valid values | Description |
43+
|------------------------------|----------------|----------|--------------|---------------------------------------------------------|-------------|
44+
| limit | integer | False | | >0 | The maximum number of tokens allowed within a given time interval. At least one of `limit` and `instances.limit` should be configured. |
45+
| time_window | integer | False | | >0 | The time interval corresponding to the rate limiting `limit` in seconds. At least one of `time_window` and `instances.time_window` should be configured. |
46+
| show_limit_quota_header | boolean | False | true | | If true, includes `X-AI-RateLimit-Limit-*`, `X-AI-RateLimit-Remaining-*`, and `X-AI-RateLimit-Reset-*` headers in the response, where `*` is the instance name. |
4747
| limit_strategy | string | False | total_tokens | [total_tokens, prompt_tokens, completion_tokens] | Type of token to apply rate limiting. `total_tokens` is the sum of `prompt_tokens` and `completion_tokens`. |
48-
| instances | array[object] | False | | | LLM instance rate limiting configurations. |
49-
| instances.name | string | True | | | Name of the LLM service instance. |
50-
| instances.limit | integer | True | | >0 | The maximum number of tokens allowed within a given time interval for an instance. |
51-
| instances.time_window | integer | True | | >0 | The time interval corresponding to the rate limiting `limit` in seconds for an instance. |
52-
| rejected_code | integer | False | 503 | [200, 599] | The HTTP status code returned when a request exceeding the quota is rejected. |
53-
| rejected_msg | string | False | | | The response body returned when a request exceeding the quota is rejected. |
48+
| instances | array[object] | False | | | LLM instance rate limiting configurations. |
49+
| instances.name | string | True | | | Name of the LLM service instance. |
50+
| instances.limit | integer | True | | >0 | The maximum number of tokens allowed within a given time interval for an instance. |
51+
| instances.time_window | integer | True | | >0 | The time interval corresponding to the rate limiting `limit` in seconds for an instance. |
52+
| rejected_code | integer | False | 503 | [200, 599] | The HTTP status code returned when a request exceeding the quota is rejected. |
53+
| rejected_msg | string | False | | | The response body returned when a request exceeding the quota is rejected. |
54+
| policy | string | False | local | [local, redis, redis-cluster] | Storage policy for the rate limiting counter. Use `redis` or `redis-cluster` to share quotas across APISIX nodes or persist counters across restarts. |
55+
| allow_degradation | boolean | False | false | | If true, allows APISIX to continue proxying traffic when the Redis backend is unavailable. |
56+
| redis_host | string | False | | | Address of the Redis node. Required when `policy` is `redis`. |
57+
| redis_port | integer | False | 6379 | >=1 | Port of the Redis node when `policy` is `redis`. |
58+
| redis_username | string | False | | | Username for Redis ACL authentication when `policy` is `redis`. Leave empty when using `requirepass`. |
59+
| redis_password | string | False | | | Password for the Redis node when `policy` is `redis` or `redis-cluster`. |
60+
| redis_database | integer | False | 0 | >=0 | Database index for Redis when `policy` is `redis`. |
61+
| redis_timeout | integer | False | 1000 | >=1 | Redis operation timeout in milliseconds when `policy` is `redis` or `redis-cluster`. |
62+
| redis_ssl | boolean | False | false | | If true, uses TLS when connecting to Redis for the `redis` policy. |
63+
| redis_ssl_verify | boolean | False | false | | If true, verifies the Redis server certificate when `policy` is `redis` and TLS is enabled. |
64+
| redis_cluster_nodes | array[string] | False | | | List of Redis Cluster node addresses (for example, `["10.0.0.1:6379","10.0.0.2:6379"]`). Required when `policy` is `redis-cluster`. |
65+
| redis_cluster_name | string | False | | | Cluster name used by Redis Cluster clients. Required when `policy` is `redis-cluster`. |
66+
| redis_cluster_ssl | boolean | False | false | | If true, uses TLS when connecting to Redis Cluster. |
67+
| redis_cluster_ssl_verify | boolean | False | false | | If true, verifies the Redis Cluster server certificate when TLS is enabled. |
5468

5569
## Examples
5670

@@ -138,6 +152,67 @@ You should receive a response similar to the following:
138152

139153
If the rate limiting quota of 300 prompt tokens has been consumed in a 30-second window, all additional requests will be rejected.
140154

155+
### Share Quotas Across Gateways with Redis
156+
157+
By default, `ai-rate-limiting` keeps counters in the memory of each APISIX node. When you run multiple gateways or need quotas that survive restarts, switch the `policy` to `redis` or `redis-cluster` so every node consults the same Redis backend. You can also set `allow_degradation` to `true` to keep proxying even if Redis is temporarily unreachable.
158+
159+
The following example builds on the previous Route and persists the counter to Redis with TLS enabled:
160+
161+
```shell
162+
curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
163+
-H "X-API-KEY: ${admin_key}" \
164+
-d '{
165+
"id": "ai-rate-limiting-redis-route",
166+
"uri": "/anything",
167+
"methods": ["POST"],
168+
"plugins": {
169+
"ai-proxy": {
170+
"provider": "openai",
171+
"auth": {
172+
"header": {
173+
"Authorization": "Bearer '"$OPENAI_API_KEY"'"
174+
}
175+
},
176+
"options": {
177+
"model": "gpt-4o-mini",
178+
"max_tokens": 256
179+
}
180+
},
181+
"ai-rate-limiting": {
182+
"limit": 1200,
183+
"time_window": 60,
184+
"policy": "redis",
185+
"redis_host": "redis.internal",
186+
"redis_port": 6380,
187+
"redis_password": "'"$REDIS_PASSWORD"'",
188+
"redis_ssl": true,
189+
"redis_ssl_verify": true,
190+
"allow_degradation": true
191+
}
192+
}
193+
}'
194+
```
195+
196+
To use Redis Cluster instead, set `"policy": "redis-cluster"` and configure the `redis_cluster_nodes`, `redis_cluster_name`, and optional TLS fields:
197+
198+
```json
199+
"ai-rate-limiting": {
200+
"limit": 1200,
201+
"time_window": 60,
202+
"policy": "redis-cluster",
203+
"redis_cluster_nodes": [
204+
"10.0.0.10:6379",
205+
"10.0.0.11:6379",
206+
"10.0.0.12:6379"
207+
],
208+
"redis_cluster_name": "apisix-rate-limit",
209+
"redis_password": "my-secret-password",
210+
"redis_cluster_ssl": true
211+
}
212+
```
213+
214+
With either Redis-backed policy, requests that spend tokens on one APISIX instance immediately affect the quota seen by all of the others.
215+
141216
### Rate Limit One Instance Among Multiple
142217

143218
The following example demonstrates how you can use `ai-proxy-multi` to configure two models for load balancing, forwarding 80% of the traffic to one instance and 20% to the other. Additionally, use `ai-rate-limiting` to configure token-based rate limiting on the instance that receives 80% of the traffic, such that when the configured quota is fully consumed, the additional traffic will be forwarded to the other instance.

0 commit comments

Comments
 (0)