Skip to content

Commit a9ece5b

Browse files
leeyyileeyyi
committed
FEAT: 在模型请求失败时能够自动重试
1 parent 8080800 commit a9ece5b

24 files changed

Lines changed: 1641 additions & 303 deletions

‎docs/mkdocs/en/model.md‎

Lines changed: 71 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,7 @@ Models in tRPC-Agent have the following core features:
1010
- **Streaming response support**: Supports streaming output for real-time interactive experiences
1111
- **Multimodal capabilities**: Supports multimodal content processing including text, images, etc. (e.g., Hunyuan multimodal models)
1212
- **Prompt Cache support**: Provides unified prompt cache configuration across OpenAI, Anthropic, and LiteLLM routes to reduce repeated input cost for long prompts and multi-turn conversations
13+
- **Model retry support**: Supports configuring retry at the model layer. The SDK automatically retries when exceptions such as rate limits occur, and backs off using an exponential backoff strategy
1314
- **Extensible configuration**: Supports custom configuration options such as GenerateContentConfig, HttpOptions, client_args to meet various scenario requirements
1415

1516
## Quick Start
@@ -514,6 +515,76 @@ Different model services report different fields. OpenAI-compatible endpoints us
514515

515516
For a complete runnable example, see [examples/llmagent_with_prompt_cache](../../../examples/llmagent_with_prompt_cache/README.md).
516517

518+
### Model Retry
519+
520+
Model retry is useful when LLM requests encounter transient failures such as rate limits, timeouts, network fluctuations, or temporary service unavailability. By passing `ModelRetryConfig` when constructing a model, you can configure retry behavior at the model layer. When retryable exceptions occur, the SDK automatically retries the request and backs off according to the configured exponential backoff strategy.
521+
522+
`OpenAIModel`, `AnthropicModel`, and `LiteLLMModel` all support model retry. This capability is disabled by default and is enabled only when `model_retry_config` is explicitly provided.
523+
524+
#### Basic usage
525+
526+
```python
527+
from trpc_agent_sdk.configs import ExponentialBackoffConfig
528+
from trpc_agent_sdk.configs import ModelRetryConfig
529+
from trpc_agent_sdk.models import OpenAIModel
530+
531+
model = OpenAIModel(
532+
model_name="deepseek-chat",
533+
api_key="your-api-key",
534+
base_url="https://api.deepseek.com/v1",
535+
model_retry_config=ModelRetryConfig(
536+
num_retries=2, # Additional retry attempts after the initial request fails
537+
backoff=ExponentialBackoffConfig(
538+
initial_backoff=1.0, # Base delay in seconds before the first retry
539+
max_backoff=8.0, # Upper bound in seconds for a single retry delay
540+
multiplier=2.0, # Exponential backoff multiplier
541+
jitter=True, # Whether to apply full jitter to avoid synchronized retries
542+
),
543+
),
544+
)
545+
```
546+
547+
#### Configuration fields
548+
549+
| Field | Default | Description |
550+
|-------|---------|-------------|
551+
| `num_retries` | `2` | Additional retry attempts after the initial request fails; does not include the initial request. Set to `0` to disable additional retries |
552+
| `backoff.initial_backoff` | `1.0` | Base delay in seconds before the first retry |
553+
| `backoff.max_backoff` | `10.0` | Upper bound in seconds for a single retry delay |
554+
| `backoff.multiplier` | `2.0` | Exponential backoff multiplier, e.g. retry delays are computed as `initial_backoff * multiplier^attempt` |
555+
| `backoff.jitter` | `True` | Whether to apply full jitter. When enabled, the SDK waits for a random duration between `0` and the current backoff delay to avoid synchronized retries |
556+
557+
If the provider returns `Retry-After` or `retry-after-ms`, and the suggested delay is within a reasonable range, the SDK uses the server-suggested delay first. Otherwise, it computes the delay from the exponential backoff configuration.
558+
559+
#### Retry decisions
560+
561+
The SDK combines response headers, HTTP status codes, and provider SDK exception semantics to decide whether to retry. The general precedence is:
562+
563+
- `x-should-retry: true` forces the error to be treated as retryable, while `x-should-retry: false` forces it to be treated as non-retryable.
564+
- HTTP status `408`, `409`, `429`, and `>=500` are generally retryable.
565+
- Other `4xx` errors such as `400`, `401`, `403`, and `404` are generally not retried.
566+
- Timeout and connection-related OpenAI / Anthropic exceptions are generally retryable.
567+
- `LiteLLMModel` primarily uses the header and status information on LiteLLM-normalized exceptions.
568+
569+
#### Streaming behavior
570+
571+
To avoid duplicated output, model retry only happens before the current model call has produced user-visible content (text or tool calls). If a streaming response has already emitted partial content and then fails, the SDK returns a final error response directly instead of replaying the whole request.
572+
573+
Runner code does not need additional retry handling:
574+
575+
```python
576+
async for event in runner.run_async(
577+
user_id=user_id,
578+
session_id=session_id,
579+
new_message=user_content,
580+
):
581+
...
582+
```
583+
584+
If the model call encounters a retryable error before producing content, the SDK retries according to `ModelRetryConfig`. If the retry budget is exhausted or the error is non-retryable, the SDK surfaces an `LlmResponse` with `error_code` / `error_message`.
585+
586+
For a complete runnable example, see [examples/llmagent_with_model_retry](../../../examples/llmagent_with_model_retry/README.md).
587+
517588
### Custom HTTP Headers
518589

519590
Pass additional headers via `client_args`'s `default_headers` or `GenerateContentConfig`'s `HttpOptions`, suitable for gateways, proprietary platforms, or proxy environments. For example:

‎docs/mkdocs/zh/model.md‎

Lines changed: 58 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,7 @@ tRPC-Agent 内的模型具有以下核心特性：
1010
- **流式响应支持**：支持流式输出，实现实时交互体验
1111
- **多模态能力**：支持文本、图像等多模态内容处理（如 hunyuan 多模态模型）
1212
- **Prompt Cache 支持**：支持跨 OpenAI、Anthropic 与 LiteLLM 路由的统一 prompt cache 配置，降低长提示词和多轮会话的重复输入成本
13+
- **模型重试支持**：支持在模型层配置重试，SDK 将在限流等异常发生时自动重试，并按指数退避策略进行退避
1314
- **可扩展配置**：支持 GenerateContentConfig、HttpOptions、client_args 等自定义配置项，满足不同场景需求
1415

1516
## 快速上手
@@ -514,6 +515,63 @@ async for event in runner.run_async(...):
514515

515516
完整可运行示例见 [examples/llmagent_with_prompt_cache](../../../examples/llmagent_with_prompt_cache/README.md)。
516517

518+
### 模型重试
519+
520+
模型重试适用于 LLM 请求遇到限流、超时、网络抖动、临时服务不可用等瞬时错误的场景。通过在模型构造时传入 `ModelRetryConfig`，可以让 SDK 在模型层统一处理重试，业务代码和 Runner 调用无需自己实现重试循环。
521+
522+
目前 `OpenAIModel`、`AnthropicModel` 和 `LiteLLMModel` 均支持模型重试。该能力默认关闭，只有显式传入 `model_retry_config` 时才会启用。
523+
524+
#### 基本用法
525+
526+
```python
527+
from trpc_agent_sdk.configs import ExponentialBackoffConfig
528+
from trpc_agent_sdk.configs import ModelRetryConfig
529+
from trpc_agent_sdk.models import OpenAIModel
530+
531+
model = OpenAIModel(
532+
model_name="deepseek-chat",
533+
api_key="your-api-key",
534+
base_url="https://api.deepseek.com/v1",
535+
model_retry_config=ModelRetryConfig(
536+
num_retries=2, # 初始请求失败后的额外重试次数，不包含第一次请求
537+
backoff=ExponentialBackoffConfig(
538+
initial_backoff=1.0, # 第一次重试前的基础等待时间，单位秒
539+
max_backoff=8.0, # 单次重试等待时间上限，单位秒
540+
multiplier=2.0, # 指数退避倍数
541+
jitter=True, # 是否启用 full jitter，避免并发请求同时重试
542+
),
543+
),
544+
)
545+
```
546+
547+
#### 配置项
548+
549+
| 配置项 | 默认值 | 说明 |
550+
|--------|--------|------|
551+
| `num_retries` | `2` | 初始请求失败后的额外重试次数，不包含第一次请求；设置为 `0` 表示不额外重试 |
552+
| `backoff.initial_backoff` | `1.0` | 第一次重试前的基础等待时间，单位秒 |
553+
| `backoff.max_backoff` | `10.0` | 单次重试等待时间上限，单位秒 |
554+
| `backoff.multiplier` | `2.0` | 指数退避倍数，例如第 1、2、3 次重试前分别按 `initial_backoff * multiplier^attempt` 计算 |
555+
| `backoff.jitter` | `True` | 是否启用 full jitter；开启后会在 `0` 到当前退避时间之间随机等待，避免并发请求同时重试 |
556+
557+
如果 provider 返回 `Retry-After` 或 `retry-after-ms`，且等待时间在合理范围内，SDK 会优先使用服务端建议的等待时间；否则使用指数退避配置计算等待时间。
558+
559+
#### 重试判定
560+
561+
SDK 会结合响应 header、HTTP status 和 provider SDK 的异常语义判断是否重试。通用优先级如下：
562+
563+
- `x-should-retry: true` 会强制视为可重试，`x-should-retry: false` 会强制视为不可重试。
564+
- HTTP status `408`、`409`、`429` 和 `>=500` 通常视为可重试。
565+
- 其他 `4xx` 错误（如 `400`、`401`、`403`、`404`）通常不会重试。
566+
- OpenAI / Anthropic 的超时、连接类异常通常视为可重试。
567+
- `LiteLLMModel` 优先使用 LiteLLM 归一化异常上的 header 和 status 信息做判断。
568+
569+
#### 流式输出注意事项
570+
571+
为了避免重复输出内容，模型重试只会发生在本次模型调用尚未产出用户可见内容（文本或工具调用）之前。如果流式响应已经产出部分内容后又发生异常，SDK 会直接返回最终错误响应，而不会重放整个请求。
572+
573+
完整可运行示例见 [examples/llmagent_with_model_retry](../../../examples/llmagent_with_model_retry/README.md)。
574+
517575
### 自定义 HTTP Header
518576

519577
通过 `client_args` 的 `default_headers` 或 `GenerateContentConfig` 的 `HttpOptions` 传递额外头部，适用于网关、专有平台或代理环境。例如：

‎examples/llmagent_with_model_retry/.env‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
# Copy this file or edit it in place before running the example.
2+
# The example uses an OpenAI-compatible endpoint.
3+
4+
TRPC_AGENT_API_KEY=
5+
TRPC_AGENT_BASE_URL=
6+
TRPC_AGENT_MODEL_NAME=
7+
8+
# Optional model retry tuning.
9+
# num_retries is the number of retry attempts in addition to the initial call.
10+
TRPC_AGENT_MODEL_RETRY_NUM_RETRIES=2
11+
TRPC_AGENT_MODEL_RETRY_INITIAL_BACKOFF=1.0
12+
TRPC_AGENT_MODEL_RETRY_MAX_BACKOFF=8.0
13+
TRPC_AGENT_MODEL_RETRY_BACKOFF_MULTIPLIER=2.0
14+
TRPC_AGENT_MODEL_RETRY_JITTER=true

‎examples/llmagent_with_model_retry/README.md‎

Lines changed: 157 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,157 @@
1+
# LLM Agent 模型重试示例
2+
3+
本示例演示如何在模型构造时传入 `ModelRetryConfig`，让业务代码不用自己实现重试逻辑；当 LLM 请求遇到限流、超时、网络波动等瞬时错误时，SDK 会自动重试。
4+
5+
## 关键特性
6+
7+
- **按需开启重试**：只有显式传入 `ModelRetryConfig` 的模型才会启用 SDK 托管重试。
8+
- **一套配置，多种模型可用**：同一个 `ModelRetryConfig` 可以传给 OpenAI、Anthropic、LiteLLM 等模型实现，统一控制重试次数和指数退避。
9+
- **Provider 自己判断是否重试**：OpenAI、Anthropic、LiteLLM 会按各自 SDK 的错误语义、响应 header 和 HTTP status 判断是否重试。
10+
- **避免重复输出内容**：流式输出已经产生内容后如果再失败，错误会直接透出，不会重试并重复输出内容。
11+
- **优先使用服务端等待时间**：默认会读取 `Retry-After` / `retry-after-ms`，有服务端等待时间时优先使用。
12+
13+
## Agent 层级结构说明
14+
15+
本例是单 Agent 示例，重试配置绑定在模型上：
16+
17+
```text
18+
weather_retry_agent (LlmAgent)
19+
├── model: OpenAIModel(..., model_retry_config=ModelRetryConfig(...))
20+
├── tool: get_weather_report(city)
21+
└── runner: 无自定义重试逻辑
22+
```
23+
24+
关键文件：
25+
26+
- [examples/llmagent_with_model_retry/agent/agent.py](./agent/agent.py)：创建模型并注入 `ModelRetryConfig`
27+
- [examples/llmagent_with_model_retry/agent/config.py](./agent/config.py)：读取模型连接与重试环境变量
28+
- [examples/llmagent_with_model_retry/agent/tools.py](./agent/tools.py)：天气工具实现
29+
- [examples/llmagent_with_model_retry/agent/prompts.py](./agent/prompts.py)：提示词
30+
- [examples/llmagent_with_model_retry/run_agent.py](./run_agent.py)：运行入口，展示业务层无需手写重试
31+
32+
## 关键代码解释
33+
34+
### 1) 创建重试配置
35+
36+
`agent/config.py` 从环境变量构造：
37+
38+
```python
39+
from trpc_agent_sdk.configs import ExponentialBackoffConfig
40+
from trpc_agent_sdk.configs import ModelRetryConfig
41+
42+
ModelRetryConfig(
43+
num_retries=2, # 最多重试 2 次（不包含首次请求）
44+
backoff=ExponentialBackoffConfig(
45+
initial_backoff=1.0, # 首次重试前等待 1 秒
46+
max_backoff=8.0, # 单次重试等待时间最多 8 秒
47+
multiplier=2.0, # 每次失败后等待时间按 2 倍递增
48+
jitter=True, # 加入随机抖动，避免并发重试同时打到服务
49+
),
50+
)
51+
```
52+
53+
### 2) 注入模型
54+
55+
`agent/agent.py` 将配置传给模型构造器：
56+
57+
```python
58+
OpenAIModel(
59+
model_name=model_name,
60+
api_key=api_key,
61+
base_url=base_url,
62+
model_retry_config=retry_config, # 将重试配置注入模型，调用失败时按配置自动重试
63+
)
64+
```
65+
66+
### 3) 配置字段说明
67+
68+
- `num_retries`：初始请求失败后的额外重试次数，不包含第一次请求。
69+
- `initial_backoff`：第一次重试前的基础等待时间，单位秒。
70+
- `max_backoff`：单次等待时间上限，单位秒。
71+
- `multiplier`：指数退避倍数。
72+
- `jitter`：是否启用 full jitter，避免并发请求同时重试。
73+
74+
SDK 会优先尊重 provider 返回的 `Retry-After` / `retry-after-ms` 服务端建议等待时间。
75+
76+
### 4) Runner 不需要重试逻辑
77+
78+
`run_agent.py` 仍然只调用：
79+
80+
```python
81+
async for event in runner.run_async(...):
82+
...
83+
```
84+
85+
如果模型调用在产出内容前遇到可重试错误，SDK 会按 `ModelRetryConfig` 自动重试。
86+
87+
## 会重试和不会重试的场景
88+
89+
不同 provider 会按各自 SDK 的错误语义判断异常类型；对于带响应信息的错误，通用优先级如下：
90+
91+
### 会重试
92+
93+
- 响应 header `x-should-retry: true`。
94+
- HTTP status 为 `408` / `409` / `429` / `>=500`。
95+
- OpenAI / Anthropic 相关的超时、连接类瞬时错误。
96+
97+
### 不会重试
98+
99+
- 响应 header `x-should-retry: false`。
100+
- HTTP status 为其他 `4xx` 错误，例如 `400` / `401` / `403` / `404`。
101+
- 重试次数已耗尽。
102+
- 流式输出已经产生内容后才发生的错误。
103+
104+
## 环境与运行
105+
106+
### 环境要求
107+
108+
- Python 3.10+
109+
110+
### 安装步骤
111+
112+
```bash
113+
git clone https://github.com/trpc-group/trpc-agent-python.git
114+
cd trpc-agent-python
115+
python3 -m venv .venv
116+
source .venv/bin/activate
117+
pip3 install -e .
118+
```
119+
120+
### 环境变量要求
121+
122+
在 [examples/llmagent_with_model_retry/.env](./.env) 中配置（或通过 `export` 设置）：
123+
124+
```bash
125+
# 必填：模型连接配置
126+
TRPC_AGENT_API_KEY=<your-api-key>
127+
TRPC_AGENT_BASE_URL=<openai-compatible-base-url>
128+
TRPC_AGENT_MODEL_NAME=<model-name>
129+
130+
# 可选：模型重试配置；不设置时使用示例默认值
131+
TRPC_AGENT_MODEL_RETRY_NUM_RETRIES=2
132+
TRPC_AGENT_MODEL_RETRY_INITIAL_BACKOFF=1.0
133+
TRPC_AGENT_MODEL_RETRY_MAX_BACKOFF=8.0
134+
TRPC_AGENT_MODEL_RETRY_BACKOFF_MULTIPLIER=2.0
135+
TRPC_AGENT_MODEL_RETRY_JITTER=true
136+
```
137+
138+
### 运行命令
139+
140+
```bash
141+
cd examples/llmagent_with_model_retry
142+
python3 run_agent.py
143+
```
144+
145+
## 运行结果示例
146+
147+
```text
148+
Model retry enabled: {'num_retries': 2, 'backoff': {'initial_backoff': 1.0, 'max_backoff': 8.0, 'multiplier': 2.0, 'jitter': True}}
149+
Session ID: fdb9e370...
150+
User: What's the current weather in Beijing?
151+
Assistant:
152+
Invoke Tool: get_weather_report({'city': 'Beijing'})
153+
Tool Result: {'temperature': '25°C', 'condition': 'Sunny', 'humidity': '60%'}
154+
155+
# 当模型服务返回可重试错误时，SDK 会在模型层自动重试：
156+
[WARNING] Model call failed (exception=RateLimitError); retrying in 0.46s (attempt 1/2).
157+
```

‎examples/llmagent_with_model_retry/agent/__init__.py‎

Whitespace-only changes.

‎examples/llmagent_with_model_retry/agent/agent.py‎

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
# Tencent is pleased to support the open source community by making tRPC-Agent-Python available.
2+
#
3+
# Copyright (C) 2026 Tencent. All rights reserved.
4+
#
5+
# tRPC-Agent-Python is licensed under Apache-2.0.
6+
"""Agent module for the model retry example."""
7+
8+
from trpc_agent_sdk.agents import LlmAgent
9+
from trpc_agent_sdk.models import LLMModel
10+
from trpc_agent_sdk.models import OpenAIModel
11+
from trpc_agent_sdk.tools import FunctionTool
12+
13+
from .config import get_model_config
14+
from .config import get_model_retry_config
15+
from .prompts import INSTRUCTION
16+
from .tools import get_weather_report
17+
18+
19+
def _create_model() -> LLMModel:
20+
"""Create an OpenAI-compatible model with SDK-managed retry enabled."""
21+
api_key, base_url, model_name = get_model_config()
22+
retry_config = get_model_retry_config()
23+
print(f"Model retry enabled: {retry_config.model_dump()}")
24+
return OpenAIModel(
25+
model_name=model_name,
26+
api_key=api_key,
27+
base_url=base_url,
28+
model_retry_config=retry_config,
29+
)
30+
31+
32+
def create_agent() -> LlmAgent:
33+
"""Create a weather agent that uses model-level retry."""
34+
return LlmAgent(
35+
name="weather_retry_agent",
36+
description="A weather assistant with SDK-managed model retry enabled.",
37+
model=_create_model(),
38+
instruction=INSTRUCTION,
39+
tools=[FunctionTool(get_weather_report)],
40+
)
41+
42+
43+
root_agent = create_agent()

0 commit comments

Comments
 (0)