Skip to content

feat(subagent): adaptive rate-limit governor - #43

Open
asto18089 wants to merge 2 commits into
Pinvou:pinvou3-cleanfrom
asto18089:feature/swarm-rate-limit-adaptive
Open

feat(subagent): adaptive rate-limit governor#43
asto18089 wants to merge 2 commits into
Pinvou:pinvou3-cleanfrom
asto18089:feature/swarm-rate-limit-adaptive

Conversation

@asto18089

@asto18089 asto18089 commented Sep 5, 2026

Copy link
Copy Markdown

摘要

蜂群模式(父仓 PR 配对)下子智能体数量可达底座设计上限(128 并发),会大量触发模型提供商的 429 并行限流。本 PR 为子智能体 launch gate 增加限流自适应调度:

  • launch_gate 从固定容量 tokio::sync::Semaphore 改为自定义 DynamicGate(可收缩容量;Semaphore 收缩只能换 Arc,有 permit 在手时静默失效)。
  • 新增 RateLimitGovernor(引擎级共享,经 SubAgentRuntime 派生树继承):60s 滑动窗口统计 429 率,AIMD 策略——窗口内限流 ≥2 次或比例 >30% 容量减半,≥4 次暂停新 launch(排队原因 queued: waiting for provider rate-limit recovery);限流事件老化后按「连续成功」或时间驱动探针(排队子智能体周期调用 recover_if_window_drained,避免在飞集群清空后队列冻结到 wall-time 超时)以 1/4 容量恢复、加性回升。运行时改 launch 并发经 governor 生效,不会静默解除暂停。
  • 429 重试尊重 Retry-After;无则指数退避 + full jitter(250ms 起、cap 120s)错峰防 thundering herd;QuotaExhausted 不算临时限流,走既有失败路径。
  • fleet governor 在 spawn_background_with_assignment_options 汇聚点为全部 spawn 路径统一 stamp(交互会话 agent 工具链、Op::SpawnSubAgent、direct workflow 根 runtime),整棵派生树经 SubAgentRuntime 继承;不 stamp 的裸 runtime(测试、tool-only)才保持 None

验证

  • cargo fmt --checkcargo clippy -p codewhale-tui --all-targets 新代码 0 警告
  • cargo test -p codewhale-tui --lib subagent:: 491 通过 0 失败(含 governor 12 个单测:窗口统计、AIMD 升降、暂停/恢复/时间自愈边界、gate 取消重派发);cargo test --lib forkguard_ 60 通过 0 失败

配套

父仓 PR(gitlink 已指向本 head 6d81cfa83,评审修复见下方「评审跟进」);合并后需发布 r14(tag pinvou-v0.9.5-r14)并由父仓收尾更新 docs/fork-modifications.mdscripts/fork-guard.shPUBLISHED_HEAD/PUBLISHED_COMMITS(37→39)。父仓已在同 PR 内完成 docs/fork-modifications.md(zh/en)与 T5 指纹登记。

评审跟进(2026-09-06)

审计发现三处实质问题,已随 head 6d81cfa83 修复:

  1. governor 未接线:原实现只有 direct workflow 根 runtime stamp 了 governor,主用的交互会话 agent 工具链(core/engine.rs 根 runtime → spawn_background_*)整棵树 governor 恒为 None,AIMD 在主用例上是死代码。现改为 manager 在 spawn 汇聚点统一 stamp,mod.rs 失实注释同步修正。
  2. 暂停后活性缺陷:解除 paused 需要「窗口排空后新到达的 success 事件」;若在飞集群在窗口排空前全部终结,排队队列会冻结到各自 wall-time 超时(默认 30 分钟)。新增排队期时间驱动探针,窗口排空即恢复 1/4 容量。
  3. DynamicGate 丢唤醒竞态:原「唤醒后由 waiter 复查」设计下,waiter 被唤醒后、复查前被取消会吞掉唤醒(无后续 release 时槽位永久空转)。改为 permit 经 oneshot 预先计数交接,被取消 waiter 的 permit 随 future drop 自动重派发。

另含:set_max_capacity 改为「未暂停时即应用配置容量(可升可降)、暂停时保持 0」,运行时改限不再静默解除暂停;摘除过时 #[allow(dead_code)];新增 3 条 forkguard_* 行为测试钉住自愈/暂停保持/取消重派发。

Signed-off-by: asto asto18089@126.com

Provider 429 storms from a large subagent fleet now feed a shared
RateLimitGovernor: a sliding window drives AIMD shrink of the launch
gate capacity (halve on throttling, pause at sustained 429s, additive
recovery on consecutive successes). 429 retries honor Retry-After with
full-jitter exponential backoff (cap 120s). The root runtime is
stamped with the fleet governor so descendant attempts report into
the window.

Signed-off-by: asto <asto18089@126.com>
@github-actions

github-actions Bot commented Sep 5, 2026

Copy link
Copy Markdown

Thanks @asto18089 for taking the time to contribute.

This repository is observing a maintainer-managed PR intake gate in dry-run mode, so this pull request is staying open. This note helps maintainers prepare the allowlist before any enforcement is considered.

Please read CONTRIBUTING.md for the expected contribution shape. A maintainer can grant recurring PR access by commenting /lgtm on a pull request.

Review follow-up to the adaptive rate-limit governor:

- Stamp the fleet governor on every manager-spawned runtime at the
  `spawn_background_with_assignment_options` chokepoint. Only the direct
  workflow root was stamped before, so the interactive `agent` tool
  chain (the primary swarm path) never reported 429s and the AIMD
  scheduler stayed inert there.
- Time-driven recovery: queued launches periodically probe the governor
  so a pause lifts once the window drains even when the in-flight fleet
  finished without a final success. Without it the queue froze until
  each queued child hit its wall-time deadline.
- `set_max_capacity` now applies the configured launch capacity to the
  live gate directly (raising or lowering) but keeps capacity 0 while
  paused, so runtime limit changes cannot silently lift a pause.
  `with_launch_concurrency`/`update_runtime_limits` route through it.
- `DynamicGate` grants queued waiters an already-counted permit through
  a oneshot channel: a waiter cancelled after dispatch drops the
  permit, whose `Drop` re-dispatches the slot instead of losing the
  wakeup.
- Drop the stale `#[allow(dead_code)]` on `rate_limit_governor` and fix
  the spawn-stamping doc comments.
- Add `forkguard_rate_limit_governor_*` and `forkguard_dynamic_gate_*`
  behavior tests (window-drain recovery, pause survives a limit
  change, cancelled-waiter grant re-dispatch).

Signed-off-by: asto <asto18089@126.com>
@asto18089

Copy link
Copy Markdown
Author

审阅结论(父仓 PR 配对审计)

结论:问题真实(蜂群 128 并发下 429 锁步重试 + t0 洪峰,现有机制防不住)、方案对症(准入层 AIMD 是正确 scope,DynamicGateSemaphore 无法收缩确属必要)、无重复造轮子。 但审计发现 3 个实质问题,已随 6d81cfa83 代修:

  1. governor 在主路径未接线(AIMD 死代码):原实现只有 build_direct_workflow_tool 一处 stamp;交互会话 agent 工具链(core/engine.rs 根 runtime → spawn_background_*)整棵派生树 governor 恒 None,429 上报永不触发——而这条正是蜂群的主用例。现改为 manager 在 spawn_background_with_assignment_options 汇聚点为全部 spawn 路径统一 stamp(原注释声称的接线至此才成真),mod.rs 两处失实注释同步修正。
  2. 暂停后活性缺陷:解除 paused 需要「窗口排空后新到达的 success 事件」。若在飞集群在 60s 窗口排空前全部终结,排队子智能体没有任何唤醒机制,会冻结到各自 wall-time 超时(默认 30 分钟、蜂群场景可达 24h)。新增排队期时间驱动探针:窗口排空即恢复 1/4 容量,无需等待 success 事件。
  3. DynamicGate 丢唤醒竞态:原「唤醒后由 waiter 复查」设计下,waiter 在被唤醒与复查之间被取消会吞掉唤醒;若当时 active 已归零则槽位永久空转。改为 permit 经 oneshot 预先计数交接:被取消 waiter 的 permit 随 future drop,其 Drop 自动把槽位重派给下一个 waiter(tokio::sync::Semaphore 同款取消安全语义)。

另含:set_max_capacity 改为「未暂停时即应用配置容量(可升可降)、暂停时保持 0」——运行时改 launch 并发不再能静默解除暂停(原实现直接 gate.set_capacity 会绕过 pause 不变量),with_launch_concurrency/update_runtime_limits 统一走 governor;摘除 rate_limit_governor 上已过时的 #[allow(dead_code)];新增 3 条 forkguard_* 行为测试(窗口排空时间自愈、暂停在改限下保持容量 0、取消 waiter 的 grant 重派发)。

实测subagent:: 491 通过 0 失败、forkguard_ 60 通过 0 失败、fmt/clippy 干净。父仓已在同 PR 登记 T5 指纹与 fork-modifications 条目;合并后发布 pinvou-v0.9.5-r14 即可让父仓第 0 层转绿。

遗留披露:仓库退避实现至此有 4 套(subagent_transient_provider_retry_delay 无 jitter、subagent_api_timeout_retry_delay ±20%、llm_client::RetryConfig 区间 jitter、本 PR full jitter)——full jitter 语义不同值得保留,其余三套的收敛属跨层重构,建议留独立 PR。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant