Skip to content

Canary OCR concurrency 2/3/4 with StepFun 5-request cap awareness #2673

Description

@acoliver

Problem

OCR is currently pinned to concurrency 2 (down from the upstream default of 8). This was set conservatively, but the measured cost is significant:

Run Concurrency Files reviewed OCR step time
actions/runs/29921065524 8 (default) 62 13:44
actions/runs/29923648413 2 (explicit) 62 49:59
actions/runs/29946076923 2 (explicit) 63 46:31

The 3.6x slowdown correlates with the 4x concurrency reduction.

StepFun constraint

Our LLM provider (StepFun) caps concurrent API requests at 5 per account. This is a per-account limit, not per-session — multiple simultaneous OCR runs across different PRs share the same budget.

OCR's --concurrency flag controls file-level parallelism (number of files reviewed simultaneously). Each concurrent file review makes independent LLM API calls. OCR also has auxiliary LLM calls (planning, fact-checking, summarization) that add to the concurrent load.

Concurrency budget analysis

OCR concurrency 1 PR active 2 PRs active 3 PRs active
2 ~2-3 requests (safe) ~4-6 (borderline) ~6-9 (exceeds)
3 ~3-4 (safe) ~6-8 (exceeds) ~9-12 (exceeds)
4 ~4-5 (borderline) ~8-10 (exceeds)
5 ~5+ (at limit)

Key insight

StepFun does not increase its concurrency limit because there are multiple PRs. Two simultaneous OCR runs at concurrency 3 each would attempt 6 concurrent requests against a limit of 5.

Proposed experiment

Controlled concurrency canary with StepFun awareness:

Experiment design

  1. Pick a single PR with a known diff size (~60 files, similar to the reference runs).
  2. Lock OCR version, model, rules, and base ref.
  3. Run OCR at concurrency 2, 3, and 4 on the same PR head (via manual /ocr trigger to avoid interference).
  4. For each run, record:
    • Wall-clock OCR step time.
    • Total tokens consumed.
    • Number of 429/rate-limit responses (if any).
    • Number of retry/backoff events.
    • Finding count and category distribution (to verify quality is not affected).
  5. Between runs, verify no other OCR runs are active on other PRs.

Decision criteria

  • If concurrency 3 completes without 429s and shows >30% speedup over concurrency 2: adopt 3 as the new default.
  • If concurrency 3 triggers 429s: keep 2 and investigate whether StepFun's limit can be raised.
  • If concurrency 4 is significantly faster than 3 without 429s: consider 4 for single-PR scenarios, with a guard that reduces to 2 when concurrent runs are detected.

Future: dynamic concurrency

The ideal solution is dynamic: detect the current StepFun load (or number of concurrent OCR runs) and adjust --concurrency accordingly. This is not possible with GitHub Actions today, but once the auto-review limit (#2666) reduces simultaneous runs, concurrency 3 becomes safer because fewer PRs will trigger OCR simultaneously.

Acceptance criteria

  • Concurrency 2/3/4 canary runs are executed on the same PR head with fixed config.
  • Each run records time, tokens, 429 count, retries, and finding distribution.
  • A recommendation is documented with evidence.
  • If concurrency is changed, it is changed in isolation (no other OCR config changes in the same experiment).

Relationship to other issues

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions