U8: k6 load tests update by jaturapornchai · Pull Request #12 · jaturapornchai/bcproxyai

jaturapornchai · 2026-04-09T21:44:15Z

Summary

Add aggressive thresholds to smoke/chat/stress: p(95)<5s, p(99)<15s (<10s for stress), http_req_failed rate<0.02.
Rotate 4 diverse prompt categories per VU iteration (thai/code/general, small/medium) via shared loadtest/_categories.js helper; each request tagged with category for per-category metric breakdown.
Smoke now also hits /v1/chat/completions (5 VUs / 30s). Chat ramps 0->20->0 VUs. Stress ramps 0->50->100->0 VUs with a per-request chat_req_duration Trend retained.

Test plan

rtk npx next build -> Errors: 0 | Warnings: 0
k6 run loadtest/smoke.js against a running bcproxyai (BASE_URL=http://localhost:3334)
k6 run loadtest/chat.js -> thresholds pass under normal load
k6 run loadtest/stress.js -> observe aggressive-mode latency under 100 VUs

Note: pre-existing vitest failures in scanner.test.ts (mock config) are unrelated to this JS-only loadtest change.

Generated with Claude Code

Add p95<5s / p99<15s (10s for stress) thresholds and <2% error rate across smoke/chat/stress. Each script now rotates through 4 diverse prompt categories (thai/code/general, small/medium) via a shared _categories.js helper, tagged per request for per-category metrics. Smoke exercises the chat endpoint in addition to homepage/status; chat ramps to 20 VUs; stress caps at 100 VUs (previously 500) to surface aggressive-mode latency without swamping the host. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Copilot

Pull request overview

Updates the k6 load-testing suite for BCProxyAI to add stricter performance/error thresholds, broaden coverage (smoke now exercises chat completions), and introduce per-request prompt “category” tagging to segment metrics by prompt type/size.

Changes:

Tightened k6 thresholds across smoke/chat/stress (p95/p99 latency + http_req_failed rate) and adjusted VU ramp profiles.
Added shared pickCategory() helper and tagged each chat request with a category for per-category metric breakdown.
Smoke test now includes a /v1/chat/completions request in addition to basic health endpoints.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 5 comments.

File	Description
loadtest/_categories.js	Adds shared prompt categories and a helper to pick one per request.
loadtest/smoke.js	Increases load, adds thresholds, and adds a chat completion request with category tagging.
loadtest/chat.js	Adjusts ramp profile and thresholds; uses shared categories and tags requests.
loadtest/stress.js	Changes ramp profile, adds strict thresholds, and switches to shared categories + request tags while retaining Trend.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-04-09T21:48:15Z

+  vus: 5,
+  duration: '30s',
  thresholds: {
-    http_req_failed: ['rate<0.01'],
+    'http_req_duration{expected_response:true}': ['p(95)<5000', 'p(99)<15000'],
+    'http_req_failed': ['rate<0.02'],


With the server-side rate limit (100 req/60s per IP on /v1/chat/completions), running 5 VUs for 30s will very likely produce a large number of 429s, causing the new http_req_failed rate<0.02 threshold to fail consistently. Consider lowering VUs/iteration rate to stay under the limiter, or explicitly treating 429 as an expected status in k6 (e.g., via a responseCallback/expectedStatuses) or by varying X-Real-IP/X-Forwarded-For per VU if the goal is to measure backend latency rather than the limiter.

Copilot · 2026-04-09T21:48:16Z

+  );
+  check(res, {
+    'chat status 200': (r) => r.status === 200,
+    'chat body has content': (r) => r.body && r.body.includes('content'),


chat body has content uses a string substring check on the raw response body, which can produce false positives (e.g., error payloads that include the word "content") and false negatives (format changes). It’s more reliable to parse JSON (e.g., res.json() / selector) and assert the expected shape like choices[0].message.content (with a guard for JSON parse errors).

Suggested change

'chat body has content': (r) => r.body && r.body.includes('content'),

'chat body has content': (r) => {

if (!r.body) {

return false;

}

try {

const body = JSON.parse(r.body);

return Array.isArray(body.choices)

&& body.choices.length > 0

&& body.choices[0]

&& body.choices[0].message

&& typeof body.choices[0].message.content === 'string'

&& body.choices[0].message.content.length > 0;

} catch (_) {

return false;

}

},

Copilot · 2026-04-09T21:48:16Z

  stages: [
-    { duration: '30s', target: 10 },
-    { duration: '1m', target: 10 },
-    { duration: '30s', target: 30 },
-    { duration: '1m', target: 30 },
+    { duration: '30s', target: 20 },
+    { duration: '1m', target: 20 },
    { duration: '30s', target: 0 },
  ],
  thresholds: {
-    http_req_duration: ['p(95)<10000'],
-    http_req_failed: ['rate<0.15'],
+    'http_req_duration{expected_response:true}': ['p(95)<5000', 'p(99)<15000'],
+    'http_req_failed': ['rate<0.02'],
  },


Given /v1/chat/completions is rate-limited to 100 req/60s per IP, the stage (20 VUs with ~1–3s think time) will exceed the limiter and generate many 429s. That will violate http_req_failed rate<0.02 (since 429 counts as failed by default), so the test is likely to fail in normal local/staging runs even though the script backs off on 429. Consider either (a) lowering the request rate below the limiter, or (b) marking 429 as an expected status in k6 so it doesn’t count toward http_req_failed, depending on what this test is intended to validate.

Copilot · 2026-04-09T21:48:16Z

+  thresholds: {
+    'http_req_duration{expected_response:true}': ['p(95)<5000', 'p(99)<10000'],
+    'http_req_failed': ['rate<0.02'],
+  },


The new http_req_failed rate<0.02 threshold will likely fail quickly under stress conditions because /v1/chat/completions is rate-limited (429) and 429s are counted as failures by default in k6. If the goal is to stress the backend (not the limiter), consider varying X-Real-IP/X-Forwarded-For per VU or adjusting expected statuses/thresholds so rate limiting doesn’t dominate the failure signal.

Copilot · 2026-04-09T21:48:16Z

+export function pickCategory() {
+  return categories[Math.floor(Math.random() * categories.length)];


The PR description says categories are “rotated” per VU iteration, but pickCategory() currently selects randomly, which can skew per-category breakdowns and make runs less reproducible. If rotation is required, consider a deterministic selection (e.g., based on __ITER/__VU) to cycle through categories evenly.

/guide Hermes snippet was based on old CLI (hermes model add …, config.toml) that doesn't match today's Hermes v0.10.0. Replaced with the exact commands verified on a fresh Ubuntu 24.04 (WSL2) install — including the provider=custom trick that redirects Hermes away from openrouter.ai. Also bumps README: - Smart Routing: 13 → 15 steps. Added #12 Mistral message-order auto-patch + #13 Client-400 short-circuit (both shipped today as fixes for Hermes/OpenClaw compatibility). - Feature table: new 🤖 Agent-ready row. - Integration: new "Hermes Agent (Nous Research)" subsection with install + config + one-liner demo. Verified in production: 13/16 built-in Hermes tools pass cleanly through SMLGateway (terminal, file RW, code_execution, todo, vision, tts, clarify, session_search, skills, delegation, moa). 3 require extra config (memory-persist must be explicit, cronjob needs gateway service, delegation subagent self-corrects unknown tool names — all non-gateway issues). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Copilot AI review requested due to automatic review settings April 9, 2026 21:44

Copilot started reviewing on behalf of jaturapornchai April 9, 2026 21:45 View session

Copilot AI reviewed Apr 9, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

U8: k6 load tests update#12

U8: k6 load tests update#12
jaturapornchai wants to merge 1 commit into
mainfrom
worktree-agent-a61ee34f

jaturapornchai commented Apr 9, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Apr 9, 2026

Uh oh!

Copilot AI Apr 9, 2026

Uh oh!

Copilot AI Apr 9, 2026

Uh oh!

Copilot AI Apr 9, 2026

Uh oh!

Copilot AI Apr 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

-    'chat body has content': (r) => r.body && r.body.includes('content'),
+    'chat body has content': (r) => {
+      if (!r.body) {
+        return false;
+      }
+      try {
+        const body = JSON.parse(r.body);
+        return Array.isArray(body.choices)
+          && body.choices.length > 0
+          && body.choices[0]
+          && body.choices[0].message
+          && typeof body.choices[0].message.content === 'string'
+          && body.choices[0].message.content.length > 0;
+      } catch (_) {
+        return false;
+      }
+    },

		export function pickCategory() {
		return categories[Math.floor(Math.random() * categories.length)];

Conversation

jaturapornchai commented Apr 9, 2026

Summary

Test plan

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Apr 9, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 9, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 9, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 9, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 9, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants