docs: adversarial codebase audit (2026-07-03) — assertions, parser, benchmark-stats integrity

.github/workflows/ci.yml

@@ -24,4 +24,7 @@ jobs:
       - run: pnpm lint
       - run: pnpm typecheck
       - run: pnpm test
+      # Build the artifact npm actually ships and confirm the compiled bin runs;
+      # otherwise a build-only breakage merges green and only surfaces at release (F-37).
+      - run: pnpm build && node dist/cli.js --version
       - run: pnpm audit --audit-level=high --prod

.gitignore

@@ -5,7 +5,6 @@ node_modules/
 .DS_Store
 dist/
 coverage/
-examples/deepseek-support-agent/benchmark-results.md
 
 # doctier:begin — managed by 'doctier init', do not edit
 **/_prototype-*

CHANGELOG.md

@@ -4,6 +4,30 @@ All notable changes to this project will be documented in this file.
 
 The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
 
+## [Unreleased]
+
+### Added
+- `target_error` failure mode: transport failures (network error, non-2xx, malformed or non-JSON response, timeout) are now classified separately from the behavioral `no_tool_call` mode, so an agent outage is no longer counted as locale drift.
+- `--timeout MS` flag on `run` (default 30000); a hung locale is recorded as `target_error` instead of stalling the whole run.
+- `noToolCall` can forbid multiple tools via `anyOf: [a, b]`.
+- `translate` now emits `responseLanguage` for the target locale (when the source asserts one) and warns when the model returns fewer locales than requested.
+- `lint` flags duplicate scenario ids across a directory and `responseLanguage` values whose script cannot be determined (the check can never fail).
+- The CLI JSON report includes a 95% Wilson confidence interval per locale.
+
+### Changed
+- `responseLanguage`: `ja` now requires kana so pure-Chinese text no longer passes; the non-Latin detector covers every script in the table (including Georgian and Ethiopic); the measured in-script ratio is included in the `detail`.
+- Directory input always emits the matrix report shape, even for a single file.
+- Markdown run report includes a `Detail` column; the matrix highlights failing cells instead of passing ones and escapes `|` in cell values.
+- The `agent:` field is now sent in the POST body to the target as routing metadata.
+- Minimum Node version relaxed to `>=22`.
+
+### Fixed
+- Scalar argument assertions no longer let a non-scalar (array/object/null) pass via `String()` coercion.
+- Scenario parser: `oneOf` items with quoted commas parse correctly; duplicate locale keys, nameless tool-call list items, tab/odd indentation, and block scalars are rejected with line-numbered errors; an empty `noToolCall:` no longer absorbs a sibling's `name:`.
+- `translate` serializes `oneOf` matchers correctly instead of `[object Object]`.
+- `--allow-fail` warns when a value matches no locale; `-v`/`-h` no longer hijack a `run` invocation; the target URL is validated.
+- CI now builds the compiled artifact and runs it.
+
 ## [0.3.1] - 2026-06-25
 
 ### Fixed

README.md

@@ -113,6 +113,8 @@ locales:
       responseLanguage: fr
 ```
 
+Scenario files are a **strict 2-space-indented subset of YAML**, not full YAML. Use exactly two spaces per level (no tabs), and keep every value on one line — block scalars (`input: |`), flow mappings, and multi-line strings are not supported. Out-of-subset input is rejected with a line-numbered error rather than parsed loosely. Run `langdrift lint` to catch these and other issues early.
+
 Run it against your agent:
 
 ```bash
@@ -150,9 +152,25 @@ expect:
 
 `oneOf` is an inline list and must contain at least one value; `langdrift lint` reports an error otherwise.
 
+### Forbidden tools
+
+`noToolCall` fails the locale if the agent calls a tool it should not. Forbid one tool with `name`, or several with `anyOf`:
+
+```yaml
+expect:
+  toolCall:
+    name: create_refund_ticket
+  noToolCall:
+    anyOf: [escalate_to_human, contact_seller]
+```
+
 ### Response script
 
-`responseLanguage` is a **script-family check**, not language detection. It confirms a reply uses the script a locale is written in (for example, that an `ar` reply is in Arabic script). It cannot distinguish languages that share a script: a `fr` assertion passes for any Latin-script reply, and `ar` cannot be told apart from `fa` or `ur`. For a locale whose script LangDrift cannot determine, the check passes rather than guessing.
+`responseLanguage` is a **script-family check**, not language detection. It confirms a reply uses the script a locale is written in (for example, that an `ar` reply is in Arabic script). Know its limits before relying on it:
+
+- **It cannot distinguish languages that share a script.** A `fr` assertion passes for any Latin-script reply; `ar` cannot be told apart from `fa` or `ur`; `zh` accepts any Han text. The one Han exception is `ja`, which additionally requires kana, so pure-Chinese text does not pass `responseLanguage: ja`.
+- **The thresholds are asymmetric.** A non-Latin locale passes when at least 10% of letters are in its script; a Latin locale fails only when more than 50% of letters are non-Latin. The measured ratio is included in the failure/pass `detail` so near-misses are visible.
+- **For a locale whose script LangDrift cannot determine, the check passes** rather than guessing. `langdrift lint` warns when a `responseLanguage` value is not script-determinable, since the check can then never fail.
 
 ## HTTP Target Contract
 
@@ -164,10 +182,13 @@ Request:
 {
   "locale": "fr",
   "input": "J'ai été facturé deux fois. Pouvez-vous me rembourser un paiement?",
-  "scenarioId": "refund_request"
+  "scenarioId": "refund_request",
+  "agent": "support"
 }
 ```
 
+`agent` is the scenario's `agent:` field, sent as routing metadata; agents that serve one workflow can ignore it.
+
 Response:
 
 ```json
@@ -209,8 +230,11 @@ langdrift translate <scenario.yaml> [--locales fr,ar,zh,...] [--write]
 Useful CI flags:
 
 - `--min-pass-rate N`: fail only if the overall pass rate is below `N`.
-- `--allow-fail <locale>`: keep reporting a known weak locale without letting it fail the build.
+- `--allow-fail <locale>`: keep reporting a known weak locale without letting it fail the build. A value that matches no locale prints a warning.
 - `--format markdown`: write a table suitable for GitHub Actions summaries or PR comments.
+- `--timeout MS`: per-request timeout (default 30000). A hung locale is recorded as `target_error` instead of stalling the run.
+
+A transport failure (network error, non-2xx, malformed or non-JSON response, or timeout) is classified as `target_error`, distinct from the behavioral `no_tool_call` mode, so an agent outage is not mistaken for locale drift.
 
 See [docs/ci.md](docs/ci.md) for GitHub Actions examples.
 
@@ -246,7 +270,8 @@ langdrift run ./examples/scenarios/support-routing.yaml --target http://127.0.0.
 - **Behavior over text.** LangDrift checks tool calls and structured behavior, not whether a reply sounds fluent.
 - **Deterministic assertions first.** No LLM-as-judge in the core loop; failures are explainable and CI-friendly.
 - **HTTP contract over framework lock-in.** Any agent that can accept one POST request can be tested.
-- **Small, inspectable core.** Zero runtime dependencies, TypeScript source, Node >= 24.
+- **Small, inspectable core.** Zero runtime dependencies, TypeScript source, Node >= 22.
+- **CLI, not a library.** The published package exposes the `langdrift` command only; there is no importable JavaScript API. To reuse the internals, work from a clone of the TypeScript source.
 - **Demo without API keys.** The fake agent makes the failure mode visible locally before connecting a real model.
 
 ## More Context

RESEARCH.md

@@ -16,6 +16,8 @@ I built a minimal eval harness and ran scenarios across 3 domains and 12 locales
 - 10 iterations per scenario for each model
 - English baseline confirmed before any other locale was tested
 
+In the tables below, **Failing locale checks** lists *every* cell below 10/10, ordered worst first (fewest passes first); a cell not listed passed all 10 iterations. Per-cell 95% Wilson confidence intervals for every failing cell are in the [appendix](#appendix-per-cell-confidence-intervals).
+
 **Results — gpt-4o-mini (10 iterations × 12 locales):**
 
 | Scenario | Pass rate | Failing locale checks |
@@ -31,22 +33,22 @@ I built a minimal eval harness and ran scenarios across 3 domains and 12 locales
 
 | Scenario | Pass rate | Failing locale checks |
 | -------- | --------- | ------------------- |
-| support-routing | 59% (71/120) | mn (1/10), sw (3/10), yo (3/10), vi (4/10), cy (5/10), eu (5/10), zh (6/10) |
-| support-cancel-subscription | 68% (82/120) | yo (0/10), sw (1/10), mn (2/10), eu (4/10), cy (6/10) |
-| ecommerce-cancel-order | 69% (83/120) | yo (0/10), cy (1/10), eu (2/10), zh (6/10), mn (6/10) |
-| ecommerce-track-order | 46% (55/120) | mn (0/10), cy (1/10), eu (1/10), en (3/10), sw (4/10) |
-| scheduling-reschedule | 88% (106/120) | sw (4/10), eu (7/10), mn (8/10) |
-| scheduling-book-new | 40% (48/120) | id (0/10), sw (0/10), cy (0/10), fr (2/10), ar (4/10) |
+| support-routing | 59% (71/120) | mn (1/10), sw (3/10), yo (3/10), vi (4/10), cy (5/10), eu (5/10), zh (6/10), id (7/10), ru (8/10), ar (9/10) |
+| support-cancel-subscription | 68% (82/120) | yo (0/10), sw (1/10), mn (2/10), eu (4/10), cy (6/10), zh (9/10) |
+| ecommerce-cancel-order | 69% (83/120) | yo (0/10), cy (1/10), eu (2/10), zh (6/10), mn (6/10), vi (9/10), sw (9/10) |
+| ecommerce-track-order | 46% (55/120) | mn (0/10), cy (1/10), eu (1/10), zh (2/10), en (3/10), sw (4/10), id (5/10), fr (7/10), ar (8/10), ru (8/10), vi (8/10), yo (8/10) |
+| scheduling-reschedule | 88% (106/120) | sw (4/10), eu (7/10), mn (8/10), en (9/10), zh (9/10), cy (9/10) |
+| scheduling-book-new | 40% (48/120) | id (0/10), sw (0/10), cy (0/10), fr (2/10), ar (4/10), vi (4/10), eu (4/10), yo (4/10), zh (6/10), mn (7/10), ru (8/10), en (9/10) |
 
 **Results — DeepSeek deepseek-chat (10 iterations × 12 locales):**
 
 | Scenario | Pass rate | Failing locale checks |
 | -------- | --------- | ------------------- |
-| support-routing | 84% (101/120) | mn (1/10), cy (2/10), sw (9/10), zh (9/10) |
-| support-cancel-subscription | 62% (74/120) | zh (0/10), eu (0/10), sw (1/10), id (2/10), ar (3/10) |
-| ecommerce-cancel-order | 57% (69/120) | zh (0/10), ru (0/10), eu (0/10), yo (0/10), sw (3/10), en (8/10) |
-| ecommerce-track-order | 42% (50/120) | ar (0/10), zh (0/10), vi (0/10), sw (0/10), fr (1/10), yo (1/10), mn (3/10) |
-| scheduling-reschedule | 64% (77/120) | ar (0/10), zh (0/10), sw (0/10, wrong tool), id (1/10), mn (8/10) |
+| support-routing | 84% (101/120) | mn (1/10, wrong_tool), cy (2/10, wrong_tool), sw (9/10, wrong_tool), zh (9/10) |
+| support-cancel-subscription | 62% (74/120) | zh (0/10), eu (0/10), sw (1/10), id (2/10), ar (3/10), cy (9/10), yo (9/10) |
+| ecommerce-cancel-order | 57% (69/120) | zh (0/10), ru (0/10), eu (0/10), yo (0/10), sw (3/10), en (8/10), cy (9/10), mn (9/10) |
+| ecommerce-track-order | 42% (50/120) | ar (0/10), zh (0/10), vi (0/10), sw (0/10), fr (1/10), yo (1/10), mn (3/10), id (6/10), eu (9/10) |
+| scheduling-reschedule | 64% (77/120) | ar (0/10), zh (0/10), sw (0/10, wrong_tool), id (1/10), mn (8/10), eu (9/10), yo (9/10) |
 | scheduling-book-new | 93% (112/120) | eu (2/10) |
 
 English is not a perfect baseline on every model. It passes every scenario on gpt-4o-mini except the model-behavior divergence in `scheduling-book-new`, while claude-haiku misses `ecommerce-track-order` in 7/10 runs and DeepSeek misses `ecommerce-cancel-order` in 2/10 runs. That matters: LangDrift surfaces both locale drift and scenario/model reliability issues.
@@ -57,7 +59,7 @@ This is an applied experiment, not a scientific claim.
 
 **Three models, one architecture.** The benchmark now covers gpt-4o-mini, claude-haiku-4-5-20251001, and DeepSeek deepseek-chat via the same HTTP agent wrapper with the same system prompt and tool set. Cross-model patterns (Basque, Yoruba, low-resource language clusters) are therefore more credible than when a single model was used. However, the agent architecture is still simple: single-turn, 5 tools per domain, no RAG, no multi-turn context. More complex setups may show different failure patterns.
 
-**Small sample, reported with uncertainty.** Each scenario/model/locale cell uses 10 iterations. The agent runs at `temperature 0`, so these iterations are near-deterministic: they capture API-side variance, not a sampling distribution. N=10 is enough to expose repeated failure patterns but is not a large-sample statistical benchmark, and a single 7/10-vs-9/10 difference is well within noise. The benchmark report now prints a 95% Wilson confidence interval per locale, and per-cell pass rates throughout this document should be read as estimates with that uncertainty, not exact rankings.
+**Small sample, reported with uncertainty.** Each scenario/model/locale cell uses 10 iterations. The agent runs at `temperature 0`, so these iterations are near-deterministic: they capture API-side variance, not a sampling distribution. N=10 is enough to expose repeated failure patterns but is not a large-sample statistical benchmark, and a single 7/10-vs-9/10 difference is well within noise. The [appendix](#appendix-per-cell-confidence-intervals) gives a 95% Wilson confidence interval for every failing cell, computed directly from the committed pass counts (the CLI's own `--format json` report also emits these intervals per locale). Per-cell pass rates throughout this document should be read as estimates with that uncertainty, not exact rankings.
 
 **Unreviewed locale prompts.** The locale inputs were written by one author to preserve intent but were not reviewed by native speakers. Some failures may reflect phrasing gaps rather than model behavior. This is acknowledged as a real threat to validity, but native review at scale is not practical for a solo project. Results should be interpreted with that caveat explicitly in mind.
 
@@ -144,3 +146,128 @@ LangDrift is the harness I used for this experiment, cleaned up and made general
 - Exits non-zero on failure, so it works in CI
 
 The goal is to let any team run localized behavior checks against their own agents, not just refund routing, but any workflow where the right behavior matters across languages.
+
+## Appendix: per-cell confidence intervals
+
+95% Wilson score intervals for every failing cell (any cell below 10/10), computed from the committed pass counts in `examples/benchmark/results/`. Cells not listed passed all 10 iterations; a 10/10 cell has the interval [0.72, 1.00] at N=10. These are presentation over the same committed data, not a rerun.
+
+### gpt-4o-mini
+
+| Scenario | Locale | Pass | 95% CI |
+| -------- | ------ | ---- | ------ |
+| ecommerce-cancel-order | eu | 0/10 | [0.00, 0.28] |
+| ecommerce-track-order | — | — | — |
+| scheduling-book-new | en | 0/10 | [0.00, 0.28] |
+|  | fr | 0/10 | [0.00, 0.28] |
+|  | ar | 0/10 | [0.00, 0.28] |
+|  | zh | 0/10 | [0.00, 0.28] |
+|  | ru | 0/10 | [0.00, 0.28] |
+|  | id | 0/10 | [0.00, 0.28] |
+|  | vi | 0/10 | [0.00, 0.28] |
+|  | sw | 0/10 | [0.00, 0.28] |
+|  | cy | 0/10 | [0.00, 0.28] |
+|  | eu | 0/10 | [0.00, 0.28] |
+|  | mn | 0/10 | [0.00, 0.28] |
+|  | yo | 0/10 | [0.00, 0.28] |
+| scheduling-reschedule | eu | 0/10 | [0.00, 0.28] |
+| support-cancel-subscription | — | — | — |
+| support-routing | — | — | — |
+
+### claude-haiku-4-5-20251001
+
+| Scenario | Locale | Pass | 95% CI |
+| -------- | ------ | ---- | ------ |
+| ecommerce-cancel-order | yo | 0/10 | [0.00, 0.28] |
+|  | cy | 1/10 | [0.02, 0.40] |
+|  | eu | 2/10 | [0.06, 0.51] |
+|  | zh | 6/10 | [0.31, 0.83] |
+|  | mn | 6/10 | [0.31, 0.83] |
+|  | vi | 9/10 | [0.60, 0.98] |
+|  | sw | 9/10 | [0.60, 0.98] |
+| ecommerce-track-order | mn | 0/10 | [0.00, 0.28] |
+|  | cy | 1/10 | [0.02, 0.40] |
+|  | eu | 1/10 | [0.02, 0.40] |
+|  | zh | 2/10 | [0.06, 0.51] |
+|  | en | 3/10 | [0.11, 0.60] |
+|  | sw | 4/10 | [0.17, 0.69] |
+|  | id | 5/10 | [0.24, 0.76] |
+|  | fr | 7/10 | [0.40, 0.89] |
+|  | ar | 8/10 | [0.49, 0.94] |
+|  | ru | 8/10 | [0.49, 0.94] |
+|  | vi | 8/10 | [0.49, 0.94] |
+|  | yo | 8/10 | [0.49, 0.94] |
+| scheduling-book-new | id | 0/10 | [0.00, 0.28] |
+|  | sw | 0/10 | [0.00, 0.28] |
+|  | cy | 0/10 | [0.00, 0.28] |
+|  | fr | 2/10 | [0.06, 0.51] |
+|  | ar | 4/10 | [0.17, 0.69] |
+|  | vi | 4/10 | [0.17, 0.69] |
+|  | eu | 4/10 | [0.17, 0.69] |
+|  | yo | 4/10 | [0.17, 0.69] |
+|  | zh | 6/10 | [0.31, 0.83] |
+|  | mn | 7/10 | [0.40, 0.89] |
+|  | ru | 8/10 | [0.49, 0.94] |
+|  | en | 9/10 | [0.60, 0.98] |
+| scheduling-reschedule | sw | 4/10 | [0.17, 0.69] |
+|  | eu | 7/10 | [0.40, 0.89] |
+|  | mn | 8/10 | [0.49, 0.94] |
+|  | en | 9/10 | [0.60, 0.98] |
+|  | zh | 9/10 | [0.60, 0.98] |
+|  | cy | 9/10 | [0.60, 0.98] |
+| support-cancel-subscription | yo | 0/10 | [0.00, 0.28] |
+|  | sw | 1/10 | [0.02, 0.40] |
+|  | mn | 2/10 | [0.06, 0.51] |
+|  | eu | 4/10 | [0.17, 0.69] |
+|  | cy | 6/10 | [0.31, 0.83] |
+|  | zh | 9/10 | [0.60, 0.98] |
+| support-routing | mn | 1/10 | [0.02, 0.40] |
+|  | sw | 3/10 | [0.11, 0.60] |
+|  | yo | 3/10 | [0.11, 0.60] |
+|  | vi | 4/10 | [0.17, 0.69] |
+|  | cy | 5/10 | [0.24, 0.76] |
+|  | eu | 5/10 | [0.24, 0.76] |
+|  | zh | 6/10 | [0.31, 0.83] |
+|  | id | 7/10 | [0.40, 0.89] |
+|  | ru | 8/10 | [0.49, 0.94] |
+|  | ar | 9/10 | [0.60, 0.98] |
+
+### DeepSeek deepseek-chat
+
+| Scenario | Locale | Pass | 95% CI |
+| -------- | ------ | ---- | ------ |
+| ecommerce-cancel-order | zh | 0/10 | [0.00, 0.28] |
+|  | ru | 0/10 | [0.00, 0.28] |
+|  | eu | 0/10 | [0.00, 0.28] |
+|  | yo | 0/10 | [0.00, 0.28] |
+|  | sw | 3/10 | [0.11, 0.60] |
+|  | en | 8/10 | [0.49, 0.94] |
+|  | cy | 9/10 | [0.60, 0.98] |
+|  | mn | 9/10 | [0.60, 0.98] |
+| ecommerce-track-order | ar | 0/10 | [0.00, 0.28] |
+|  | zh | 0/10 | [0.00, 0.28] |
+|  | vi | 0/10 | [0.00, 0.28] |
+|  | sw | 0/10 | [0.00, 0.28] |
+|  | fr | 1/10 | [0.02, 0.40] |
+|  | yo | 1/10 | [0.02, 0.40] |
+|  | mn | 3/10 | [0.11, 0.60] |
+|  | id | 6/10 | [0.31, 0.83] |
+|  | eu | 9/10 | [0.60, 0.98] |
+| scheduling-book-new | eu | 2/10 | [0.06, 0.51] |
+| scheduling-reschedule | ar | 0/10 | [0.00, 0.28] |
+|  | zh | 0/10 | [0.00, 0.28] |
+|  | sw | 0/10 | [0.00, 0.28] |
+|  | id | 1/10 | [0.02, 0.40] |
+|  | mn | 8/10 | [0.49, 0.94] |
+|  | eu | 9/10 | [0.60, 0.98] |
+|  | yo | 9/10 | [0.60, 0.98] |
+| support-cancel-subscription | zh | 0/10 | [0.00, 0.28] |
+|  | eu | 0/10 | [0.00, 0.28] |
+|  | sw | 1/10 | [0.02, 0.40] |
+|  | id | 2/10 | [0.06, 0.51] |
+|  | ar | 3/10 | [0.11, 0.60] |
+|  | cy | 9/10 | [0.60, 0.98] |
+|  | yo | 9/10 | [0.60, 0.98] |
+| support-routing | mn | 1/10 | [0.02, 0.40] |
+|  | cy | 2/10 | [0.06, 0.51] |
+|  | sw | 9/10 | [0.60, 0.98] |
+|  | zh | 9/10 | [0.60, 0.98] |

docs/ci.md

@@ -75,8 +75,11 @@ jobs:
         env:
           OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
 
+      # Wait on the TCP port, not the POST endpoint: the agent route only
+      # answers POST, so an HTTP GET probe against it 404s and times out even
+      # when the agent is up. Use tcp: (or a GET health route if your agent has one).
       - name: Wait for agent
-        run: npx wait-on http://127.0.0.1:3010/api/agent --timeout 15000
+        run: npx wait-on tcp:127.0.0.1:3010 --timeout 15000
 
       - name: Run locale eval
         run: |

docs/integrations.md

@@ -171,4 +171,4 @@ app.listen({ port: 3010 });
 
 **`scenarioId` is informational.** LangDrift sends it so your agent can log or branch on it. Most adapters can ignore it.
 
-**Non-2xx responses fail the locale.** If your handler throws or returns a non-2xx status, LangDrift records it as `no_tool_call` with the HTTP status as the detail.
+**Non-2xx responses fail the locale.** If your handler throws or returns a non-2xx status, LangDrift records it as `target_error` (a transport failure, kept distinct from behavioral modes like `no_tool_call`) with the HTTP status as the detail. A malformed or non-JSON 2xx body is also a `target_error`.