feat(recognition): add a shelf-scanner bench and retune the Qwen adapter - #43
feat(recognition): add a shelf-scanner bench and retune the Qwen adapter#43arenier wants to merge 9 commits into
Conversation
Step 6 of #10: a manual bench that pits the two ShelfScannerPort adapters (Gemini, Qwen/OpenRouter) against each other on real shelf photos, to pick the production default — the phase-3 instrumentation ADR 0005 calls for. - libs/shared/text-match: reusable string normalization + fuzzy equality (Levenshtein ratio, threshold 0.85). A shared primitive: the bench scores a detection against the ground truth with it, and bibliographic reconciliation will match reads against the reference the same way. - tools/bench: dedicated Nx project (type:app / scope:api, so it may know infrastructure and wire the real adapters). Pure, CI-tested logic — ground-truth YAML loading (zod), scoring (recall, precision, per-field accuracy, structuring errors, high-confidence hallucination), Markdown rendering — and a live runner that instruments fetch for latency and token usage. The runner is never in CI: it costs a paid call per photo. - Add tools/* to the Yarn workspaces so the project is linked and keeps its Nx tags (hence the module boundaries). - Docs: bench protocol (tools/bench/README.md), ground-truth template, and a lower-level decision note (docs/decisions/0001), updated from a live run. Live run (2026-09-02): Gemini reads end to end (9/10 photos, one transient 503, ~0.27c/scan, ~20s median). Qwen is unreachable from this environment — its network policy blocks openrouter.ai (403 at the proxy); the adapter correctly surfaces that as ShelfScanFailed. Quality metrics (recall/precision/hallucination) stay pending: they need a human-verified ground truth, which by design a VLM draft cannot stand in for. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PGqVYV5GLC8qBTSKqAq19k
OpenRouter is reachable now that its domain is on the environment egress. Both providers ran end to end: near-equal cost (~0.25c vs ~0.23c/scan), Qwen ~7x faster, but Qwen is unstable under the shared json_object prompt — 5 of 10 photos return 0 books, one returns 158, one fails on truncated JSON — while Gemini stays regular via native schema-constrained decoding. An operational signal, not the quality verdict: that still waits on the human-verified ground truth. Also document that Node's fetch needs NODE_USE_ENV_PROXY=1 to honor HTTPS_PROXY in a proxied environment, otherwise calls 403 on the egress allowlist. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PGqVYV5GLC8qBTSKqAq19k
Wraps `nx build bench && node tools/bench/dist/main.js` behind `yarn bench`, documented in tools/bench/README.md and the CLAUDE.md command list. Env vars go in front, e.g. `BENCH_PROVIDERS=qwen yarn bench`. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PGqVYV5GLC8qBTSKqAq19k
…SON (#10) The Qwen adapter asked for `response_format: json_object` — free-form JSON. On dense shelves that let the model truncate its answer mid-structure and emit empty author/title fields, failing whole scans (bench run on #10: 0 books on half the photos for the 235B, one 158-book reply, truncated JSON). Switch to `response_format: json_schema` carrying `SHELF_SCAN_JSON_SCHEMA` — the same contract Gemini already decodes against, now held over Qwen's grammar. All three candidate Qwen-VL models advertise structured-output support on OpenRouter. `strict` narrows the shape, not the meaning: the answer is still validated downstream by the shared response mapper, so an off-contract value still fails closed rather than leaking a bad DetectedBook. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PGqVYV5GLC8qBTSKqAq19k
…tokens (#10) The bench default 235B was unusable on dense shelves — 0 books on half the photos, truncated JSON on others — and json_schema alone did not rescue it. The 72B (the OCR-focused VL line) stays stable, so it becomes the Qwen default in the adapter and the bench. Also send a generous max_tokens so a legitimate long list is not cut off mid-JSON; a runaway repetition still hits the cap and fails, which is the outcome we want. This does not fix the deeper issue that all providers under-detect on these dense ressourcerie shelves — that is the finding logged on #10, and needs the human-verified ground truth to quantify. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PGqVYV5GLC8qBTSKqAq19k
|
Review · 🔴 À ne pas merger
La PR livre l'étape 6 de #10 : le bench qualité qui départage les deux adapters CI rouge sur 🔴 Bloquants
🟠 À corriger
💬 Questions
✍️ Style & altitudeRien à signaler. Les commentaires portent systématiquement un pourquoi (glouton plutôt qu'assignation optimale, zéro plutôt que Relecture statique : lecture de code + CI (read-only). Ni lint, ni test, ni build lancés — un 🟢 veut dire « rien trouvé en lecture », pas « ça compile ». Generated by Claude Code |
… pure logic Review triage on #43. - max-lines-per-function (blocking, CI red): the json_schema + max_tokens block pushed QwenShelfScannerAdapter.post past 50 lines and a spec describe past 50. Extract requestBody() from post, split the request-shape describe. oxlint green. - Untested pure logic: the bench vitest include was src/lib/** only, leaving the cost/usage parsing (usage.ts) and provider selection (config.ts) — which feed the decision-note numbers — outside CI. Widen the include to src/**, export readUsage, and add specs for estimateCost, readUsage (both payload shapes), mediaTypeOf, env/requireEnv and selectedProviders (unknown provider throws). The live runner still has no spec, so nothing paid runs in CI. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PGqVYV5GLC8qBTSKqAq19k
|
Triage de la review du 02/09. Les deux points ont été vérifiés dans le code (le 🔴 reproduit par
Vérifications point par point (2)1.
|
Follow-up on the bench findings (#10). - Prompt: the shared scan prompt now pushes exhaustiveness (scan the whole shelf band by band, a short list means stopping too early). On a throwaway probe this roughly tripled Qwen's reads on a sparse shelf (7 -> 19) and quadrupled them on a dense one (10 -> 42). To keep that recall without breaking the contract, it also forbids empty author/title (the domain rejects the whole payload on an empty field) and asks for each book once. - Scoring: dedupe detections on the normalized (author, title) before matching, so a spine the model lists twice is not double-counted nor charged as a false positive. Keyed on shared-text-match normalization; the highest-confidence copy survives. The prompt is shared by both adapters on purpose (the bench measures models, not prompts); its quality is validated by the manual bench, not a unit test (ADR 0005). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PGqVYV5GLC8qBTSKqAq19k
Rename the 10 reference-shelf photos from their camera basenames (20260801_HHMMSS.jpg) to shelf-fixture-1..10.jpg, on the GCS bucket and in every tracked reference: the ground-truth template, the config spec example, and the Gemini recorded-response provenance note. The bench discovers photos dynamically, so no runner code changes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PGqVYV5GLC8qBTSKqAq19k
shelf-fixture-2 framed the same shelf as shelf-fixture-1, only less wide, so it added no coverage to the reference set. Removed from the GCS bucket and from the ground-truth template. Remaining fixtures keep their numbers (1, 3..10) so existing identifiers stay stable. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PGqVYV5GLC8qBTSKqAq19k
Livre l'étape 6 de #10 — le bench qualité qui départage les deux adapters
ShelfScannerPortsur photos réelles — et retouche l'adapter Qwen + le prompt de scan à la lumière de ce que le
bench a révélé (les étapes 1-5 et 7 sont déjà sur
main, PR #39). C'est l'instrumentation dephase 3 prévue par l'ADR 0005.
Contexte
Le contexte
recognitiona ses deux adapters (Gemini, Qwen/OpenRouter) derrière le port, l'endpointPOST /scanet les tests sur réponses enregistrées. Ce qui manquait : de quoi choisir lequel desdeux devient le défaut, mesuré sur de vraies étagères plutôt qu'au jugé. Le bench a servi tout de
suite : il a montré l'adapter Qwen instable et le prompt trop timide, d'où leurs retouches ici.
Modifications
Regroupées par intention :
libs/shared/text-matchtools/bench(logique pure)tools/bench(runner live)fetch(latence, tokens), appelle les deux adapters, écrit le tableau. Jamais en CI — un appel payant par photo.libs/recognition/infrastructure(prod)qwen3-vl-235b→qwen2.5-vl-72b;json_object→json_schema;max_tokens. Et le prompt de scan partagé rendu exhaustif (balayage systématique de toute l'étagère) — le modèle s'arrêtait trop tôt. Le prompt interdit aussi les champs vides (le domaine rejette tout le payload sur un champ vide) et les doublons.package.json(racine)tools/*ajouté au glob des workspaces Yarn. Raccourciyarn bench.tools/bench/README.md), modèle de vérité terrain, note de décisiondocs/decisions/0001, carte d'architectureCLAUDE.md.Le bench est un projet Nx dédié tagué
type:app/scope:api. La logique pure reste sans provider.Tests
yarn check(lint + test + build, 8 projets) vert. Notamment :Le prompt est partagé par les deux adapters à dessein (le bench mesure les modèles, pas les
prompts) ; sa qualité se valide par le bench manuel, pas par un test unitaire (ADR 0005). Le runner
live n'a pas de test automatisé (un appel payant par photo) ; exécuté à la main sur le jeu de
référence — voir Points d'attention.
ADR concernés
bench est son instrumentation de phase 3. Le choix de modèle, de format et le prompt sont des
décisions de niveau inférieur, réversibles par configuration, hors ADR.
type:appconnaîtinfrastructure, d'où le tag du bench.Aucune décision structurante sans ADR ; le rejet en bloc d'un payload à champ vide (ADR 0005) est
conservé — c'est le prompt qui empêche les champs vides, pas le validateur qui s'assouplit.
Points d'attention
jetable (même modèle/prompt), Qwen 72b passe de 7 → 19 livres sur une étagère clairsemée et de
10 → 42 sur une dense — le modèle s'arrêtait trop tôt, pas un problème d'image ni de tokens
(complétion loin du plafond). Le prompt agressif rouvrait deux défauts, tous deux traités ici :
champs vides (le prompt les interdit, sinon le domaine perd toute la photo) et doublons
(dédupliqués dans le scoring). Reste à confirmer contre la vérité terrain.
main. Tant qu'elle n'est pas saisie,
SHELF_SCANNER_PROVIDERreste àstub.tools/bench/tsconfig.lib.jsonépinglelib: es2023(pourtoSorted) — outil Node-26 only.en premier :
tools/bench/src/lib/scoring.tset ses tests.Hors scope
SHELF_SCANNER_PROVIDER(après vérité terrain).Avance #10 (étape 6) — ne clôt pas l'issue : la sélection du fournisseur et la bascule du défaut
restent en attente de la vérité terrain vérifiée à la main.
🤖 Generated with Claude Code