Why
Everything in src/core/ is tested against canned ConverseFn responses — 175
tests, no network, no credentials, no billable inference. That covers the loop, the
dispatch, the failure handling and the gap reporting. What it cannot cover is
whether a real model, given this system prompt and this toolset, honours the
grounding contract in practice.
That's a prompt-quality check, not a regression test, and it needs a human to read
the answer. Procedure is written up in docs/live-bedrock.md.
What to check, ranked by what actually goes wrong
- The uncatalogued case — ask about a code that is NOT one of the six KB
entries (LAMMPS, NAMD, CP2K). The premise must be stated before the
recommendation, with an invitation to correct it. A fluent answer with no stated
premise is a failure even when the hardware advice happens to be good.
- Every instance name and price in the prose appears in
answer.calls. Anything
in the text but not in a tool result means the model answered from memory —
the failure this library exists to prevent, and the most convincing-looking one.
recommend_shape was called first, before find_instances.- Gaps are voiced, not merely recorded: run with
live and capacity absent
and confirm the answer says quota and spot were not checked.
- No implication that it can launch.
Constraints
- Manual and opt-in behind an explicit env var. Never in CI — Bedrock inference
is billable, matching spawn-ts's gated live-smoke pattern.
- Fix findings in
src/core/prompt.ts and add the assertion to
prompt.test.ts in the same change. Those tests look like testing prose because
the failure mode is silent: drop the "label your inference" paragraph and every
answer still looks fine, just with the premise gone.
- Do not respond to a misbehaving model by loosening the tools (accepting a
model-supplied price, coercing "8" → 8). A tool that tolerates a made-up
input teaches the model that made-up inputs work.
Why
Everything in
src/core/is tested against cannedConverseFnresponses — 175tests, no network, no credentials, no billable inference. That covers the loop, the
dispatch, the failure handling and the gap reporting. What it cannot cover is
whether a real model, given this system prompt and this toolset, honours the
grounding contract in practice.
That's a prompt-quality check, not a regression test, and it needs a human to read
the answer. Procedure is written up in
docs/live-bedrock.md.What to check, ranked by what actually goes wrong
entries (LAMMPS, NAMD, CP2K). The premise must be stated before the
recommendation, with an invitation to correct it. A fluent answer with no stated
premise is a failure even when the hardware advice happens to be good.
answer.calls. Anythingin the text but not in a tool result means the model answered from memory —
the failure this library exists to prevent, and the most convincing-looking one.
recommend_shapewas called first, beforefind_instances.liveandcapacityabsentand confirm the answer says quota and spot were not checked.
Constraints
is billable, matching spawn-ts's gated
live-smokepattern.src/core/prompt.tsand add the assertion toprompt.test.tsin the same change. Those tests look like testing prose becausethe failure mode is silent: drop the "label your inference" paragraph and every
answer still looks fine, just with the premise gone.
model-supplied price, coercing
"8"→8). A tool that tolerates a made-upinput teaches the model that made-up inputs work.