Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 6 additions & 6 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ When performing tasks in this repository, always do the following:
- Read the nearest subdirectory `AGENTS.md` if present.
- Inspect the relevant README, package configuration, and existing tests before changing behavior.
- Prefer the smallest change that satisfies the task.
- Do not run expensive, provider-backed, or full benchmark commands without explicit approval.
- Obtain explicit approval for external model calls and potentially expensive evaluation runs.
- Keep changes small, public-safe, and reviewable.
- Preserve Quantiles as a local-first, offline system by default.
- Verify commands, links, package names, benchmark and evaluation names, and release status before documenting them.
Expand Down Expand Up @@ -91,15 +91,15 @@ The `typescript/` directory contains an unreleased Quantiles TypeScript SDK for

When editing the TypeScript SDK subdirectory, preserve strict typing, JSON-serializable public surfaces, ESM behavior, and documented package exports.

### Using provider-backed models
### Running evaluations with hosted models

Start with the smallest useful sample limit before running a full benchmark with either a demo model or a provider-backed model to validate configuration, catch setup issues early, and avoid unnecessary cost.
Start with the smallest useful sample limit before running a full benchmark with either a demo model or a model from a supported provider to validate configuration, catch setup issues early, and avoid unnecessary cost.

Ask before running any evaluation that is expected to be slow, expensive, provider-backed, network-dependent, destructive, or likely to meaningfully modify local run state. Note that demo model runs are for workflow validation only. Do not treat them as model-quality benchmark evidence.
Ask before running any evaluation that is expected to be slow, expensive, call an external model API, network-dependent, destructive, or likely to meaningfully modify local run state. Note that demo model runs are for workflow validation only. Do not treat them as model-quality benchmark evidence.

Do not run provider-backed evaluations unless the user explicitly asks for them or provides a provider-prefixed model name. Configure providers in the `quantiles.toml` config file. Follow configuration examples in the [`cli/examples/configs`](./cli/examples/configs) directory. Before running a provider-backed evaluation, verify that the required provider API key is configured, but never print or expose the key value.
Do not run evaluations that call external model APIs unless the user explicitly requests one or provides a provider-prefixed model name. Configure providers in the `quantiles.toml` config file using the [model configuration guide](https://quantiles.io/documentation/model-configuration). Follow configuration examples in the [`cli/examples/configs`](./cli/examples/configs) directory. Before running an evaluation that calls an external model API, verify that the required provider API key is configured, but never print or expose the key value.

Provider-backed model inputs should use provider-prefixed model names, for example:
Model inputs should use provider-prefixed model names, for example:

- `openai:<model>`
- `anthropic:<model>`
Expand Down
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -101,6 +101,7 @@ You can customize how the CLI executes [built-in-benchmarks](https://quantiles.i
See the following resources for more details:

- [Configuration guide](https://quantiles.io/documentation/configuration) - Detailed configuration instructions and reference documentation for supported fields, validation rules, and examples.
- [Model configuration guide](https://quantiles.io/documentation/model-configuration) - Configure provider models and credentials, and troubleshoot common setup issues.
- [Configuration examples](./cli/examples/configs) - Complete examples, including a [custom-code evaluation](./cli/examples/configs/custom_code/quantiles.toml)

#### Built-in benchmarks
Expand Down
2 changes: 1 addition & 1 deletion cli/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@ model = "openai:gpt-5.6"
max_workers = 100
```

See the [configuration guide](https://quantiles.io/documentation/configuration) for file location, supported fields, validation behavior, and examples. Additional runnable configurations are available in [CLI configuration examples](./examples/configs) and [custom no-code examples](../custom-nocode-examples/quantiles.toml).
See the [configuration guide](https://quantiles.io/documentation/configuration) for file location, supported fields, validation behavior, and examples. See the [model configuration guide](https://quantiles.io/documentation/model-configuration) for guidance on setting up provider models, managing credentials, and troubleshooting configuration issues. Additional runnable configurations are available in [CLI configuration examples](./examples/configs) and [custom no-code examples](../custom-nocode-examples/quantiles.toml).

## Architecture

Expand Down
Loading