diff --git a/AGENTS.md b/AGENTS.md index 91d2e86..637132c 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -37,7 +37,7 @@ When performing tasks in this repository, always do the following: - Read the nearest subdirectory `AGENTS.md` if present. - Inspect the relevant README, package configuration, and existing tests before changing behavior. - Prefer the smallest change that satisfies the task. -- Do not run expensive, provider-backed, or full benchmark commands without explicit approval. +- Obtain explicit approval for external model calls and potentially expensive evaluation runs. - Keep changes small, public-safe, and reviewable. - Preserve Quantiles as a local-first, offline system by default. - Verify commands, links, package names, benchmark and evaluation names, and release status before documenting them. @@ -91,15 +91,15 @@ The `typescript/` directory contains an unreleased Quantiles TypeScript SDK for When editing the TypeScript SDK subdirectory, preserve strict typing, JSON-serializable public surfaces, ESM behavior, and documented package exports. -### Using provider-backed models +### Running evaluations with hosted models -Start with the smallest useful sample limit before running a full benchmark with either a demo model or a provider-backed model to validate configuration, catch setup issues early, and avoid unnecessary cost. +Start with the smallest useful sample limit before running a full benchmark with either a demo model or a model from a supported provider to validate configuration, catch setup issues early, and avoid unnecessary cost. -Ask before running any evaluation that is expected to be slow, expensive, provider-backed, network-dependent, destructive, or likely to meaningfully modify local run state. Note that demo model runs are for workflow validation only. Do not treat them as model-quality benchmark evidence. +Ask before running any evaluation that is expected to be slow, expensive, call an external model API, network-dependent, destructive, or likely to meaningfully modify local run state. Note that demo model runs are for workflow validation only. Do not treat them as model-quality benchmark evidence. -Do not run provider-backed evaluations unless the user explicitly asks for them or provides a provider-prefixed model name. Configure providers in the `quantiles.toml` config file. Follow configuration examples in the [`cli/examples/configs`](./cli/examples/configs) directory. Before running a provider-backed evaluation, verify that the required provider API key is configured, but never print or expose the key value. +Do not run evaluations that call external model APIs unless the user explicitly requests one or provides a provider-prefixed model name. Configure providers in the `quantiles.toml` config file using the [model configuration guide](https://quantiles.io/documentation/model-configuration). Follow configuration examples in the [`cli/examples/configs`](./cli/examples/configs) directory. Before running an evaluation that calls an external model API, verify that the required provider API key is configured, but never print or expose the key value. -Provider-backed model inputs should use provider-prefixed model names, for example: +Model inputs should use provider-prefixed model names, for example: - `openai:` - `anthropic:` diff --git a/README.md b/README.md index 63db577..60c17ff 100644 --- a/README.md +++ b/README.md @@ -101,6 +101,7 @@ You can customize how the CLI executes [built-in-benchmarks](https://quantiles.i See the following resources for more details: - [Configuration guide](https://quantiles.io/documentation/configuration) - Detailed configuration instructions and reference documentation for supported fields, validation rules, and examples. +- [Model configuration guide](https://quantiles.io/documentation/model-configuration) - Configure provider models and credentials, and troubleshoot common setup issues. - [Configuration examples](./cli/examples/configs) - Complete examples, including a [custom-code evaluation](./cli/examples/configs/custom_code/quantiles.toml) #### Built-in benchmarks diff --git a/cli/README.md b/cli/README.md index e9c1f22..0f02450 100644 --- a/cli/README.md +++ b/cli/README.md @@ -48,7 +48,7 @@ model = "openai:gpt-5.6" max_workers = 100 ``` -See the [configuration guide](https://quantiles.io/documentation/configuration) for file location, supported fields, validation behavior, and examples. Additional runnable configurations are available in [CLI configuration examples](./examples/configs) and [custom no-code examples](../custom-nocode-examples/quantiles.toml). +See the [configuration guide](https://quantiles.io/documentation/configuration) for file location, supported fields, validation behavior, and examples. See the [model configuration guide](https://quantiles.io/documentation/model-configuration) for guidance on setting up provider models, managing credentials, and troubleshooting configuration issues. Additional runnable configurations are available in [CLI configuration examples](./examples/configs) and [custom no-code examples](../custom-nocode-examples/quantiles.toml). ## Architecture