diff --git a/README.md b/README.md index 82ccee3..2313164 100644 --- a/README.md +++ b/README.md @@ -8,7 +8,7 @@ Quantiles centralizes its components in this monorepo so developers, researchers ## ![New](./docs/assets/new-badge.svg) What's New -**[2026.07.07]** Added `custom_nocode` evaluations, which let users configure custom evals in `quantiles.toml` without writing or maintaining custom code. See the [custom no-code evaluation documentation](https://quantiles.io/documentation/custom-nocode-evaluations) for details. +**[2026.07.07]** Added `custom_nocode` evaluations, which let users configure custom evals in `quantiles.toml` without writing or maintaining custom code. See the [custom no-code evaluation documentation](https://quantiles.io/documentation/custom-evaluations/custom-nocode-evaluations) for details. ## Why use Quantiles? @@ -81,7 +81,7 @@ See the [CLI reference](https://quantiles.io/documentation/reference/cli) for av ### Configuration and customization -You can customize how the CLI executes [built-in-benchmarks](https://quantiles.io/documentation/built-in-benchmarks), [custom no-code evaluations](https://quantiles.io/documentation/custom-nocode-evaluations), and [custom code evaluations](https://quantiles.io/documentation/custom-evaluations) using a `quantiles.toml` or `.quantiles.toml` configuration file in the current working directory or a parent directory. The CLI uses this configuration each time you run the benchmark with `qt run`. +You can customize how the CLI executes [built-in-benchmarks](https://quantiles.io/documentation/built-in-benchmarks), [custom no-code evaluations](https://quantiles.io/documentation/custom-evaluations/custom-nocode-evaluations), and [custom code evaluations](https://quantiles.io/documentation/custom-evaluations) using a `quantiles.toml` or `.quantiles.toml` configuration file in the current working directory or a parent directory. The CLI uses this configuration each time you run the benchmark with `qt run`. See the following resources for more details: @@ -100,7 +100,7 @@ The [benchmark hub](https://quantiles.io/benchmark-hub) describes available benc Custom evaluations measure behavior specific to your product, workflow, prompt, dataset, rubric, or release process. Quantiles provides two ways to build them: -- [`custom_nocode`](https://quantiles.io/documentation/custom-nocode-evaluations): define a custom evaluation entirely in configuration. +- [`custom_nocode`](https://quantiles.io/documentation/custom-evaluations/custom-nocode-evaluations): define a custom evaluation entirely in configuration. - [`custom_code`](https://quantiles.io/documentation/custom-evaluations): build specialized evaluation logic with [Python](https://quantiles.io/documentation/reference/python-sdk). Prefer to use `custom_nocode` evaluations wherever possible, since they're easier for humans and agents to create and maintain. When required, fall back to `custom_code` evaluations. diff --git a/cli/README.md b/cli/README.md index 21ab163..e9c1f22 100644 --- a/cli/README.md +++ b/cli/README.md @@ -36,7 +36,7 @@ See the [CLI reference](https://quantiles.io/documentation/reference/cli) for a The CLI supports three evaluation types: - [Built-in benchmarks](https://quantiles.io/documentation/built-in-benchmarks) run predefined datasets and scoring methods. They work without configuration, but you can override settings such as the model, sample count, and concurrency. -- [`custom_nocode` evaluations](https://quantiles.io/documentation/custom-nocode-evaluations) define the dataset, prompt template, model, and scoring method entirely in configuration. +- [`custom_nocode` evaluations](https://quantiles.io/documentation/custom-evaluations/custom-nocode-evaluations) define the dataset, prompt template, model, and scoring method entirely in configuration. - [`custom_code` evaluations](https://quantiles.io/documentation/custom-evaluations) run your own Python evaluation through the Quantiles Python SDK. Add a `quantiles.toml` or `.quantiles.toml` file to configure an evaluation. For example: diff --git a/custom-nocode-examples/prompts/gpqa.txt b/custom-nocode-examples/prompts/gpqa.txt index 9da3c7a..ff394ae 100644 --- a/custom-nocode-examples/prompts/gpqa.txt +++ b/custom-nocode-examples/prompts/gpqa.txt @@ -1,8 +1,8 @@ {# -This is a prompt template for a custom no-code benchmark. -See the [`quantiles.toml`](../quantiles.toml) configuration to -understand how this template is used. -Read the [custom no-code evaluation documentation](https://quantiles.io/documentation/custom-nocode-evaluations) +This is a prompt template for a custom no-code benchmark. +See the [`quantiles.toml`](../quantiles.toml) configuration to +understand how this template is used. +Read the [custom no-code evaluation documentation](https://quantiles.io/documentation/custom-evaluations/custom-nocode-evaluations) to learn how to configure and run custom no-code evaluations. #} {{ row["Question"] }} diff --git a/custom-nocode-examples/prompts/medmcqa.txt b/custom-nocode-examples/prompts/medmcqa.txt index 980ec8e..8803677 100644 --- a/custom-nocode-examples/prompts/medmcqa.txt +++ b/custom-nocode-examples/prompts/medmcqa.txt @@ -1,8 +1,8 @@ {# -This is a prompt template for a custom no-code benchmark. -See the [`quantiles.toml`](../quantiles.toml) configuration to -understand how this template is used. -Read the [custom no-code evaluation documentation](https://quantiles.io/documentation/custom-nocode-evaluations) +This is a prompt template for a custom no-code benchmark. +See the [`quantiles.toml`](../quantiles.toml) configuration to +understand how this template is used. +Read the [custom no-code evaluation documentation](https://quantiles.io/documentation/custom-evaluations/custom-nocode-evaluations) to learn how to configure and run custom no-code evaluations. #} {{ row.question }} diff --git a/custom-nocode-examples/prompts/medqa.txt b/custom-nocode-examples/prompts/medqa.txt index 980ec8e..8803677 100644 --- a/custom-nocode-examples/prompts/medqa.txt +++ b/custom-nocode-examples/prompts/medqa.txt @@ -1,8 +1,8 @@ {# -This is a prompt template for a custom no-code benchmark. -See the [`quantiles.toml`](../quantiles.toml) configuration to -understand how this template is used. -Read the [custom no-code evaluation documentation](https://quantiles.io/documentation/custom-nocode-evaluations) +This is a prompt template for a custom no-code benchmark. +See the [`quantiles.toml`](../quantiles.toml) configuration to +understand how this template is used. +Read the [custom no-code evaluation documentation](https://quantiles.io/documentation/custom-evaluations/custom-nocode-evaluations) to learn how to configure and run custom no-code evaluations. #} {{ row.question }} diff --git a/custom-nocode-examples/prompts/mmlu-pro.txt b/custom-nocode-examples/prompts/mmlu-pro.txt index 980ec8e..8803677 100644 --- a/custom-nocode-examples/prompts/mmlu-pro.txt +++ b/custom-nocode-examples/prompts/mmlu-pro.txt @@ -1,8 +1,8 @@ {# -This is a prompt template for a custom no-code benchmark. -See the [`quantiles.toml`](../quantiles.toml) configuration to -understand how this template is used. -Read the [custom no-code evaluation documentation](https://quantiles.io/documentation/custom-nocode-evaluations) +This is a prompt template for a custom no-code benchmark. +See the [`quantiles.toml`](../quantiles.toml) configuration to +understand how this template is used. +Read the [custom no-code evaluation documentation](https://quantiles.io/documentation/custom-evaluations/custom-nocode-evaluations) to learn how to configure and run custom no-code evaluations. #} {{ row.question }} diff --git a/custom-nocode-examples/prompts/qa.txt b/custom-nocode-examples/prompts/qa.txt index 0262274..09c9882 100644 --- a/custom-nocode-examples/prompts/qa.txt +++ b/custom-nocode-examples/prompts/qa.txt @@ -1,8 +1,8 @@ {# -This is a prompt template for a custom no-code benchmark. -See the [`quantiles.toml`](../quantiles.toml) configuration to -understand how this template is used. -Read the [custom no-code evaluation documentation](https://quantiles.io/documentation/custom-nocode-evaluations) +This is a prompt template for a custom no-code benchmark. +See the [`quantiles.toml`](../quantiles.toml) configuration to +understand how this template is used. +Read the [custom no-code evaluation documentation](https://quantiles.io/documentation/custom-evaluations/custom-nocode-evaluations) to learn how to configure and run custom no-code evaluations. #} {{ row.problem }} diff --git a/custom-nocode-examples/quantiles.toml b/custom-nocode-examples/quantiles.toml index e44533a..d757f61 100644 --- a/custom-nocode-examples/quantiles.toml +++ b/custom-nocode-examples/quantiles.toml @@ -13,7 +13,7 @@ # # To read more about custom no-code evaluations, see the following documentation: # -# https://quantiles.io/documentation/custom-nocode-evaluations +# https://quantiles.io/documentation/custom-evaluations/custom-nocode-evaluations # The SimpleQA-verified becnhmark configured to use the random model.