Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ Quantiles centralizes its components in this monorepo so developers, researchers

## ![New](./docs/assets/new-badge.svg) What's New

**[2026.07.07]** Added `custom_nocode` evaluations, which let users configure custom evals in `quantiles.toml` without writing or maintaining custom code. See the [custom no-code evaluation documentation](https://quantiles.io/documentation/custom-nocode-evaluations) for details.
**[2026.07.07]** Added `custom_nocode` evaluations, which let users configure custom evals in `quantiles.toml` without writing or maintaining custom code. See the [custom no-code evaluation documentation](https://quantiles.io/documentation/custom-evaluations/custom-nocode-evaluations) for details.

## Why use Quantiles?

Expand Down Expand Up @@ -81,7 +81,7 @@ See the [CLI reference](https://quantiles.io/documentation/reference/cli) for av

### Configuration and customization

You can customize how the CLI executes [built-in-benchmarks](https://quantiles.io/documentation/built-in-benchmarks), [custom no-code evaluations](https://quantiles.io/documentation/custom-nocode-evaluations), and [custom code evaluations](https://quantiles.io/documentation/custom-evaluations) using a `quantiles.toml` or `.quantiles.toml` configuration file in the current working directory or a parent directory. The CLI uses this configuration each time you run the benchmark with `qt run`.
You can customize how the CLI executes [built-in-benchmarks](https://quantiles.io/documentation/built-in-benchmarks), [custom no-code evaluations](https://quantiles.io/documentation/custom-evaluations/custom-nocode-evaluations), and [custom code evaluations](https://quantiles.io/documentation/custom-evaluations) using a `quantiles.toml` or `.quantiles.toml` configuration file in the current working directory or a parent directory. The CLI uses this configuration each time you run the benchmark with `qt run`.

See the following resources for more details:

Expand All @@ -100,7 +100,7 @@ The [benchmark hub](https://quantiles.io/benchmark-hub) describes available benc

Custom evaluations measure behavior specific to your product, workflow, prompt, dataset, rubric, or release process. Quantiles provides two ways to build them:

- [`custom_nocode`](https://quantiles.io/documentation/custom-nocode-evaluations): define a custom evaluation entirely in configuration.
- [`custom_nocode`](https://quantiles.io/documentation/custom-evaluations/custom-nocode-evaluations): define a custom evaluation entirely in configuration.
- [`custom_code`](https://quantiles.io/documentation/custom-evaluations): build specialized evaluation logic with [Python](https://quantiles.io/documentation/reference/python-sdk).

Prefer to use `custom_nocode` evaluations wherever possible, since they're easier for humans and agents to create and maintain. When required, fall back to `custom_code` evaluations.
Expand Down
2 changes: 1 addition & 1 deletion cli/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ See the [CLI reference](https://quantiles.io/documentation/reference/cli) for a
The CLI supports three evaluation types:

- [Built-in benchmarks](https://quantiles.io/documentation/built-in-benchmarks) run predefined datasets and scoring methods. They work without configuration, but you can override settings such as the model, sample count, and concurrency.
- [`custom_nocode` evaluations](https://quantiles.io/documentation/custom-nocode-evaluations) define the dataset, prompt template, model, and scoring method entirely in configuration.
- [`custom_nocode` evaluations](https://quantiles.io/documentation/custom-evaluations/custom-nocode-evaluations) define the dataset, prompt template, model, and scoring method entirely in configuration.
- [`custom_code` evaluations](https://quantiles.io/documentation/custom-evaluations) run your own Python evaluation through the Quantiles Python SDK.

Add a `quantiles.toml` or `.quantiles.toml` file to configure an evaluation. For example:
Expand Down
8 changes: 4 additions & 4 deletions custom-nocode-examples/prompts/gpqa.txt
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
{#
This is a prompt template for a custom no-code benchmark.
See the [`quantiles.toml`](../quantiles.toml) configuration to
understand how this template is used.
Read the [custom no-code evaluation documentation](https://quantiles.io/documentation/custom-nocode-evaluations)
This is a prompt template for a custom no-code benchmark.
See the [`quantiles.toml`](../quantiles.toml) configuration to
understand how this template is used.
Read the [custom no-code evaluation documentation](https://quantiles.io/documentation/custom-evaluations/custom-nocode-evaluations)
to learn how to configure and run custom no-code evaluations.
#}
{{ row["Question"] }}
Expand Down
8 changes: 4 additions & 4 deletions custom-nocode-examples/prompts/medmcqa.txt
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
{#
This is a prompt template for a custom no-code benchmark.
See the [`quantiles.toml`](../quantiles.toml) configuration to
understand how this template is used.
Read the [custom no-code evaluation documentation](https://quantiles.io/documentation/custom-nocode-evaluations)
This is a prompt template for a custom no-code benchmark.
See the [`quantiles.toml`](../quantiles.toml) configuration to
understand how this template is used.
Read the [custom no-code evaluation documentation](https://quantiles.io/documentation/custom-evaluations/custom-nocode-evaluations)
to learn how to configure and run custom no-code evaluations.
#}
{{ row.question }}
Expand Down
8 changes: 4 additions & 4 deletions custom-nocode-examples/prompts/medqa.txt
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
{#
This is a prompt template for a custom no-code benchmark.
See the [`quantiles.toml`](../quantiles.toml) configuration to
understand how this template is used.
Read the [custom no-code evaluation documentation](https://quantiles.io/documentation/custom-nocode-evaluations)
This is a prompt template for a custom no-code benchmark.
See the [`quantiles.toml`](../quantiles.toml) configuration to
understand how this template is used.
Read the [custom no-code evaluation documentation](https://quantiles.io/documentation/custom-evaluations/custom-nocode-evaluations)
to learn how to configure and run custom no-code evaluations.
#}
{{ row.question }}
Expand Down
8 changes: 4 additions & 4 deletions custom-nocode-examples/prompts/mmlu-pro.txt
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
{#
This is a prompt template for a custom no-code benchmark.
See the [`quantiles.toml`](../quantiles.toml) configuration to
understand how this template is used.
Read the [custom no-code evaluation documentation](https://quantiles.io/documentation/custom-nocode-evaluations)
This is a prompt template for a custom no-code benchmark.
See the [`quantiles.toml`](../quantiles.toml) configuration to
understand how this template is used.
Read the [custom no-code evaluation documentation](https://quantiles.io/documentation/custom-evaluations/custom-nocode-evaluations)
to learn how to configure and run custom no-code evaluations.
#}
{{ row.question }}
Expand Down
8 changes: 4 additions & 4 deletions custom-nocode-examples/prompts/qa.txt
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
{#
This is a prompt template for a custom no-code benchmark.
See the [`quantiles.toml`](../quantiles.toml) configuration to
understand how this template is used.
Read the [custom no-code evaluation documentation](https://quantiles.io/documentation/custom-nocode-evaluations)
This is a prompt template for a custom no-code benchmark.
See the [`quantiles.toml`](../quantiles.toml) configuration to
understand how this template is used.
Read the [custom no-code evaluation documentation](https://quantiles.io/documentation/custom-evaluations/custom-nocode-evaluations)
to learn how to configure and run custom no-code evaluations.
#}
{{ row.problem }}
2 changes: 1 addition & 1 deletion custom-nocode-examples/quantiles.toml
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@
#
# To read more about custom no-code evaluations, see the following documentation:
#
# https://quantiles.io/documentation/custom-nocode-evaluations
# https://quantiles.io/documentation/custom-evaluations/custom-nocode-evaluations


# The SimpleQA-verified becnhmark configured to use the random model.
Expand Down
Loading