From bd2a8e2a118250004539917063ccd8236b990d94 Mon Sep 17 00:00:00 2001 From: Paulo Lacerda Date: Wed, 10 Jun 2026 12:20:54 -0300 Subject: [PATCH] docs: explain the auto-generated rubric evaluator in step 11 Step 11 previously told readers to `find the entry with a local_uri` without saying what that evaluator (`smoke-core` in the seeded eval.yaml) actually is or where it came from. Add a short paragraph right before `Find the evaluator name` that explains the difference between the built-in evaluators and the auto-generated local rubric evaluator, and why `rubrics:` in agentops.yaml reference its name. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- docs/tutorial-prompt-agent-quickstart.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/docs/tutorial-prompt-agent-quickstart.md b/docs/tutorial-prompt-agent-quickstart.md index 68f332e..24d8e75 100644 --- a/docs/tutorial-prompt-agent-quickstart.md +++ b/docs/tutorial-prompt-agent-quickstart.md @@ -902,6 +902,15 @@ Fill in two kinds of real names: the rubric evaluator name and the rubric dimension names. Do not invent values — both must come from files `agentops eval init` already generated on disk. +> **About the auto-generated evaluator.** When you ran `agentops eval +> init`, azd seeded `src/travel-agent/eval.yaml` with two kinds of +> evaluators: built-ins like `builtin.coherence` and `builtin.fluency` +> (general response-quality checks) plus a local rubric evaluator — +> typically `name: smoke-core` — whose `local_uri` points at a JSON file +> with rubric dimensions specific to this Travel Agent. That local +> evaluator is the hook AgentOps `rubrics:` bind to. You will reference +> its `name:` and its dimension `id`s in the next two steps. + **1. Find the evaluator name.** Open `src/travel-agent/eval.yaml` and look under `evaluators:` for the entry with a `local_uri`: