Adding online agent evaluation sample notebooks for all evaluators

scenarios/evaluate/Supported_Evaluation_Metrics/Agent_Evaluation/AI_Judge_Evaluator_Coherence.ipynb

@@ -0,0 +1,245 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "da22a6e7",
+   "metadata": {},
+   "source": [
+    "# Coherence Evaluator\n",
+    "\n",
+    "## Objective\n",
+    "This sample demonstrates how to use the Coherence evaluator to assess the quality and readability of AI-generated responses. The evaluator measures the ability of the language model to generate text that reads naturally, flows smoothly, and resembles human-like language.\n",
+    "\n",
+    "## Time\n",
+    "\n",
+    "You should expect to spend about 20 minutes running this notebook. \n",
+    "\n",
+    "## Before you begin\n",
+    "For quality evaluation, you need to deploy a `gpt` model supporting JSON mode. We recommend using `gpt-4o` or `gpt-4.1`.    \n",
+    "\n",
+    "### Prerequisite\n",
+    "```bash\n",
+    "pip install azure-ai-projects azure-identity openai\n",
+    "```\n",
+    "Set these environment variables with your own values:\n",
+    "1) **AZURE_AI_PROJECT_ENDPOINT** - Your Azure AI project endpoint in format: `https://<account_name>.services.ai.azure.com/api/projects/<project_name>`\n",
+    "2) **AZURE_AI_MODEL_DEPLOYMENT_NAME** - The deployment name of the model for this AI-assisted evaluator (e.g., gpt-4o-mini)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "48f8cb64",
+   "metadata": {},
+   "source": [
+    "The Coherence evaluator assesses the ability of the language model to generate text that reads naturally, flows smoothly, and resembles human-like language in its responses. It measures the readability and user-friendliness of the model's generated responses.\n",
+    "\n",
+    "Coherence scores range from 1 to 5:\n",
+    "\n",
+    "<pre>\n",
+    "Score 1: Very Poor - The response is completely incoherent with no logical flow or connection between ideas.\n",
+    "Score 2: Poor - The response has significant coherence issues with disjointed ideas and unclear transitions.\n",
+    "Score 3: Fair - The response is somewhat coherent but has noticeable gaps in logical flow or organization.\n",
+    "Score 4: Good - The response is mostly coherent with good flow and logical connections between ideas.\n",
+    "Score 5: Excellent - The response is perfectly coherent with smooth, natural flow and clear logical progression.\n",
+    "</pre>\n",
+    "\n",
+    "The evaluation requires the following input pattern:\n",
+    "\n",
+    "**Pattern 1: Query-Response Evaluation**\n",
+    "- Query: The user's question or prompt. (string)\n",
+    "- Response: The AI-generated response to be evaluated for coherence. (string)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "542c1340",
+   "metadata": {},
+   "source": [
+    "### Initialize Coherence Evaluator"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "35799119",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import os\n",
+    "from openai.types.evals.create_eval_jsonl_run_data_source_param import SourceFileContentContent\n",
+    "from pprint import pprint\n",
+    "from agent_utils import run_evaluator\n",
+    "\n",
+    "# Get environment variables\n",
+    "deployment_name = os.environ[\"AZURE_AI_MODEL_DEPLOYMENT_NAME\"]\n",
+    "\n",
+    "# Data source configuration (defines the schema for evaluation inputs)\n",
+    "data_source_config = {\n",
+    "    \"type\": \"custom\",\n",
+    "    \"item_schema\": {\n",
+    "        \"type\": \"object\",\n",
+    "        \"properties\": {\n",
+    "            \"query\": {\n",
+    "                \"type\": \"string\"\n",
+    "            },\n",
+    "            \"response\": {\n",
+    "                \"type\": \"string\"\n",
+    "            }\n",
+    "        },\n",
+    "        \"required\": []\n",
+    "    },\n",
+    "    \"include_sample_schema\": True\n",
+    "}\n",
+    "\n",
+    "# Data mapping (maps evaluation inputs to evaluator parameters)\n",
+    "data_mapping = {\n",
+    "    \"query\": \"{{item.query}}\",\n",
+    "    \"response\": \"{{item.response}}\"\n",
+    "}\n",
+    "\n",
+    "# Initialization parameters for the evaluator\n",
+    "initialization_parameters = {\n",
+    "    \"deployment_name\": deployment_name\n",
+    "}\n",
+    "\n",
+    "# Initialize the evaluation_contents list - we'll append all test cases here\n",
+    "evaluation_contents = []"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "b10aff00",
+   "metadata": {},
+   "source": [
+    "### Samples"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "724adfc8",
+   "metadata": {},
+   "source": [
+    "#### Query and Response as Strings (str)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "82a1be93",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Test Case 1: High coherence response with logical flow and clear connections\n",
+    "query = \"Can you explain how machine learning works?\"\n",
+    "response = \"Machine learning is a subset of artificial intelligence that enables systems to learn and improve from experience without being explicitly programmed. It works by analyzing data patterns to make predictions or decisions. First, the system is trained on a dataset, which allows it to identify patterns and relationships. Then, it uses these learned patterns to make predictions on new, unseen data. The accuracy improves over time as the model processes more data and refines its understanding.\"\n",
+    "\n",
+    "# Append to evaluation_contents\n",
+    "evaluation_contents.append(\n",
+    "    SourceFileContentContent(\n",
+    "        item={\n",
+    "            \"query\": query,\n",
+    "            \"response\": response\n",
+    "        }\n",
+    "    )\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "bc250127",
+   "metadata": {},
+   "source": [
+    "#### Example of Poor Coherence"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "62ad5478",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Test Case 2: Poor coherence example with disjointed ideas and unclear transitions\n",
+    "query = \"What is blockchain technology?\"\n",
+    "response = \"Blockchain is decentralized. Bitcoin uses it. Blocks contain data. Cryptography is involved. Banks might use blockchain. It's secure because of the chain. Mining is how new blocks are added. Digital currencies exist. The ledger is distributed.\"\n",
+    "\n",
+    "# Append to evaluation_contents\n",
+    "evaluation_contents.append(\n",
+    "    SourceFileContentContent(\n",
+    "        item={\n",
+    "            \"query\": query,\n",
+    "            \"response\": response\n",
+    "        }\n",
+    "    )\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "ad4b46bd",
+   "metadata": {},
+   "source": [
+    "### Run Evaluation on All Test Cases\n",
+    "\n",
+    "Now that we've defined all test cases, let's run the evaluation once on all of them."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "141e26ab",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "results = run_evaluator(\n",
+    "    evaluator_name=\"coherence\",\n",
+    "    evaluation_contents=evaluation_contents,\n",
+    "    data_source_config=data_source_config,\n",
+    "    initialization_parameters=initialization_parameters,\n",
+    "    data_mapping=data_mapping\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "2e15dccc",
+   "metadata": {},
+   "source": [
+    "### Display Results\n",
+    "\n",
+    "View the evaluation results for each test case."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "c164710e",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "pprint(results)"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "samples",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.19"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}