diff --git a/scenarios/evaluate/Supported_Evaluation_Metrics/Agent_Evaluation/AI_Judge_Evaluator_Coherence.ipynb b/scenarios/evaluate/Supported_Evaluation_Metrics/Agent_Evaluation/AI_Judge_Evaluator_Coherence.ipynb
new file mode 100644
index 00000000..99fa87ee
--- /dev/null
+++ b/scenarios/evaluate/Supported_Evaluation_Metrics/Agent_Evaluation/AI_Judge_Evaluator_Coherence.ipynb
@@ -0,0 +1,245 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "da22a6e7",
+   "metadata": {},
+   "source": [
+    "# Coherence Evaluator\n",
+    "\n",
+    "## Objective\n",
+    "This sample demonstrates how to use the Coherence evaluator to assess the quality and readability of AI-generated responses. The evaluator measures the ability of the language model to generate text that reads naturally, flows smoothly, and resembles human-like language.\n",
+    "\n",
+    "## Time\n",
+    "\n",
+    "You should expect to spend about 20 minutes running this notebook. \n",
+    "\n",
+    "## Before you begin\n",
+    "For quality evaluation, you need to deploy a `gpt` model supporting JSON mode. We recommend using `gpt-4o` or `gpt-4.1`.    \n",
+    "\n",
+    "### Prerequisite\n",
+    "```bash\n",
+    "pip install azure-ai-projects azure-identity openai\n",
+    "```\n",
+    "Set these environment variables with your own values:\n",
+    "1) **AZURE_AI_PROJECT_ENDPOINT** - Your Azure AI project endpoint in format: `https://<account_name>.services.ai.azure.com/api/projects/<project_name>`\n",
+    "2) **AZURE_AI_MODEL_DEPLOYMENT_NAME** - The deployment name of the model for this AI-assisted evaluator (e.g., gpt-4o-mini)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "48f8cb64",
+   "metadata": {},
+   "source": [
+    "The Coherence evaluator assesses the ability of the language model to generate text that reads naturally, flows smoothly, and resembles human-like language in its responses. It measures the readability and user-friendliness of the model's generated responses.\n",
+    "\n",
+    "Coherence scores range from 1 to 5:\n",
+    "\n",
+    "<pre>\n",
+    "Score 1: Very Poor - The response is completely incoherent with no logical flow or connection between ideas.\n",
+    "Score 2: Poor - The response has significant coherence issues with disjointed ideas and unclear transitions.\n",
+    "Score 3: Fair - The response is somewhat coherent but has noticeable gaps in logical flow or organization.\n",
+    "Score 4: Good - The response is mostly coherent with good flow and logical connections between ideas.\n",
+    "Score 5: Excellent - The response is perfectly coherent with smooth, natural flow and clear logical progression.\n",
+    "</pre>\n",
+    "\n",
+    "The evaluation requires the following input pattern:\n",
+    "\n",
+    "**Pattern 1: Query-Response Evaluation**\n",
+    "- Query: The user's question or prompt. (string)\n",
+    "- Response: The AI-generated response to be evaluated for coherence. (string)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "542c1340",
+   "metadata": {},
+   "source": [
+    "### Initialize Coherence Evaluator"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "35799119",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import os\n",
+    "from openai.types.evals.create_eval_jsonl_run_data_source_param import SourceFileContentContent\n",
+    "from pprint import pprint\n",
+    "from agent_utils import run_evaluator\n",
+    "\n",
+    "# Get environment variables\n",
+    "deployment_name = os.environ[\"AZURE_AI_MODEL_DEPLOYMENT_NAME\"]\n",
+    "\n",
+    "# Data source configuration (defines the schema for evaluation inputs)\n",
+    "data_source_config = {\n",
+    "    \"type\": \"custom\",\n",
+    "    \"item_schema\": {\n",
+    "        \"type\": \"object\",\n",
+    "        \"properties\": {\n",
+    "            \"query\": {\n",
+    "                \"type\": \"string\"\n",
+    "            },\n",
+    "            \"response\": {\n",
+    "                \"type\": \"string\"\n",
+    "            }\n",
+    "        },\n",
+    "        \"required\": []\n",
+    "    },\n",
+    "    \"include_sample_schema\": True\n",
+    "}\n",
+    "\n",
+    "# Data mapping (maps evaluation inputs to evaluator parameters)\n",
+    "data_mapping = {\n",
+    "    \"query\": \"{{item.query}}\",\n",
+    "    \"response\": \"{{item.response}}\"\n",
+    "}\n",
+    "\n",
+    "# Initialization parameters for the evaluator\n",
+    "initialization_parameters = {\n",
+    "    \"deployment_name\": deployment_name\n",
+    "}\n",
+    "\n",
+    "# Initialize the evaluation_contents list - we'll append all test cases here\n",
+    "evaluation_contents = []"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "b10aff00",
+   "metadata": {},
+   "source": [
+    "### Samples"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "724adfc8",
+   "metadata": {},
+   "source": [
+    "#### Query and Response as Strings (str)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "82a1be93",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Test Case 1: High coherence response with logical flow and clear connections\n",
+    "query = \"Can you explain how machine learning works?\"\n",
+    "response = \"Machine learning is a subset of artificial intelligence that enables systems to learn and improve from experience without being explicitly programmed. It works by analyzing data patterns to make predictions or decisions. First, the system is trained on a dataset, which allows it to identify patterns and relationships. Then, it uses these learned patterns to make predictions on new, unseen data. The accuracy improves over time as the model processes more data and refines its understanding.\"\n",
+    "\n",
+    "# Append to evaluation_contents\n",
+    "evaluation_contents.append(\n",
+    "    SourceFileContentContent(\n",
+    "        item={\n",
+    "            \"query\": query,\n",
+    "            \"response\": response\n",
+    "        }\n",
+    "    )\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "bc250127",
+   "metadata": {},
+   "source": [
+    "#### Example of Poor Coherence"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "62ad5478",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Test Case 2: Poor coherence example with disjointed ideas and unclear transitions\n",
+    "query = \"What is blockchain technology?\"\n",
+    "response = \"Blockchain is decentralized. Bitcoin uses it. Blocks contain data. Cryptography is involved. Banks might use blockchain. It's secure because of the chain. Mining is how new blocks are added. Digital currencies exist. The ledger is distributed.\"\n",
+    "\n",
+    "# Append to evaluation_contents\n",
+    "evaluation_contents.append(\n",
+    "    SourceFileContentContent(\n",
+    "        item={\n",
+    "            \"query\": query,\n",
+    "            \"response\": response\n",
+    "        }\n",
+    "    )\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "ad4b46bd",
+   "metadata": {},
+   "source": [
+    "### Run Evaluation on All Test Cases\n",
+    "\n",
+    "Now that we've defined all test cases, let's run the evaluation once on all of them."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "141e26ab",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "results = run_evaluator(\n",
+    "    evaluator_name=\"coherence\",\n",
+    "    evaluation_contents=evaluation_contents,\n",
+    "    data_source_config=data_source_config,\n",
+    "    initialization_parameters=initialization_parameters,\n",
+    "    data_mapping=data_mapping\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "2e15dccc",
+   "metadata": {},
+   "source": [
+    "### Display Results\n",
+    "\n",
+    "View the evaluation results for each test case."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "c164710e",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "pprint(results)"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "samples",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.19"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
diff --git a/scenarios/evaluate/Supported_Evaluation_Metrics/Agent_Evaluation/AI_Judge_Evaluator_Fluency.ipynb b/scenarios/evaluate/Supported_Evaluation_Metrics/Agent_Evaluation/AI_Judge_Evaluator_Fluency.ipynb
new file mode 100644
index 00000000..e9422f37
--- /dev/null
+++ b/scenarios/evaluate/Supported_Evaluation_Metrics/Agent_Evaluation/AI_Judge_Evaluator_Fluency.ipynb
@@ -0,0 +1,233 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "21599c07",
+   "metadata": {},
+   "source": [
+    "# Fluency Evaluator\n",
+    "\n",
+    "## Objective\n",
+    "This sample demonstrates how to use the Fluency evaluator to assess the linguistic quality of AI-generated responses. The evaluator measures how well generated text conforms to grammatical rules, syntactic structures, and appropriate vocabulary usage.\n",
+    "\n",
+    "## Time\n",
+    "\n",
+    "You should expect to spend about 20 minutes running this notebook. \n",
+    "\n",
+    "## Before you begin\n",
+    "For quality evaluation, you need to deploy a `gpt` model supporting JSON mode.  We recommend using `gpt-4o` or `gpt-4.1`.    \n",
+    "\n",
+    "### Prerequisite\n",
+    "```bash\n",
+    "pip install azure-ai-projects azure-identity openai\n",
+    "```\n",
+    "Set these environment variables with your own values:\n",
+    "1) **AZURE_AI_PROJECT_ENDPOINT** - Your Azure AI project endpoint in format: `https://<account_name>.services.ai.azure.com/api/projects/<project_name>`\n",
+    "2) **AZURE_AI_MODEL_DEPLOYMENT_NAME** - The deployment name of the model for this AI-assisted evaluator (e.g., gpt-4o-mini)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "7add5ec1",
+   "metadata": {},
+   "source": [
+    "The Fluency evaluator assesses the extent to which generated text conforms to grammatical rules, syntactic structures, and appropriate vocabulary usage, resulting in linguistically correct responses.\n",
+    "\n",
+    "Fluency scores range from 1 to 5:\n",
+    "\n",
+    "<pre>\n",
+    "Score 1: Very Poor - The response is incomprehensible with severe grammatical errors and improper vocabulary.\n",
+    "Score 2: Poor - The response has frequent grammatical errors and awkward phrasing that hinder understanding.\n",
+    "Score 3: Fair - The response is understandable but contains noticeable grammatical errors or awkward expressions.\n",
+    "Score 4: Good - The response is mostly fluent with minor grammatical issues that don't significantly impact readability.\n",
+    "Score 5: Excellent - The response is perfectly fluent with proper grammar, syntax, and vocabulary usage.\n",
+    "</pre>\n",
+    "\n",
+    "The evaluation requires the following input format:\n",
+    "\n",
+    "**Query-Response Evaluation**\n",
+    "- Response: The AI's response to be evaluated for fluency (string)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "2abd4d05",
+   "metadata": {},
+   "source": [
+    "### Initialize Fluency Evaluator"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "bfd097f8",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import os\n",
+    "from openai.types.evals.create_eval_jsonl_run_data_source_param import SourceFileContentContent\n",
+    "from pprint import pprint\n",
+    "from agent_utils import run_evaluator\n",
+    "\n",
+    "# Get environment variables\n",
+    "deployment_name = os.environ[\"AZURE_AI_MODEL_DEPLOYMENT_NAME\"]\n",
+    "\n",
+    "# Data source configuration (defines the schema for evaluation inputs)\n",
+    "data_source_config = {\n",
+    "    \"type\": \"custom\",\n",
+    "    \"item_schema\": {\n",
+    "        \"type\": \"object\",\n",
+    "        \"properties\": {\"query\": {\"type\": \"string\"}, \"response\": {\"type\": \"string\"}},\n",
+    "        \"required\": [],\n",
+    "    },\n",
+    "    \"include_sample_schema\": True,\n",
+    "}\n",
+    "\n",
+    "# Data mapping (maps evaluation inputs to evaluator parameters)\n",
+    "data_mapping = {\n",
+    "    \"query\": \"{{item.query}}\",\n",
+    "    \"response\": \"{{item.response}}\"\n",
+    "}\n",
+    "\n",
+    "# Initialization parameters for the evaluator\n",
+    "initialization_parameters = {\n",
+    "    \"deployment_name\": deployment_name\n",
+    "}\n",
+    "\n",
+    "# Initialize the evaluation_contents list - we'll append all test cases here\n",
+    "evaluation_contents = []"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "4793759a",
+   "metadata": {},
+   "source": [
+    "### Samples"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "46dc8414",
+   "metadata": {},
+   "source": [
+    "#### Response as String (str)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "3c2df39a",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "response = \"The weather in Seattle is currently partly cloudy with a temperature of 15°C. The forecast indicates that conditions will remain stable throughout the day, with a gentle breeze from the northwest.\"\n",
+    "\n",
+    "evaluation_contents.append(\n",
+    "    SourceFileContentContent(\n",
+    "        item={\n",
+    "            \"query\": None,\n",
+    "            \"response\": response\n",
+    "        }\n",
+    "    )\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "59709c65",
+   "metadata": {},
+   "source": [
+    "#### Example of Poor Fluency"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "d9c624ab",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Poor fluency example\n",
+    "response = \"Email draft attach is. You review and giving feedback must. Important very for project success it being.\"\n",
+    "\n",
+    "# Append to evaluation_contents\n",
+    "evaluation_contents.append(\n",
+    "    SourceFileContentContent(\n",
+    "        item={\n",
+    "            \"query\": None,\n",
+    "            \"response\": response\n",
+    "        }\n",
+    "    )\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "91b113ad",
+   "metadata": {},
+   "source": [
+    "### Run Evaluation on All Test Cases\n",
+    "\n",
+    "Now that we've defined all test cases, let's run the evaluation once on all of them."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "b1d14a1c",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "results = run_evaluator(\n",
+    "    evaluator_name=\"fluency\",\n",
+    "    evaluation_contents=evaluation_contents,\n",
+    "    data_source_config=data_source_config,\n",
+    "    initialization_parameters=initialization_parameters,\n",
+    "    data_mapping=data_mapping\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "80270783",
+   "metadata": {},
+   "source": [
+    "### Display Results\n",
+    "\n",
+    "View the evaluation results for each test case."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "ecc32e9a",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "pprint(results)"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "samples",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.19"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
diff --git a/scenarios/evaluate/Supported_Evaluation_Metrics/Agent_Evaluation/AI_Judge_Evaluator_Groundedness.ipynb b/scenarios/evaluate/Supported_Evaluation_Metrics/Agent_Evaluation/AI_Judge_Evaluator_Groundedness.ipynb
new file mode 100644
index 00000000..5d8e6dcb
--- /dev/null
+++ b/scenarios/evaluate/Supported_Evaluation_Metrics/Agent_Evaluation/AI_Judge_Evaluator_Groundedness.ipynb
@@ -0,0 +1,484 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "6a3f10c0",
+   "metadata": {},
+   "source": [
+    "# Groundedness Evaluator\n",
+    "\n",
+    "## Objective\n",
+    "This sample demonstrates how to use the Groundedness evaluator to assess whether AI-generated responses are grounded in the provided context. The evaluator supports multiple input formats including:\n",
+    "- Simple response and context evaluation\n",
+    "- Query, response, and context evaluation\n",
+    "- Agent responses with tool calls (file_search)\n",
+    "- Multi-turn conversations\n",
+    "\n",
+    "This notebook uses consistent examples with the ToolCallAccuracyEvaluator for better comparison across evaluators.\n",
+    "\n",
+    "## Time\n",
+    "\n",
+    "You should expect to spend about 20 minutes running this notebook.\n",
+    "\n",
+    "## Before you begin\n",
+    "For quality evaluation, you need to deploy a `gpt` model supporting JSON mode. We recommend using `gpt-4o`.\n",
+    "\n",
+    "### Prerequisite\n",
+    "```bash\n",
+    "pip install azure-ai-projects azure-identity openai\n",
+    "```\n",
+    "Set these environment variables with your own values:\n",
+    "1) **AZURE_AI_PROJECT_ENDPOINT** - Your Azure AI project endpoint in format: `https://<account_name>.services.ai.azure.com/api/projects/<project_name>`\n",
+    "2) **AZURE_AI_MODEL_DEPLOYMENT_NAME** - The deployment name of the model for this AI-assisted evaluator (e.g., gpt-4o-mini)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "fe3af2a1",
+   "metadata": {},
+   "source": [
+    "## What is Groundedness?\n",
+    "\n",
+    "The Groundedness evaluator assesses the correspondence between claims in an AI-generated response and the source context. It ensures that responses are substantiated by the provided context, preventing hallucinations and unsupported claims.\n",
+    "\n",
+    "**Key Points:**\n",
+    "- Even factually correct responses are considered ungrounded if they can't be verified against the provided context\n",
+    "- Essential for RAG (Retrieval-Augmented Generation) applications\n",
+    "- Helps ensure AI responses are trustworthy and verifiable\n",
+    "\n",
+    "**Scoring:** Groundedness scores range from 1 to 5, with:\n",
+    "- **1**: Completely ungrounded - no claims supported by context\n",
+    "- **2**: Mostly ungrounded - few claims supported\n",
+    "- **3**: Partially grounded - some claims supported\n",
+    "- **4**: Mostly grounded - most claims supported\n",
+    "- **5**: Fully grounded - all claims supported by context"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "bd8a00fb",
+   "metadata": {},
+   "source": [
+    "## Groundedness Evaluator Input Requirements\n",
+    "\n",
+    "The Groundedness evaluator supports multiple input formats:\n",
+    "\n",
+    "1. **Basic Context Evaluation:**\n",
+    "   - `response`: The AI response to evaluate (str)\n",
+    "   - `context`: The source context/documents (str)\n",
+    "   - `query`: Optional query for enhanced evaluation (str)\n",
+    "\n",
+    "2. **Agent Tool Evaluation:**\n",
+    "   - `query`: The user query (str)\n",
+    "   - `response`: Agent response with tool calls (List[dict])\n",
+    "   - `tool_definitions`: Available tools, only file_search supported (List[dict])\n",
+    "\n",
+    "3. **Conversation Evaluation:**\n",
+    "   - `conversation`: Multi-turn conversation with context (Conversation object)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "9aa68c67",
+   "metadata": {},
+   "source": [
+    "### Initialize Groundedness Evaluator"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "298d067a",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import os\n",
+    "from openai.types.evals.create_eval_jsonl_run_data_source_param import SourceFileContentContent\n",
+    "from pprint import pprint\n",
+    "from agent_utils import run_evaluator\n",
+    "\n",
+    "# Get environment variables\n",
+    "deployment_name = os.environ[\"AZURE_AI_MODEL_DEPLOYMENT_NAME\"]\n",
+    "\n",
+    "# Data source configuration (defines the schema for evaluation inputs)\n",
+    "data_source_config = {\n",
+    "    \"type\": \"custom\",\n",
+    "    \"item_schema\": {\n",
+    "        \"type\": \"object\",\n",
+    "        \"properties\": {\n",
+    "            \"context\": {\n",
+    "                \"type\": \"string\"\n",
+    "            },\n",
+    "            \"query\": {\n",
+    "                \"anyOf\": [\n",
+    "                    {\"type\": \"string\"},\n",
+    "                    {\n",
+    "                        \"type\": \"array\",\n",
+    "                        \"items\": {\n",
+    "                            \"type\": \"object\"\n",
+    "                        }\n",
+    "                    }\n",
+    "                ]\n",
+    "            },\n",
+    "            \"response\": {\n",
+    "                \"anyOf\": [\n",
+    "                    {\"type\": \"string\"},\n",
+    "                    {\n",
+    "                        \"type\": \"array\",\n",
+    "                        \"items\": {\n",
+    "                            \"type\": \"object\"\n",
+    "                        }\n",
+    "                    }\n",
+    "                ]\n",
+    "            },\n",
+    "            \"tool_definitions\": {\n",
+    "                \"anyOf\": [\n",
+    "                    {\"type\": \"string\"},\n",
+    "                    {\"type\": \"object\"},\n",
+    "                    {\n",
+    "                        \"type\": \"array\",\n",
+    "                        \"items\": {\n",
+    "                            \"type\": \"object\"\n",
+    "                        }\n",
+    "                    }\n",
+    "                ]\n",
+    "            }\n",
+    "        },\n",
+    "        \"required\": [\"response\"]\n",
+    "    },\n",
+    "    \"include_sample_schema\": True\n",
+    "}\n",
+    "\n",
+    "# Data mapping (maps evaluation inputs to evaluator parameters)\n",
+    "data_mapping = {\n",
+    "    \"context\": \"{{item.context}}\",\n",
+    "    \"query\": \"{{item.query}}\",\n",
+    "    \"response\": \"{{item.response}}\",\n",
+    "    \"tool_definitions\": \"{{item.tool_definitions}}\"\n",
+    "}\n",
+    "\n",
+    "# Initialization parameters for the evaluator\n",
+    "initialization_parameters = {\n",
+    "    \"deployment_name\": deployment_name\n",
+    "}\n",
+    "\n",
+    "# Initialize the evaluation_contents list - we'll append all test cases here\n",
+    "evaluation_contents = []"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "18b0a169",
+   "metadata": {},
+   "source": [
+    "## Sample Evaluations"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "b73b1ccd",
+   "metadata": {},
+   "source": [
+    "### Response and Context as Strings (str)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "4e5fa898",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Test Case 1: Well-grounded response using weather context\n",
+    "context = \"\"\"Current weather data for Seattle shows rainy conditions with a temperature of 14°C. \n",
+    "The forecast indicates overcast skies with light precipitation typical for Pacific Northwest weather. \n",
+    "Humidity is at 85% with winds from the southwest at 12 mph. Visibility is reduced to 8 miles due to rain.\"\"\"\n",
+    "\n",
+    "response = \"The current weather in Seattle is rainy with a temperature of 14°C. It's typical Pacific Northwest weather for this time of year with overcast skies and light precipitation.\"\n",
+    "\n",
+    "# Append to evaluation_contents\n",
+    "evaluation_contents.append(\n",
+    "    SourceFileContentContent(\n",
+    "        item={\n",
+    "            \"query\": None,\n",
+    "            \"tool_definitions\": None,\n",
+    "            \"response\": response,\n",
+    "            \"context\": context\n",
+    "        }\n",
+    "    )\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "0c03f745",
+   "metadata": {},
+   "source": [
+    "### Example of Ungrounded Response"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "6de6a51e",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Test Case 2: Ungrounded response with unsupported claims\n",
+    "context = \"\"\"Current weather data for Seattle shows rainy conditions with a temperature of 14°C. \n",
+    "The forecast indicates overcast skies with light precipitation typical for Pacific Northwest weather.\"\"\"\n",
+    "\n",
+    "response = \"The current weather in Seattle is rainy with a temperature of 14°C. The city is experiencing its wettest month in 50 years, and the mayor has declared a weather emergency due to flooding concerns.\"\n",
+    "\n",
+    "# Append to evaluation_contents\n",
+    "evaluation_contents.append(\n",
+    "    SourceFileContentContent(\n",
+    "        item={\n",
+    "            \"query\": None,\n",
+    "            \"tool_definitions\": None,\n",
+    "            \"response\": response,\n",
+    "            \"context\": context\n",
+    "        }\n",
+    "    )\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "9052eb13",
+   "metadata": {},
+   "source": [
+    "### Query, Response, and Context as Strings (str)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "c77316c4",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Test Case 3: Query, response, and context evaluation\n",
+    "query = \"How is the weather in Seattle?\"\n",
+    "\n",
+    "context = \"\"\"Weather report for Seattle, Washington: Currently experiencing rainy weather with temperature at 14°C. \n",
+    "Overcast conditions with light rain are expected to continue. The current conditions are typical for the Pacific Northwest region \n",
+    "during this season. Wind speed is moderate at 12 mph from southwest direction.\"\"\"\n",
+    "\n",
+    "response = \"The weather in Seattle is rainy with a temperature of 14°C. These are typical Pacific Northwest conditions with overcast skies.\"\n",
+    "\n",
+    "# Append to evaluation_contents\n",
+    "evaluation_contents.append(\n",
+    "    SourceFileContentContent(\n",
+    "        item={\n",
+    "            \"tool_definitions\": None,\n",
+    "            \"query\": query,\n",
+    "            \"response\": response,\n",
+    "            \"context\": context\n",
+    "        }\n",
+    "    )\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "fd0a420f",
+   "metadata": {},
+   "source": [
+    "### Query as String (str), Response as List[dict], Tool Definitions as List[dict]"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "72b922ee",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Test Case 4: Agent response with file_search tool call\n",
+    "query = \"Can you get me the current weather information for Seattle?\"\n",
+    "\n",
+    "# Agent response with file_search tool call\n",
+    "agent_response = [\n",
+    "    {\n",
+    "        \"role\": \"assistant\",\n",
+    "        \"content\": [\n",
+    "            {\n",
+    "                \"type\": \"tool_call\",\n",
+    "                \"tool_call_id\": \"call_filesearch_weather_123\",\n",
+    "                \"name\": \"file_search\",\n",
+    "                \"arguments\": {\"query\": \"current weather Seattle temperature conditions\"}\n",
+    "            }\n",
+    "        ]\n",
+    "    },\n",
+    "    {\n",
+    "        \"role\": \"tool\",\n",
+    "        \"tool_call_id\": \"call_filesearch_weather_123\", \n",
+    "        \"content\": [\n",
+    "            {\n",
+    "                \"type\": \"tool_result\",\n",
+    "                \"tool_result\": {\n",
+    "                    \"content\": \"Seattle weather report: Currently rainy with temperature of 14°C. Overcast skies with light precipitation. Typical Pacific Northwest weather with 85% humidity and southwest winds at 12 mph.\"\n",
+    "                }\n",
+    "            }\n",
+    "        ]\n",
+    "    },\n",
+    "    {\n",
+    "        \"role\": \"assistant\",\n",
+    "        \"content\": \"Based on the weather data, Seattle is currently experiencing rainy weather with a temperature of 14°C. The conditions include overcast skies and light precipitation, which is typical for the Pacific Northwest.\"\n",
+    "    }\n",
+    "]\n",
+    "\n",
+    "tool_definitions = [\n",
+    "    {\n",
+    "        \"name\": \"file_search\",\n",
+    "        \"description\": \"Search through uploaded files to find relevant information\",\n",
+    "        \"parameters\": {\n",
+    "            \"type\": \"object\", \n",
+    "            \"properties\": {\n",
+    "                \"query\": {\"type\": \"string\", \"description\": \"Search query\"}\n",
+    "            }\n",
+    "        }\n",
+    "    }\n",
+    "]\n",
+    "\n",
+    "# Append to evaluation_contents\n",
+    "evaluation_contents.append(\n",
+    "    SourceFileContentContent(\n",
+    "        item={\n",
+    "            \"context\": None,\n",
+    "            \"query\": query,\n",
+    "            \"response\": agent_response,\n",
+    "            \"tool_definitions\": tool_definitions\n",
+    "        }\n",
+    "    )\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "6dc31ae0",
+   "metadata": {},
+   "source": [
+    "### Conversation as Dict with Context and Messages"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "0f37ae51",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Test Case 5: Multi-turn conversation evaluation\n",
+    "conversation_context = \"Weather data shows Seattle currently has rainy conditions at 14°C with overcast skies. London shows cloudy weather at 8°C with partly cloudy conditions. Both cities are experiencing typical seasonal weather patterns.\"\n",
+    "\n",
+    "conversation_messages = [\n",
+    "    {\n",
+    "        \"role\": \"user\",\n",
+    "        \"content\": \"Can you check the weather in Seattle for me?\"\n",
+    "    },\n",
+    "    {\n",
+    "        \"role\": \"assistant\",\n",
+    "        \"content\": \"According to the current weather data, Seattle is experiencing rainy conditions with a temperature of 14°C and overcast skies.\"\n",
+    "    },\n",
+    "    {\n",
+    "        \"role\": \"user\",\n",
+    "        \"content\": \"How does that compare to London?\"\n",
+    "    },\n",
+    "    {\n",
+    "        \"role\": \"assistant\",\n",
+    "        \"content\": \"London is currently cloudier but drier than Seattle, with a temperature of 8°C and partly cloudy conditions. Seattle is warmer but rainier at 14°C.\"\n",
+    "    },\n",
+    "    {\n",
+    "        \"role\": \"user\", \n",
+    "        \"content\": \"Can you email me a summary of both cities' weather?\"\n",
+    "    },\n",
+    "    {\n",
+    "        \"role\": \"assistant\",\n",
+    "        \"content\": \"I can provide you with a weather summary: Seattle has rainy weather at 14°C with overcast skies, while London has partly cloudy conditions at 8°C. However, I would need email access to send this information to you.\"\n",
+    "    }\n",
+    "]\n",
+    "\n",
+    "# Append to evaluation_contents\n",
+    "evaluation_contents.append(\n",
+    "    SourceFileContentContent(\n",
+    "        item={\n",
+    "            \"query\": None,\n",
+    "            \"tool_definitions\": None,\n",
+    "            \"context\": conversation_context,\n",
+    "            \"response\": conversation_messages\n",
+    "        }\n",
+    "    )\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "3adcae5d",
+   "metadata": {},
+   "source": [
+    "### Run Evaluation on All Test Cases\n",
+    "\n",
+    "Now that we've defined all test cases, let's run the evaluation once on all of them."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "e543aa00",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "results = run_evaluator(\n",
+    "    evaluator_name=\"groundedness\",\n",
+    "    evaluation_contents=evaluation_contents,\n",
+    "    data_source_config=data_source_config,\n",
+    "    initialization_parameters=initialization_parameters,\n",
+    "    data_mapping=data_mapping\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "2f4d7c8e",
+   "metadata": {},
+   "source": [
+    "### Display Results\n",
+    "\n",
+    "View the evaluation results for each test case."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "87aadc9a",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "pprint(results)"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "samples",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.19"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
diff --git a/scenarios/evaluate/Supported_Evaluation_Metrics/Agent_Evaluation/AI_Judge_Evaluator_Intent_Resolution.ipynb b/scenarios/evaluate/Supported_Evaluation_Metrics/Agent_Evaluation/AI_Judge_Evaluator_Intent_Resolution.ipynb
index ab69e4fc..ab70531d 100644
--- a/scenarios/evaluate/Supported_Evaluation_Metrics/Agent_Evaluation/AI_Judge_Evaluator_Intent_Resolution.ipynb
+++ b/scenarios/evaluate/Supported_Evaluation_Metrics/Agent_Evaluation/AI_Judge_Evaluator_Intent_Resolution.ipynb
@@ -16,18 +16,15 @@
     "You should expect to spend about 20 minutes running this notebook. \n",
     "\n",
     "## Before you begin\n",
-    "For quality evaluation, you need to deploy a `gpt` model supporting JSON mode. We recommend a model `gpt-4o` or `gpt-4o-mini` for their strong reasoning capabilities.    \n",
+    "For quality evaluation, you need to deploy a `gpt` model supporting JSON mode. We recommend using `gpt-4o` or `gpt-4.1`.   \n",
     "\n",
     "### Prerequisite\n",
     "```bash\n",
-    "pip install azure-ai-projects azure-identity azure-ai-evaluation\n",
+    "pip install azure-ai-projects azure-identity openai\n",
     "```\n",
     "Set these environment variables with your own values:\n",
-    "1) **AZURE_AI_PROJECT** - The project connection string, as found in the overview page of your Azure AI Foundry project.\n",
-    "2) **MODEL_DEPLOYMENT_NAME** - The deployment name of the model for this AI-assisted evaluator, as found under the \"Name\" column in the \"Models + endpoints\" tab in your Azure AI Foundry project.\n",
-    "3) **AZURE_OPENAI_ENDPOINT** - Azure Open AI Endpoint to be used for evaluation.\n",
-    "4) **AZURE_OPENAI_API_KEY** - Azure Open AI Key to be used for evaluation.\n",
-    "5) **AZURE_OPENAI_API_VERSION** - Azure Open AI Api version to be used for evaluation."
+    "1) **AZURE_AI_PROJECT_ENDPOINT** - Your Azure AI project endpoint in format: `https://<account_name>.services.ai.azure.com/api/projects/<project_name>`\n",
+    "2) **AZURE_AI_MODEL_DEPLOYMENT_NAME** - The deployment name of the model for this AI-assisted evaluator (e.g., gpt-4o-mini)"
    ]
   },
   {
@@ -66,18 +63,72 @@
    "outputs": [],
    "source": [
     "import os\n",
-    "from azure.ai.evaluation import AzureOpenAIModelConfiguration\n",
-    "from azure.ai.evaluation import IntentResolutionEvaluator\n",
+    "from openai.types.evals.create_eval_jsonl_run_data_source_param import SourceFileContentContent\n",
     "from pprint import pprint\n",
+    "from agent_utils import run_evaluator\n",
     "\n",
-    "model_config = AzureOpenAIModelConfiguration(\n",
-    "    azure_endpoint=os.environ[\"AZURE_OPENAI_ENDPOINT\"],\n",
-    "    api_key=os.environ[\"AZURE_OPENAI_API_KEY\"],\n",
-    "    api_version=os.environ[\"AZURE_OPENAI_API_VERSION\"],\n",
-    "    azure_deployment=os.environ[\"MODEL_DEPLOYMENT_NAME\"],\n",
-    ")\n",
+    "# Get environment variables\n",
+    "deployment_name = os.environ[\"AZURE_AI_MODEL_DEPLOYMENT_NAME\"]\n",
     "\n",
-    "intent_resolution_evaluator = IntentResolutionEvaluator(model_config)"
+    "# Data source configuration (defines the schema for evaluation inputs)\n",
+    "data_source_config = {\n",
+    "    \"type\": \"custom\",\n",
+    "    \"item_schema\": {\n",
+    "        \"type\": \"object\",\n",
+    "        \"properties\": {\n",
+    "            \"query\": {\n",
+    "                \"anyOf\": [\n",
+    "                    {\"type\": \"string\"},\n",
+    "                    {\n",
+    "                        \"type\": \"array\",\n",
+    "                        \"items\": {\n",
+    "                            \"type\": \"object\"\n",
+    "                        }\n",
+    "                    }\n",
+    "                ]\n",
+    "            },\n",
+    "            \"response\": {\n",
+    "                \"anyOf\": [\n",
+    "                    {\"type\": \"string\"},\n",
+    "                    {\n",
+    "                        \"type\": \"array\",\n",
+    "                        \"items\": {\n",
+    "                            \"type\": \"object\"\n",
+    "                        }\n",
+    "                    }\n",
+    "                ]\n",
+    "            },\n",
+    "            \"tool_definitions\": {\n",
+    "                \"anyOf\": [\n",
+    "                    {\"type\": \"object\"},\n",
+    "                    {\n",
+    "                        \"type\": \"array\",\n",
+    "                        \"items\": {\n",
+    "                            \"type\": \"object\"\n",
+    "                        }\n",
+    "                    }\n",
+    "                ]\n",
+    "            }\n",
+    "        },\n",
+    "        \"required\": [\"query\", \"response\"]\n",
+    "    },\n",
+    "    \"include_sample_schema\": True\n",
+    "}\n",
+    "\n",
+    "# Data mapping (maps evaluation inputs to evaluator parameters)\n",
+    "data_mapping = {\n",
+    "    \"query\": \"{{item.query}}\",\n",
+    "    \"response\": \"{{item.response}}\",\n",
+    "    \"tool_definitions\": \"{{item.tool_definitions}}\"\n",
+    "}\n",
+    "\n",
+    "# Initialization parameters for the evaluator\n",
+    "initialization_parameters = {\n",
+    "    \"deployment_name\": deployment_name\n",
+    "}\n",
+    "\n",
+    "# Initialize the evaluation_contents list - we'll append all test cases here\n",
+    "evaluation_contents = []"
    ]
   },
   {
@@ -100,12 +151,20 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "# Success example. Intent is identified and understood and the response correctly resolves user intent\n",
-    "result = intent_resolution_evaluator(\n",
-    "    query=\"What are the opening hours of the Eiffel Tower?\",\n",
-    "    response=\"Opening hours of the Eiffel Tower are 9:00 AM to 11:00 PM.\",\n",
-    ")\n",
-    "pprint(result)"
+    "# Test Case 1: Successful intent resolution (should score high)\n",
+    "query1 = \"What are the opening hours of the Eiffel Tower?\"\n",
+    "response1 = \"Opening hours of the Eiffel Tower are 9:00 AM to 11:00 PM.\"\n",
+    "\n",
+    "# Append to evaluation_contents\n",
+    "evaluation_contents.append(\n",
+    "    SourceFileContentContent(\n",
+    "        item={\n",
+    "            \"query\": query1,\n",
+    "            \"response\": response1,\n",
+    "            \"tool_definitions\": None\n",
+    "        }\n",
+    "    )\n",
+    ")"
    ]
   },
   {
@@ -114,12 +173,58 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "# Failure example. Even though intent is correctly identified, the response does not resolve the user intent\n",
-    "result = intent_resolution_evaluator(\n",
-    "    query=\"What is the opening hours of the Eiffel Tower?\",\n",
-    "    response=\"Please check the official website for the up-to-date information on Eiffel Tower opening hours.\",\n",
-    ")\n",
-    "pprint(result)"
+    "# Test Case 2: Intent identified but not resolved (should score low)\n",
+    "query2 = \"What is the opening hours of the Eiffel Tower?\"\n",
+    "response2 = \"Please check the official website for the up-to-date information on Eiffel Tower opening hours.\"\n",
+    "\n",
+    "# Append to evaluation_contents\n",
+    "evaluation_contents.append(\n",
+    "    SourceFileContentContent(\n",
+    "        item={\n",
+    "            \"query\": query2,\n",
+    "            \"response\": response2,\n",
+    "            \"tool_definitions\": None\n",
+    "        }\n",
+    "    )\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "#### Query and Response as String with Tool Definition as Single Dict"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Test Case 3: Order status query with tool definition (should score high)\n",
+    "query3 = \"What's the status of order #789?\"\n",
+    "response3 = \"Order #789 has been shipped and will arrive on March 18, 2025.\"\n",
+    "\n",
+    "tool_definition_dict = [{\n",
+    "    \"name\": \"get_order\",\n",
+    "    \"description\": \"Get the details of a specific order.\",\n",
+    "    \"parameters\": {\n",
+    "        \"type\": \"object\",\n",
+    "        \"properties\": {\"order_id\": {\"type\": \"string\", \"description\": \"The order ID to get the details for.\"}},\n",
+    "    },\n",
+    "}]\n",
+    "\n",
+    "# Append to evaluation_contents\n",
+    "evaluation_contents.append(\n",
+    "    SourceFileContentContent(\n",
+    "        item={\n",
+    "            \"query\": query3,\n",
+    "            \"response\": response3,\n",
+    "            \"tool_definitions\": tool_definition_dict\n",
+    "        }\n",
+    "    )\n",
+    ")"
    ]
   },
   {
@@ -135,7 +240,8 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "query = [\n",
+    "# Test Case 4: Complex conversation with multiple orders (should score high)\n",
+    "query4 = [\n",
     "    {\"role\": \"system\", \"content\": \"You are a friendly and helpful customer service agent.\"},\n",
     "    {\n",
     "        \"createdAt\": \"2025-03-14T06:14:20Z\",\n",
@@ -149,7 +255,7 @@
     "    },\n",
     "]\n",
     "\n",
-    "response = [\n",
+    "response4 = [\n",
     "    {\n",
     "        \"createdAt\": \"2025-03-14T06:14:30Z\",\n",
     "        \"run_id\": \"0\",\n",
@@ -243,9 +349,8 @@
     "    },\n",
     "]\n",
     "\n",
-    "# please note that the tool definitions are not strictly required, and that some of the tools below are not used in the example above and that is ok.\n",
-    "# if context length is a concern you can remove the unused tool definitions or even the tool definitions altogether as the impact to the intent resolution evaluation is usual minimal.\n",
-    "tool_definitions = [\n",
+    "# Tool definitions for customer service operations\n",
+    "tool_definitions4 = [\n",
     "    {\n",
     "        \"name\": \"get_orders\",\n",
     "        \"description\": \"Get the list of orders for a given account number.\",\n",
@@ -285,22 +390,25 @@
     "    },\n",
     "]\n",
     "\n",
-    "result = intent_resolution_evaluator(\n",
-    "    query=query,\n",
-    "    response=response,\n",
-    "    tool_definitions=tool_definitions,\n",
-    ")\n",
-    "pprint(result)"
+    "# Append to evaluation_contents\n",
+    "evaluation_contents.append(\n",
+    "    SourceFileContentContent(\n",
+    "        item={\n",
+    "            \"query\": query4,\n",
+    "            \"response\": response4,\n",
+    "            \"tool_definitions\": tool_definitions4\n",
+    "        }\n",
+    "    )\n",
+    ")"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## Batch evaluate and visualize results on Azure AI Foundry\n",
-    "Batch evaluate to leverage asynchronous evaluation on a dataset. \n",
+    "### Run Evaluation on All Test Cases\n",
     "\n",
-    "Optionally, you can go to AI Foundry URL for rich Azure AI Foundry data visualization. You can inspect the evaluation scores and reasoning to quickly identify bugs and issues of your agent to fix and improve. Make sure to authenticate to Azure using `az login` in your terminal before running this cell.\n"
+    "Now that we've defined all test cases, let's run the evaluation once on all of them."
    ]
   },
   {
@@ -309,27 +417,37 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from azure.ai.evaluation import evaluate\n",
-    "\n",
-    "# This sample files contains the evaluation data in JSONL format. Where each line is a run from agent.\n",
-    "# This was saved using agent thread and converter.\n",
-    "file_name = \"evaluation_data.jsonl\"\n",
+    "results = run_evaluator(\n",
+    "    evaluator_name=\"intent_resolution\",\n",
+    "    evaluation_contents=evaluation_contents,\n",
+    "    data_source_config=data_source_config,\n",
+    "    initialization_parameters=initialization_parameters,\n",
+    "    data_mapping=data_mapping\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Display Results\n",
     "\n",
-    "response = evaluate(\n",
-    "    data=file_name,\n",
-    "    evaluation_name=\"Intent Resolution Evaluation\",\n",
-    "    evaluators={\n",
-    "        \"intent_resolution\": intent_resolution_evaluator,\n",
-    "    },\n",
-    "    azure_ai_project=os.environ[\"AZURE_AI_PROJECT\"],\n",
-    ")\n",
-    "pprint(f'AI Foundary URL: {response.get(\"studio_url\")}')"
+    "View the evaluation results for each test case."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "pprint(results)"
    ]
   }
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "azureai-sample",
+   "display_name": "samples",
    "language": "python",
    "name": "python3"
   },
@@ -342,7 +460,8 @@
    "mimetype": "text/x-python",
    "name": "python",
    "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3"
+   "pygments_lexer": "ipython3",
+   "version": "3.10.19"
   }
  },
  "nbformat": 4,
diff --git a/scenarios/evaluate/Supported_Evaluation_Metrics/Agent_Evaluation/AI_Judge_Evaluator_Relevance.ipynb b/scenarios/evaluate/Supported_Evaluation_Metrics/Agent_Evaluation/AI_Judge_Evaluator_Relevance.ipynb
new file mode 100644
index 00000000..3672638f
--- /dev/null
+++ b/scenarios/evaluate/Supported_Evaluation_Metrics/Agent_Evaluation/AI_Judge_Evaluator_Relevance.ipynb
@@ -0,0 +1,251 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "bb0675a7",
+   "metadata": {},
+   "source": [
+    "# Relevance Evaluator\n",
+    "\n",
+    "## Objective\n",
+    "This sample demonstrates how to use the Relevance evaluator to assess the relevance of AI-generated responses. The evaluator supports the following input format:\n",
+    "- Simple query and response pairs\n",
+    "\n",
+    "## Time\n",
+    "\n",
+    "You should expect to spend about 15 minutes running this notebook.\n",
+    "\n",
+    "## Before you begin\n",
+    "For quality evaluation, you need to deploy a `gpt` model supporting JSON mode. We recommend using `gpt-4.1`. It is not recommended to use `gpt-4.1-nano`.\n",
+    "\n",
+    "### Prerequisite\n",
+    "```bash\n",
+    "pip install azure-ai-projects azure-identity openai\n",
+    "```\n",
+    "Set these environment variables with your own values:\n",
+    "1) **AZURE_AI_PROJECT_ENDPOINT** - Your Azure AI project endpoint in format: `https://<account_name>.services.ai.azure.com/api/projects/<project_name>`\n",
+    "2) **AZURE_AI_MODEL_DEPLOYMENT_NAME** - The deployment name of the model for this AI-assisted evaluator (e.g., gpt-4o-mini)\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "1a21aa7e",
+   "metadata": {},
+   "source": [
+    "## What is Relevance?\n",
+    "\n",
+    "The Relevance evaluator assesses the ability of AI responses to capture the key points and address the user's query appropriately. It measures how well the response relates to and answers the specific question or request.\n",
+    "\n",
+    "**Scoring:** Relevance scores range from 1 to 5, with:\n",
+    "- **1**: Completely irrelevant to the query\n",
+    "- **2**: Partially relevant but misses key points\n",
+    "- **3**: Moderately relevant with some key points addressed\n",
+    "- **4**: Highly relevant with most key points covered\n",
+    "- **5**: Perfectly relevant, fully addressing the query\n",
+    "\n",
+    "High relevance scores indicate the AI system's understanding of the input and capability to produce contextually appropriate outputs."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "f0dd0ebf",
+   "metadata": {},
+   "source": [
+    "## Relevance Evaluator Input Requirements\n",
+    "\n",
+    "The Relevance evaluator supports the following input format:\n",
+    "\n",
+    "1. **Simple Query-Response Evaluation:**\n",
+    "   - `query`: The user's question or request (str)\n",
+    "   - `response`: The AI's response to evaluate (str)\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "519631da",
+   "metadata": {},
+   "source": [
+    "### Initialize Relevance Evaluator"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "3484e790",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import os\n",
+    "from openai.types.evals.create_eval_jsonl_run_data_source_param import SourceFileContentContent\n",
+    "from pprint import pprint\n",
+    "from agent_utils import run_evaluator\n",
+    "\n",
+    "# Get environment variables\n",
+    "deployment_name = os.environ[\"AZURE_AI_MODEL_DEPLOYMENT_NAME\"]\n",
+    "\n",
+    "# Data source configuration (defines the schema for evaluation inputs)\n",
+    "data_source_config = {\n",
+    "    \"type\": \"custom\",\n",
+    "    \"item_schema\": {\n",
+    "        \"type\": \"object\",\n",
+    "        \"properties\": {\n",
+    "            \"query\": {\"anyOf\": [{\"type\": \"string\"}, {\"type\": \"array\", \"items\": {\"type\": \"object\"}}]},\n",
+    "            \"response\": {\"anyOf\": [{\"type\": \"string\"}, {\"type\": \"array\", \"items\": {\"type\": \"object\"}}]},\n",
+    "        },\n",
+    "        \"required\": [\"query\", \"response\"],\n",
+    "    },\n",
+    "    \"include_sample_schema\": True,\n",
+    "}\n",
+    "\n",
+    "# Data mapping (maps evaluation inputs to evaluator parameters)\n",
+    "data_mapping = {\n",
+    "    \"query\": \"{{item.query}}\",\n",
+    "    \"response\": \"{{item.response}}\"\n",
+    "}\n",
+    "\n",
+    "# Initialization parameters for the evaluator\n",
+    "initialization_parameters = {\n",
+    "    \"deployment_name\": deployment_name\n",
+    "}\n",
+    "\n",
+    "# Initialize the evaluation_contents list - we'll append all test cases here\n",
+    "evaluation_contents = []"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "4f739700",
+   "metadata": {},
+   "source": [
+    "## Sample Evaluations"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "83d8d0ec",
+   "metadata": {},
+   "source": [
+    "### Query and Response as Strings (str)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "43eaf25d",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Test Case 1: Highly relevant weather response (should score high)\n",
+    "query1 = \"How is the weather in Seattle?\"\n",
+    "response1 = \"The current weather in Seattle is rainy with a temperature of 14°C. It's typical Pacific Northwest weather for this time of year with overcast skies and light precipitation.\"\n",
+    "\n",
+    "# Append to evaluation_contents\n",
+    "evaluation_contents.append(\n",
+    "    SourceFileContentContent(\n",
+    "        item={\n",
+    "            \"query\": query1,\n",
+    "            \"response\": response1\n",
+    "        }\n",
+    "    )\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "89266048",
+   "metadata": {},
+   "source": [
+    "### Example of Irrelevant Response"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "69761825",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Test Case 2: Completely irrelevant response (should score low)\n",
+    "query2 = \"What are the symptoms of flu?\"\n",
+    "response2 = \"The stock market had significant fluctuations today, with technology stocks leading the decline.\"\n",
+    "\n",
+    "# Append to evaluation_contents\n",
+    "evaluation_contents.append(\n",
+    "    SourceFileContentContent(\n",
+    "        item={\n",
+    "            \"query\": query2,\n",
+    "            \"response\": response2\n",
+    "        }\n",
+    "    )\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "56752df0",
+   "metadata": {},
+   "source": [
+    "### Run Evaluation on All Test Cases\n",
+    "\n",
+    "Now that we've defined all test cases, let's run the evaluation once on all of them."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "d7519cb0",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "results = run_evaluator(\n",
+    "    evaluator_name=\"relevance\",\n",
+    "    evaluation_contents=evaluation_contents,\n",
+    "    data_source_config=data_source_config,\n",
+    "    initialization_parameters=initialization_parameters,\n",
+    "    data_mapping=data_mapping\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "1b97b1e7",
+   "metadata": {},
+   "source": [
+    "### Display Results\n",
+    "\n",
+    "View the evaluation results for each test case."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "f7343ff8",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "pprint(results)"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "samples",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.19"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
diff --git a/scenarios/evaluate/Supported_Evaluation_Metrics/Agent_Evaluation/AI_Judge_Evaluator_Response_Completeness.ipynb b/scenarios/evaluate/Supported_Evaluation_Metrics/Agent_Evaluation/AI_Judge_Evaluator_Response_Completeness.ipynb
new file mode 100644
index 00000000..685663ce
--- /dev/null
+++ b/scenarios/evaluate/Supported_Evaluation_Metrics/Agent_Evaluation/AI_Judge_Evaluator_Response_Completeness.ipynb
@@ -0,0 +1,230 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Response Completeness Evaluator"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Getting Started\n",
+    "\n",
+    "This sample demonstrates how to use Response Completeness Evaluator on agent's response when ground truth is provided. This evaluator is helpful when you have ground truth to assess the quality of the agent's final response. \n",
+    "\n",
+    "## Time\n",
+    "\n",
+    "You should expect to spend about 20 minutes running this notebook. \n",
+    "\n",
+    "## Before you begin\n",
+    "For quality evaluation, you need to deploy a `gpt` model supporting JSON mode. We recommend using `gpt-4o` or `gpt-4.1`.  \n",
+    "\n",
+    "### Prerequisite\n",
+    "```bash\n",
+    "pip install azure-ai-projects azure-identity openai\n",
+    "```\n",
+    "Set these environment variables with your own values:\n",
+    "1) **AZURE_AI_PROJECT_ENDPOINT** - Your Azure AI project endpoint in format: `https://<account_name>.services.ai.azure.com/api/projects/<project_name>`\n",
+    "2) **AZURE_AI_MODEL_DEPLOYMENT_NAME** - The deployment name of the model for this AI-assisted evaluator (e.g., gpt-4o-mini)\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "The Response Completeness evaluator assesses the quality of an agent response by examining how well it aligns with the provided ground truth. The evaluation is based on the following scoring system:\n",
+    "\n",
+    "<pre>\n",
+    "Score 1: Fully incomplete: The response misses all necessary and relevant information compared to the ground truth.\n",
+    "Score 2: Barely complete: The response contains only a small percentage of the necessary information.\n",
+    "Score 3: Moderately complete: The response includes about half of the necessary information.\n",
+    "Score 4: Mostly complete: The response contains most of the necessary information, with only minor omissions.\n",
+    "Score 5: Fully complete: The response perfectly matches all necessary and relevant information from the ground truth.\n",
+    "</pre>\n",
+    "\n",
+    "The evaluation requires the following inputs:\n",
+    "\n",
+    "- Response: The response to be evaluated. (string)\n",
+    "- Ground Truth: The correct and complete information against which the response is compared. (string)\n",
+    "\n",
+    "The evaluator uses these inputs to determine the completeness score, ensuring that the response meaningfully addresses the query while adhering to the provided definitions and data."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Initialize Completeness Evaluator\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import os\n",
+    "from openai.types.evals.create_eval_jsonl_run_data_source_param import SourceFileContentContent\n",
+    "from pprint import pprint\n",
+    "from agent_utils import run_evaluator\n",
+    "\n",
+    "# Get environment variables\n",
+    "deployment_name = os.environ[\"AZURE_AI_MODEL_DEPLOYMENT_NAME\"]\n",
+    "\n",
+    "# Data source configuration (defines the schema for evaluation inputs)\n",
+    "data_source_config = {\n",
+    "    \"type\": \"custom\",\n",
+    "    \"item_schema\": {\n",
+    "        \"type\": \"object\",\n",
+    "        \"properties\": {\n",
+    "            \"ground_truth\": {\n",
+    "                \"type\": \"string\"\n",
+    "            },\n",
+    "            \"response\": {\n",
+    "                \"type\": \"string\"\n",
+    "            }\n",
+    "        },\n",
+    "        \"required\": [\"ground_truth\", \"response\"]\n",
+    "    },\n",
+    "    \"include_sample_schema\": True\n",
+    "}\n",
+    "\n",
+    "# Data mapping (maps evaluation inputs to evaluator parameters)\n",
+    "data_mapping = {\n",
+    "    \"ground_truth\": \"{{item.ground_truth}}\",\n",
+    "    \"response\": \"{{item.response}}\"\n",
+    "}\n",
+    "\n",
+    "# Initialization parameters for the evaluator\n",
+    "initialization_parameters = {\n",
+    "    \"deployment_name\": deployment_name\n",
+    "}\n",
+    "\n",
+    "# Initialize the evaluation_contents list - we'll append all test cases here\n",
+    "evaluation_contents = []"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Samples"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "#### Response and Ground Truth as Strings (str)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Test Case 1: Incomplete response (should score low)\n",
+    "response1 = \"The capital of Japan\"\n",
+    "ground_truth1 = \"The capital of Japan is Tokyo.\"\n",
+    "\n",
+    "# Append to evaluation_contents\n",
+    "evaluation_contents.append(\n",
+    "    SourceFileContentContent(\n",
+    "        item={\n",
+    "            \"response\": response1,\n",
+    "            \"ground_truth\": ground_truth1\n",
+    "        }\n",
+    "    )\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Test Case 2: Complete response (should score high)\n",
+    "response2 = \"The capital of Japan is Tokyo.\"\n",
+    "ground_truth2 = \"The capital of Japan is Tokyo.\"\n",
+    "\n",
+    "# Append to evaluation_contents\n",
+    "evaluation_contents.append(\n",
+    "    SourceFileContentContent(\n",
+    "        item={\n",
+    "            \"response\": response2,\n",
+    "            \"ground_truth\": ground_truth2\n",
+    "        }\n",
+    "    )\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Run Evaluation on All Test Cases\n",
+    "\n",
+    "Now that we've defined all test cases, let's run the evaluation once on all of them."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "results = run_evaluator(\n",
+    "    evaluator_name=\"response_completeness\",\n",
+    "    evaluation_contents=evaluation_contents,\n",
+    "    data_source_config=data_source_config,\n",
+    "    initialization_parameters=initialization_parameters,\n",
+    "    data_mapping=data_mapping\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Display Results\n",
+    "\n",
+    "View the evaluation results for each test case."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "pprint(results)"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "samples",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.19"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
diff --git a/scenarios/evaluate/Supported_Evaluation_Metrics/Agent_Evaluation/AI_Judge_Evaluator_Task_Adherence.ipynb b/scenarios/evaluate/Supported_Evaluation_Metrics/Agent_Evaluation/AI_Judge_Evaluator_Task_Adherence.ipynb
index 075676b8..9bdb691e 100644
--- a/scenarios/evaluate/Supported_Evaluation_Metrics/Agent_Evaluation/AI_Judge_Evaluator_Task_Adherence.ipynb
+++ b/scenarios/evaluate/Supported_Evaluation_Metrics/Agent_Evaluation/AI_Judge_Evaluator_Task_Adherence.ipynb
@@ -6,8 +6,6 @@
    "source": [
     "# Task Adherence Evaluator\n",
     "\n",
-    "# Intent Resolution Evaluator\n",
-    "\n",
     "## Objective\n",
     "This sample demonstrates to how to use task adherence evaluator on agent data. The supported input formats include:\n",
     "- simple data such as strings;\n",
@@ -18,37 +16,15 @@
     "You should expect to spend about 10 minutes running this notebook. \n",
     "\n",
     "## Before you begin\n",
-    "For quality evaluation, you need to deploy a `gpt` model supporting JSON mode. We recommend a model `gpt-4o` or `gpt-4o-mini` for their strong reasoning capabilities.    \n",
+    "For quality evaluation, you need to deploy a `gpt` model supporting JSON mode. We recommend using `gpt-4o` or `gpt-4.1`.  \n",
     "\n",
     "### Prerequisite\n",
     "```bash\n",
-    "pip install azure-ai-projects azure-identity azure-ai-evaluation\n",
+    "pip install azure-ai-projects azure-identity openai\n",
     "```\n",
     "Set these environment variables with your own values:\n",
-    "1) **AZURE_AI_PROJECT** - The project url, as found in the overview page of your Azure AI Foundry project.\n",
-    "2) **MODEL_DEPLOYMENT_NAME** - The deployment name of the model for this AI-assisted evaluator, as found under the \"Name\" column in the \"Models + endpoints\" tab in your Azure AI Foundry project.\n",
-    "3) **AZURE_OPENAI_ENDPOINT** - Azure Open AI Endpoint to be used for evaluation.\n",
-    "4) **AZURE_OPENAI_API_KEY** - Azure Open AI Key to be used for evaluation.\n",
-    "5) **AZURE_OPENAI_API_VERSION** - Azure Open AI Api version to be used for evaluation.\n"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "### Getting Started\n",
-    "\n",
-    "This sample demonstrates how to use Task Adherence Evaluator\n",
-    "Before running the sample:\n",
-    "```bash\n",
-    "pip install azure-ai-projects azure-identity azure-ai-evaluation\n",
-    "```\n",
-    "Set these environment variables with your own values:\n",
-    "1) **AZURE_AI_PROJECT** - The project url, as found in the overview page of your Azure AI Foundry project.\n",
-    "2) **MODEL_DEPLOYMENT_NAME** - The deployment name of the AI model, as found under the \"Name\" column in the \"Models + endpoints\" tab in your Azure AI Foundry project.\n",
-    "3) **AZURE_OPENAI_ENDPOINT** - Azure Open AI Endpoint to be used for evaluation.\n",
-    "4) **AZURE_OPENAI_API_KEY** - Azure Open AI Key to be used for evaluation.\n",
-    "5) **AZURE_OPENAI_API_VERSION** - Azure Open AI Api version to be used for evaluation."
+    "1) **AZURE_AI_PROJECT_ENDPOINT** - Your Azure AI project endpoint in format: `https://<account_name>.services.ai.azure.com/api/projects/<project_name>`\n",
+    "2) **AZURE_AI_MODEL_DEPLOYMENT_NAME** - The deployment name of the model for this AI-assisted evaluator (e.g., gpt-4o-mini)"
    ]
   },
   {
@@ -88,16 +64,72 @@
    "outputs": [],
    "source": [
     "import os\n",
-    "from azure.ai.evaluation import TaskAdherenceEvaluator, AzureOpenAIModelConfiguration\n",
+    "from openai.types.evals.create_eval_jsonl_run_data_source_param import SourceFileContentContent\n",
     "from pprint import pprint\n",
+    "from agent_utils import run_evaluator\n",
     "\n",
-    "model_config = AzureOpenAIModelConfiguration(\n",
-    "    azure_endpoint=os.environ[\"AZURE_OPENAI_ENDPOINT\"],\n",
-    "    api_key=os.environ[\"AZURE_OPENAI_API_KEY\"],\n",
-    "    api_version=os.environ[\"AZURE_OPENAI_API_VERSION\"],\n",
-    "    azure_deployment=os.environ[\"MODEL_DEPLOYMENT_NAME\"],\n",
-    ")\n",
-    "task_adherence_evaluator = TaskAdherenceEvaluator(model_config)"
+    "# Get environment variables\n",
+    "deployment_name = os.environ[\"AZURE_AI_MODEL_DEPLOYMENT_NAME\"]\n",
+    "\n",
+    "# Data source configuration (defines the schema for evaluation inputs)\n",
+    "data_source_config = {\n",
+    "    \"type\": \"custom\",\n",
+    "    \"item_schema\": {\n",
+    "        \"type\": \"object\",\n",
+    "        \"properties\": {\n",
+    "            \"query\": {\n",
+    "                \"anyOf\": [\n",
+    "                    {\"type\": \"string\"},\n",
+    "                    {\n",
+    "                        \"type\": \"array\",\n",
+    "                        \"items\": {\n",
+    "                            \"type\": \"object\"\n",
+    "                        }\n",
+    "                    }\n",
+    "                ]\n",
+    "            },\n",
+    "            \"response\": {\n",
+    "                \"anyOf\": [\n",
+    "                    {\"type\": \"string\"},\n",
+    "                    {\n",
+    "                        \"type\": \"array\",\n",
+    "                        \"items\": {\n",
+    "                            \"type\": \"object\"\n",
+    "                        }\n",
+    "                    }\n",
+    "                ]\n",
+    "            },\n",
+    "            \"tool_definitions\": {\n",
+    "                \"anyOf\": [\n",
+    "                    {\"type\": \"object\"},\n",
+    "                    {\n",
+    "                        \"type\": \"array\",\n",
+    "                        \"items\": {\n",
+    "                            \"type\": \"object\"\n",
+    "                        }\n",
+    "                    }\n",
+    "                ]\n",
+    "            }\n",
+    "        },\n",
+    "        \"required\": [\"query\", \"response\"]\n",
+    "    },\n",
+    "    \"include_sample_schema\": True\n",
+    "}\n",
+    "\n",
+    "# Data mapping (maps evaluation inputs to evaluator parameters)\n",
+    "data_mapping = {\n",
+    "    \"query\": \"{{item.query}}\",\n",
+    "    \"response\": \"{{item.response}}\",\n",
+    "    \"tool_definitions\": \"{{item.tool_definitions}}\"\n",
+    "}\n",
+    "\n",
+    "# Initialization parameters for the evaluator\n",
+    "initialization_parameters = {\n",
+    "    \"deployment_name\": deployment_name\n",
+    "}\n",
+    "\n",
+    "# Initialize the evaluation_contents list - we'll append all test cases here\n",
+    "evaluation_contents = []"
    ]
   },
   {
@@ -120,12 +152,20 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "# Failure example, there's only a vague adherence to the task\n",
-    "result = task_adherence_evaluator(\n",
-    "    query=\"What are the best practices for maintaining a healthy rose garden during the summer?\",\n",
-    "    response=\"Make sure to water your roses regularly and trim them occasionally.\",\n",
-    ")\n",
-    "pprint(result)"
+    "# Test Case 1: Vague adherence example (should score low)\n",
+    "query1 = \"What are the best practices for maintaining a healthy rose garden during the summer?\"\n",
+    "response1 = \"Make sure to water your roses regularly and trim them occasionally.\"\n",
+    "\n",
+    "# Append to evaluation_contents\n",
+    "evaluation_contents.append(\n",
+    "    SourceFileContentContent(\n",
+    "        item={\n",
+    "            \"query\": query1,\n",
+    "            \"response\": response1,\n",
+    "            \"tool_definitions\": None,\n",
+    "        }\n",
+    "    )\n",
+    ")"
    ]
   },
   {
@@ -134,12 +174,62 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "# Success example, full adherence to the task\n",
-    "result = task_adherence_evaluator(\n",
-    "    query=\"What are the best practices for maintaining a healthy rose garden during the summer?\",\n",
-    "    response=\"For optimal summer care of your rose garden, start by watering deeply early in the morning to ensure the roots are well-hydrated without encouraging fungal growth. Apply a 2-3 inch layer of organic mulch around the base of the plants to conserve moisture and regulate soil temperature. Fertilize with a balanced rose fertilizer every 4 to 6 weeks to support healthy growth. Prune away any dead or diseased wood to promote good air circulation, and inspect regularly for pests such as aphids or spider mites, treating them promptly with an appropriate organic insecticidal soap. Finally, ensure that your roses receive at least 6 hours of direct sunlight daily for robust flowering.\",\n",
-    ")\n",
-    "pprint(result)"
+    "# Test Case 2: Full adherence example (should score high)\n",
+    "query2 = \"What are the best practices for maintaining a healthy rose garden during the summer?\"\n",
+    "response2 = \"For optimal summer care of your rose garden, start by watering deeply early in the morning to ensure the roots are well-hydrated without encouraging fungal growth. Apply a 2-3 inch layer of organic mulch around the base of the plants to conserve moisture and regulate soil temperature. Fertilize with a balanced rose fertilizer every 4 to 6 weeks to support healthy growth. Prune away any dead or diseased wood to promote good air circulation, and inspect regularly for pests such as aphids or spider mites, treating them promptly with an appropriate organic insecticidal soap. Finally, ensure that your roses receive at least 6 hours of direct sunlight daily for robust flowering.\"\n",
+    "\n",
+    "# Append to evaluation_contents\n",
+    "evaluation_contents.append(\n",
+    "    SourceFileContentContent(\n",
+    "        item={\n",
+    "            \"query\": query2,\n",
+    "            \"response\": response2,\n",
+    "            \"tool_definitions\": None,\n",
+    "        }\n",
+    "    )\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "#### Query and Response as String with Tool Definition as Single Dict"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Test Case 3: Book recommendation with tool definition as single dict\n",
+    "query3 = \"Can you recommend a science fiction book for me?\"\n",
+    "response3 = \"I found 'Dune' by Frank Herbert for you. It's a classic science fiction novel set in a distant future.\"\n",
+    "\n",
+    "tool_definition_dict = [\n",
+    "    {\n",
+    "        \"name\": \"get_book\",\n",
+    "        \"description\": \"Retrieve a book recommendation for a specified genre.\",\n",
+    "        \"parameters\": {\n",
+    "            \"type\": \"object\",\n",
+    "            \"properties\": {\n",
+    "                \"genre\": {\"type\": \"string\", \"description\": \"The genre for which a book recommendation is requested.\"}\n",
+    "            },\n",
+    "        },\n",
+    "    }\n",
+    "]\n",
+    "\n",
+    "# Append to evaluation_contents\n",
+    "evaluation_contents.append(\n",
+    "    SourceFileContentContent(\n",
+    "        item={\n",
+    "            \"query\": query3,\n",
+    "            \"response\": response3,\n",
+    "            \"tool_definitions\": tool_definition_dict\n",
+    "        }\n",
+    "    )\n",
+    ")"
    ]
   },
   {
@@ -155,7 +245,8 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "query = [\n",
+    "# Test Case 4: Historical fiction recommendation with conversation messages and tool usage\n",
+    "query4 = [\n",
     "    {\"role\": \"system\", \"content\": \"You are an expert in literature and at provid can provide book recommendations.\"},\n",
     "    {\n",
     "        \"createdAt\": \"2025-03-14T08:00:00Z\",\n",
@@ -166,7 +257,7 @@
     "    },\n",
     "]\n",
     "\n",
-    "response = [\n",
+    "response4 = [\n",
     "    {\n",
     "        \"createdAt\": \"2025-03-14T08:00:05Z\",\n",
     "        \"role\": \"assistant\",\n",
@@ -207,7 +298,7 @@
     "    },\n",
     "]\n",
     "\n",
-    "tool_definitions = [\n",
+    "tool_definitions4 = [\n",
     "    {\n",
     "        \"name\": \"get_book\",\n",
     "        \"description\": \"Retrieve a book recommendation for a specified genre.\",\n",
@@ -220,22 +311,25 @@
     "    }\n",
     "]\n",
     "\n",
-    "result = task_adherence_evaluator(\n",
-    "    query=query,\n",
-    "    response=response,\n",
-    "    tool_definitions=tool_definitions,\n",
-    ")\n",
-    "pprint(result)"
+    "# Append to evaluation_contents\n",
+    "evaluation_contents.append(\n",
+    "    SourceFileContentContent(\n",
+    "        item={\n",
+    "            \"query\": query4,\n",
+    "            \"response\": response4,\n",
+    "            \"tool_definitions\": tool_definitions4\n",
+    "        }\n",
+    "    )\n",
+    ")"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## Batch evaluate and visualize results on Azure AI Foundry\n",
-    "Batch evaluate to leverage asynchronous evaluation on a dataset. \n",
+    "### Run Evaluation on All Test Cases\n",
     "\n",
-    "Optionally, you can go to AI Foundry URL for rich Azure AI Foundry data visualization. You can inspect the evaluation scores and reasoning to quickly identify bugs and issues of your agent to fix and improve. Make sure to authenticate to Azure using `az login` in your terminal before running this cell."
+    "Now that we've defined all test cases, let's run the evaluation once on all of them."
    ]
   },
   {
@@ -244,27 +338,37 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from azure.ai.evaluation import evaluate\n",
-    "\n",
-    "# This sample files contains the evaluation data in JSONL format. Where each line is a run from agent.\n",
-    "# This was saved using agent thread and converter.\n",
-    "file_name = \"evaluation_data.jsonl\"\n",
+    "results = run_evaluator(\n",
+    "    evaluator_name=\"task_adherence\",\n",
+    "    evaluation_contents=evaluation_contents,\n",
+    "    data_source_config=data_source_config,\n",
+    "    initialization_parameters=initialization_parameters,\n",
+    "    data_mapping=data_mapping\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Display Results\n",
     "\n",
-    "response = evaluate(\n",
-    "    data=file_name,\n",
-    "    evaluation_name=\"Task Adherence Evaluation\",\n",
-    "    evaluators={\n",
-    "        \"task_adherence\": task_adherence_evaluator,\n",
-    "    },\n",
-    "    azure_ai_project=os.environ[\"AZURE_AI_PROJECT\"],\n",
-    ")\n",
-    "pprint(f'AI Foundary URL: {response.get(\"studio_url\")}')"
+    "View the evaluation results for each test case."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "pprint(results)"
    ]
   }
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "azureai-sample",
+   "display_name": "samples",
    "language": "python",
    "name": "python3"
   },
@@ -277,7 +381,8 @@
    "mimetype": "text/x-python",
    "name": "python",
    "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3"
+   "pygments_lexer": "ipython3",
+   "version": "3.10.19"
   }
  },
  "nbformat": 4,
diff --git a/scenarios/evaluate/Supported_Evaluation_Metrics/Agent_Evaluation/AI_Judge_Evaluator_Task_Completion.ipynb b/scenarios/evaluate/Supported_Evaluation_Metrics/Agent_Evaluation/AI_Judge_Evaluator_Task_Completion.ipynb
new file mode 100644
index 00000000..c5514e9b
--- /dev/null
+++ b/scenarios/evaluate/Supported_Evaluation_Metrics/Agent_Evaluation/AI_Judge_Evaluator_Task_Completion.ipynb
@@ -0,0 +1,510 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "cd88c9a4",
+   "metadata": {},
+   "source": [
+    "# Task Completion Evaluator\n",
+    "\n",
+    "## Objective\n",
+    "This sample demonstrates how to use task completion evaluator on agent data. The supported input formats include:\n",
+    "- simple data such as strings and `dict` describing task responses;\n",
+    "- user-agent conversations in the form of list of agent messages. \n",
+    "\n",
+    "## Time\n",
+    "\n",
+    "You should expect to spend about 20 minutes running this notebook. \n",
+    "\n",
+    "## Before you begin\n",
+    "For quality evaluation, you need to deploy a `gpt` model supporting JSON mode. We recommend using `gpt-4.1` or `gpt-4.1-mini`.    \n",
+    "\n",
+    "### Prerequisite\n",
+    "```bash\n",
+    "pip install azure-ai-projects azure-identity openai\n",
+    "```\n",
+    "Set these environment variables with your own values:\n",
+    "1) **AZURE_AI_PROJECT_ENDPOINT** - Your Azure AI project endpoint in format: `https://<account_name>.services.ai.azure.com/api/projects/<project_name>`\n",
+    "2) **AZURE_AI_MODEL_DEPLOYMENT_NAME** - The deployment name of the model for this AI-assisted evaluator (e.g., gpt-4o-mini)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "fe5dda1c",
+   "metadata": {},
+   "source": [
+    "The Task Completion evaluator assesses whether an AI agent successfully completes the requested task by examining:\n",
+    "- Whether the task was fully completed\n",
+    "- Quality of task execution\n",
+    "- Appropriateness of the response to the original request\n",
+    "\n",
+    "The evaluator uses a binary scoring system (0 or 1):\n",
+    "\n",
+    "    - Score 0: The task was not completed or only partially completed\n",
+    "    - Score 1: The task was successfully and fully completed\n",
+    "\n",
+    "This evaluation focuses on measuring whether the agent's response indicates successful completion of the user's request, regardless of the specific methods or tools used to achieve the task."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "c1642178",
+   "metadata": {},
+   "source": [
+    "Task Completion requires following input:\n",
+    "- Query - This can be a single query or a list of messages(conversation history with agent). The original task request from the user.\n",
+    "- Response - Response from Agent (or any GenAI App). This can be a single text response or a list of messages generated as part of Agent Response.\n",
+    "- Tool Definitions - (Optional) Tool(s) definition used by Agent to answer the query. Providing tool definitions helps the evaluator better understand the context and capabilities available to the agent.\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "ffad775f",
+   "metadata": {},
+   "source": [
+    "### Initialize Task Completion Evaluator\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "0a5fae33",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import os\n",
+    "from openai.types.evals.create_eval_jsonl_run_data_source_param import SourceFileContentContent\n",
+    "from pprint import pprint\n",
+    "from agent_utils import run_evaluator\n",
+    "\n",
+    "# Get environment variables\n",
+    "deployment_name = os.environ[\"AZURE_AI_MODEL_DEPLOYMENT_NAME\"]\n",
+    "\n",
+    "# Data source configuration (defines the schema for evaluation inputs)\n",
+    "data_source_config = {\n",
+    "    \"type\": \"custom\",\n",
+    "    \"item_schema\": {\n",
+    "        \"type\": \"object\",\n",
+    "        \"properties\": {\n",
+    "            \"query\": {\n",
+    "                \"anyOf\": [\n",
+    "                    {\"type\": \"string\"},\n",
+    "                    {\n",
+    "                        \"type\": \"array\",\n",
+    "                        \"items\": {\n",
+    "                            \"type\": \"object\"\n",
+    "                        }\n",
+    "                    }\n",
+    "                ]\n",
+    "            },\n",
+    "            \"response\": {\n",
+    "                \"anyOf\": [\n",
+    "                    {\"type\": \"string\"},\n",
+    "                    {\n",
+    "                        \"type\": \"array\",\n",
+    "                        \"items\": {\n",
+    "                            \"type\": \"object\"\n",
+    "                        }\n",
+    "                    }\n",
+    "                ]\n",
+    "            },\n",
+    "            \"tool_definitions\": {\n",
+    "                \"anyOf\": [\n",
+    "                    {\"type\": \"object\"},\n",
+    "                    {\n",
+    "                        \"type\": \"array\",\n",
+    "                        \"items\": {\n",
+    "                            \"type\": \"object\"\n",
+    "                        }\n",
+    "                    }\n",
+    "                ]\n",
+    "            }\n",
+    "        },\n",
+    "        \"required\": [\"query\", \"response\"]\n",
+    "    },\n",
+    "    \"include_sample_schema\": True\n",
+    "}\n",
+    "\n",
+    "# Data mapping (maps evaluation inputs to evaluator parameters)\n",
+    "data_mapping = {\n",
+    "    \"query\": \"{{item.query}}\",\n",
+    "    \"response\": \"{{item.response}}\",\n",
+    "    \"tool_definitions\": \"{{item.tool_definitions}}\"\n",
+    "}\n",
+    "\n",
+    "# Initialization parameters for the evaluator\n",
+    "initialization_parameters = {\n",
+    "    \"deployment_name\": deployment_name\n",
+    "}\n",
+    "\n",
+    "# Initialize the evaluation_contents list - we'll append all test cases here\n",
+    "evaluation_contents = []"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "af4839bb",
+   "metadata": {},
+   "source": [
+    "### Samples"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "95305bbd",
+   "metadata": {},
+   "source": [
+    "#### Evaluating Simple Task Completion"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "57d4592d",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Test Case 1: Simple Task Completion - Basic weather query\n",
+    "query_simple = \"How is the weather in Seattle?\"\n",
+    "response_simple = \"The current weather in Seattle is partly cloudy with a temperature of 15°C (59°F). There's a light breeze from the northwest at 8 mph, and the humidity is at 68%. No precipitation is expected for the rest of the day.\"\n",
+    "\n",
+    "# Append to evaluation_contents - Basic evaluation without tool definitions\n",
+    "evaluation_contents.append(\n",
+    "    SourceFileContentContent(\n",
+    "        item={\n",
+    "            \"query\": query_simple,\n",
+    "            \"response\": response_simple,\n",
+    "            \"tool_definitions\": None\n",
+    "        }\n",
+    "    )\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "2c66a362",
+   "metadata": {},
+   "source": [
+    "#### Task Completion with Tool Context"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "71ea8725",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Test Case 2: Task Completion with Tool Context\n",
+    "query_tool_context = \"How is the weather in Seattle?\"\n",
+    "response_tool_context = \"I've checked the weather for Seattle and found that it's currently partly cloudy with a temperature of 15°C. There's a light breeze and no rain expected today.\"\n",
+    "\n",
+    "tool_definitions_context = [\n",
+    "    {\n",
+    "        \"name\": \"fetch_weather\",\n",
+    "        \"description\": \"Fetches the weather information for the specified location.\",\n",
+    "        \"parameters\": {\n",
+    "            \"type\": \"object\",\n",
+    "            \"properties\": {\"location\": {\"type\": \"string\", \"description\": \"The location to fetch weather for.\"}},\n",
+    "        },\n",
+    "    }\n",
+    "]\n",
+    "\n",
+    "# Append to evaluation_contents\n",
+    "evaluation_contents.append(\n",
+    "    SourceFileContentContent(\n",
+    "        item={\n",
+    "            \"query\": query_tool_context,\n",
+    "            \"response\": response_tool_context,\n",
+    "            \"tool_definitions\": tool_definitions_context\n",
+    "        }\n",
+    "    )\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "1962234d",
+   "metadata": {},
+   "source": [
+    "#### Task Completion with Tool Definition as Single Dict"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "9a9cd459",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Test Case 3: Tool Definition as Single Dict\n",
+    "query_dict = \"What's the current temperature in Boston?\"\n",
+    "response_dict = \"The current temperature in Boston is 22°C (72°F) with clear skies. It's a beautiful day with low humidity and a gentle breeze.\"\n",
+    "\n",
+    "tool_definition_dict = [\n",
+    "        {\n",
+    "        \"name\": \"fetch_weather\",\n",
+    "        \"description\": \"Fetches the weather information for the specified location.\",\n",
+    "        \"parameters\": {\n",
+    "            \"type\": \"object\",\n",
+    "            \"properties\": {\"location\": {\"type\": \"string\", \"description\": \"The location to fetch weather for.\"}},\n",
+    "        },\n",
+    "    }\n",
+    "]\n",
+    "\n",
+    "# Append to evaluation_contents\n",
+    "evaluation_contents.append(\n",
+    "    SourceFileContentContent(\n",
+    "        item={\n",
+    "            \"query\": query_dict,\n",
+    "            \"response\": response_dict,\n",
+    "            \"tool_definitions\": tool_definition_dict\n",
+    "        }\n",
+    "    )\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "fc85ade7",
+   "metadata": {},
+   "source": [
+    "#### Complex Task with Multiple Steps"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "915768db",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Test Case 4: Complex Task with Multiple Steps\n",
+    "query_complex = \"Can you send me an email with weather information for Seattle?\"\n",
+    "response_complex = [\n",
+    "    {\n",
+    "        \"createdAt\": \"2025-03-26T17:27:35Z\",\n",
+    "        \"run_id\": \"run_zblZyGCNyx6aOYTadmaqM4QN\",\n",
+    "        \"role\": \"assistant\",\n",
+    "        \"content\": [\n",
+    "            {\n",
+    "                \"type\": \"text\",\n",
+    "                \"text\": \"I'll get the current weather information for Seattle and then send you an email with the details.\",\n",
+    "            }\n",
+    "        ],\n",
+    "    },\n",
+    "    {\n",
+    "        \"createdAt\": \"2025-03-26T17:27:42Z\",\n",
+    "        \"run_id\": \"run_zblZyGCNyx6aOYTadmaqM4QN\",\n",
+    "        \"role\": \"assistant\",\n",
+    "        \"content\": [\n",
+    "            {\n",
+    "                \"type\": \"text\",\n",
+    "                \"text\": \"I have successfully sent you an email with the weather information for Seattle. The current weather is partly cloudy with a temperature of 15°C. You should receive the email shortly at your registered email address.\",\n",
+    "            }\n",
+    "        ],\n",
+    "    },\n",
+    "]\n",
+    "\n",
+    "tool_definitions_complex = [\n",
+    "    {\n",
+    "        \"name\": \"fetch_weather\",\n",
+    "        \"description\": \"Fetches the weather information for the specified location.\",\n",
+    "        \"parameters\": {\n",
+    "            \"type\": \"object\",\n",
+    "            \"properties\": {\"location\": {\"type\": \"string\", \"description\": \"The location to fetch weather for.\"}},\n",
+    "        },\n",
+    "    },\n",
+    "    {\n",
+    "        \"name\": \"send_email\",\n",
+    "        \"description\": \"Sends an email with the specified subject and body to the recipient.\",\n",
+    "        \"parameters\": {\n",
+    "            \"type\": \"object\",\n",
+    "            \"properties\": {\n",
+    "                \"recipient\": {\"type\": \"string\", \"description\": \"Email address of the recipient.\"},\n",
+    "                \"subject\": {\"type\": \"string\", \"description\": \"Subject of the email.\"},\n",
+    "                \"body\": {\"type\": \"string\", \"description\": \"Body content of the email.\"},\n",
+    "            },\n",
+    "        },\n",
+    "    },\n",
+    "]\n",
+    "\n",
+    "# Append to evaluation_contents\n",
+    "evaluation_contents.append(\n",
+    "    SourceFileContentContent(\n",
+    "        item={\n",
+    "            \"query\": query_complex,\n",
+    "            \"response\": response_complex,\n",
+    "            \"tool_definitions\": tool_definitions_complex\n",
+    "        }\n",
+    "    )\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "5523dfcf",
+   "metadata": {},
+   "source": [
+    "#### Query as Conversation History (List of Messages)\n",
+    "The evaluator also supports query as a list of messages representing conversation history. This helps evaluate task completion in the context of a full conversation."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "2c4772be",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Test Case 5: Query as Conversation History\n",
+    "query_conversation = [\n",
+    "    {\n",
+    "        \"role\": \"system\",\n",
+    "        \"content\": \"You are a helpful assistant that can fetch weather information and send emails.\"\n",
+    "    },\n",
+    "    {\n",
+    "        \"role\": \"user\", \n",
+    "        \"content\": \"Hi, I need to plan my day. Can you check the weather in Seattle for me?\"\n",
+    "    },\n",
+    "    {\n",
+    "        \"role\": \"user\",\n",
+    "        \"content\": \"Also, please send me an email summary of the weather so I can reference it later.\"\n",
+    "    }\n",
+    "]\n",
+    "\n",
+    "response_conversation = \"I've checked the weather in Seattle for you. It's currently 15°C and partly cloudy with light winds. I've also sent you an email summary with these details so you can reference them throughout your day. The task has been completed successfully.\"\n",
+    "\n",
+    "tool_definitions_conversation = [\n",
+    "    {\n",
+    "        \"name\": \"fetch_weather\",\n",
+    "        \"description\": \"Fetches the weather information for the specified location.\",\n",
+    "        \"parameters\": {\n",
+    "            \"type\": \"object\",\n",
+    "            \"properties\": {\"location\": {\"type\": \"string\", \"description\": \"The location to fetch weather for.\"}},\n",
+    "        },\n",
+    "    },\n",
+    "    {\n",
+    "        \"name\": \"send_email\",\n",
+    "        \"description\": \"Sends an email with the specified subject and body to the recipient.\",\n",
+    "        \"parameters\": {\n",
+    "            \"type\": \"object\",\n",
+    "            \"properties\": {\n",
+    "                \"recipient\": {\"type\": \"string\", \"description\": \"Email address of the recipient.\"},\n",
+    "                \"subject\": {\"type\": \"string\", \"description\": \"Subject of the email.\"},\n",
+    "                \"body\": {\"type\": \"string\", \"description\": \"Body content of the email.\"},\n",
+    "            },\n",
+    "        },\n",
+    "    },\n",
+    "]\n",
+    "\n",
+    "# Append to evaluation_contents\n",
+    "evaluation_contents.append(\n",
+    "    SourceFileContentContent(\n",
+    "        item={\n",
+    "            \"query\": query_conversation,\n",
+    "            \"response\": response_conversation,\n",
+    "            \"tool_definitions\": tool_definitions_conversation\n",
+    "        }\n",
+    "    )\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "100e943b",
+   "metadata": {},
+   "source": [
+    "#### Example of Incomplete Task"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "9986fa70",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Test Case 6: Incomplete Task Example\n",
+    "query_incomplete = \"Can you send me an email with weather information for Seattle?\"\n",
+    "response_incomplete = \"I can see that you want weather information for Seattle. The weather there is usually quite nice this time of year, with temperatures ranging from mild to warm depending on the season.\"\n",
+    "\n",
+    "# Append to evaluation_contents - This response doesn't complete the email sending task\n",
+    "evaluation_contents.append(\n",
+    "    SourceFileContentContent(\n",
+    "        item={\n",
+    "            \"query\": query_incomplete,\n",
+    "            \"response\": response_incomplete,\n",
+    "            \"tool_definitions\": None,\n",
+    "        }\n",
+    "    )\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "b3a30f59",
+   "metadata": {},
+   "source": [
+    "### Execute Batch Evaluation\n",
+    "Run all test cases together using the batch evaluation approach."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "c36b9471",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "results = run_evaluator(\n",
+    "    evaluator_name=\"task_completion\",\n",
+    "    evaluation_contents=evaluation_contents,\n",
+    "    data_source_config=data_source_config,\n",
+    "    initialization_parameters=initialization_parameters,\n",
+    "    data_mapping=data_mapping\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "6d403a51",
+   "metadata": {},
+   "source": [
+    "### Results Display\n",
+    "Display the evaluation results for each test case."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "35c806e1",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "pprint(results)"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "samples",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.19"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
diff --git a/scenarios/evaluate/Supported_Evaluation_Metrics/Agent_Evaluation/AI_Judge_Evaluator_Task_Navigation_Efficiency.ipynb b/scenarios/evaluate/Supported_Evaluation_Metrics/Agent_Evaluation/AI_Judge_Evaluator_Task_Navigation_Efficiency.ipynb
new file mode 100644
index 00000000..4e06d97c
--- /dev/null
+++ b/scenarios/evaluate/Supported_Evaluation_Metrics/Agent_Evaluation/AI_Judge_Evaluator_Task_Navigation_Efficiency.ipynb
@@ -0,0 +1,498 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "28c2bc93",
+   "metadata": {},
+   "source": [
+    "# Task Navigation Efficiency Evaluator\n",
+    "\n",
+    "## Objective\n",
+    "This sample demonstrates how to use task navigation efficiency evaluator on agent data. The supported input formats include:\n",
+    "- simple data such as strings and `dict` describing task responses;\n",
+    "- user-agent conversations in the form of list of agent messages. \n",
+    "\n",
+    "## Time\n",
+    "\n",
+    "You should expect to spend about 20 minutes running this notebook. \n",
+    "\n",
+    "## Before you begin\n",
+    "This is a deterministic evaluator that compares navigation paths.\n",
+    "\n",
+    "### Prerequisite\n",
+    "```bash\n",
+    "pip install azure-ai-projects azure-identity openai\n",
+    "```\n",
+    "Set these environment variables with your own values:\n",
+    "1) **AZURE_AI_PROJECT_ENDPOINT** - Your Azure AI project endpoint in format: `https://<account_name>.services.ai.azure.com/api/projects/<project_name>`\n",
+    "2) **AZURE_AI_MODEL_DEPLOYMENT_NAME** - The deployment name of the model for this AI-assisted evaluator (e.g., gpt-4o-mini)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "d61cd511",
+   "metadata": {},
+   "source": [
+    "The Task Navigation Efficiency Evaluator measures how efficiently an agent navigates through a sequence of actions compared to an optimal task completion path.\n",
+    "\n",
+    "The evaluator provides comprehensive evaluation with both binary matching results and additional detailed P\\R\\F1 results:\n",
+    "\n",
+    "**Primary Result:**\n",
+    "- **Binary Match Result**: Pass/Fail based on the selected matching mode\n",
+    "\n",
+    "**Available Matching Modes:**\n",
+    "- **Exact Match**: Agent's tool calls must exactly match the ground truth (default)\n",
+    "- **In-Order Match**: All ground truth steps must appear in correct order (allows extra steps)\n",
+    "- **Any-Order Match**: All ground truth steps must appear with sufficient frequency (most lenient)\n",
+    "\n",
+    "**Properties Bag Additional Metrics (0.0 - 1.0):**\n",
+    "- **Precision**: How many of the agent's steps were necessary (relevant to ground truth)\n",
+    "- **Recall**: How many of the required steps were executed by the agent  \n",
+    "- **F1 Score**: Harmonic mean of precision and recall\n",
+    "\n",
+    "The evaluation requires the following inputs:\n",
+    "- **Response**: The agent's response containing tool calls as a list of messages or string\n",
+    "- **Ground Truth**: List of expected tool/action steps as strings, or tuple with parameters for matching"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "5e4e0a1d",
+   "metadata": {},
+   "source": [
+    "### Initialize Task Navigation Efficiency Evaluator\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "bc3726cd",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import os\n",
+    "from openai.types.evals.create_eval_jsonl_run_data_source_param import SourceFileContentContent\n",
+    "from pprint import pprint\n",
+    "from agent_utils import run_evaluator\n",
+    "\n",
+    "# Get environment variables\n",
+    "deployment_name = os.environ[\"AZURE_AI_MODEL_DEPLOYMENT_NAME\"]\n",
+    "\n",
+    "# Data source configuration (defines the schema for evaluation inputs)\n",
+    "data_source_config = {\n",
+    "    \"type\": \"custom\",\n",
+    "    \"item_schema\": {\n",
+    "        \"type\": \"object\",\n",
+    "        \"properties\": {\n",
+    "            \"response\": {\n",
+    "                \"type\": \"array\"\n",
+    "            },\n",
+    "            \"ground_truth\": {\n",
+    "                \"type\": \"array\"\n",
+    "            }\n",
+    "        },\n",
+    "        \"required\": [\"response\", \"ground_truth\"]\n",
+    "    },\n",
+    "    \"include_sample_schema\": True\n",
+    "}\n",
+    "\n",
+    "# Data mapping (maps evaluation inputs to evaluator parameters)\n",
+    "data_mapping = {\n",
+    "    \"response\": \"{{item.response}}\",\n",
+    "    \"ground_truth\": \"{{item.ground_truth}}\"\n",
+    "}\n",
+    "\n",
+    "# Initialization parameters for the evaluator\n",
+    "initialization_parameters = {\n",
+    "    \"matching_mode\": \"exact_match\"  # Can be \"exact_match\", \"in_order_match\", or \"any_order_match\"\n",
+    "}\n",
+    "\n",
+    "# Initialize the evaluation_contents list - we'll append all test cases here\n",
+    "evaluation_contents = []"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "15a53227",
+   "metadata": {},
+   "source": [
+    "### Samples"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "1d75457a",
+   "metadata": {},
+   "source": [
+    "#### Sample 1: Perfect Path (Exact Match)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "0cae15fb",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Test Case 1: Perfect Path (Exact Match) - Agent follows the exact optimal path\n",
+    "response_perfect = [\n",
+    "    {\n",
+    "        \"role\": \"assistant\",\n",
+    "        \"content\": [{\"type\": \"tool_call\", \"tool_call_id\": \"call_1\", \"name\": \"search\", \"arguments\": {}}],\n",
+    "    },\n",
+    "    {\n",
+    "        \"role\": \"assistant\", \n",
+    "        \"content\": [{\"type\": \"tool_call\", \"tool_call_id\": \"call_2\", \"name\": \"analyze\", \"arguments\": {}}],\n",
+    "    },\n",
+    "    {\n",
+    "        \"role\": \"assistant\",\n",
+    "        \"content\": [{\"type\": \"tool_call\", \"tool_call_id\": \"call_3\", \"name\": \"report\", \"arguments\": {}}],\n",
+    "    },\n",
+    "]\n",
+    "\n",
+    "ground_truth_perfect = [\"search\", \"analyze\", \"report\"]\n",
+    "\n",
+    "# Append to evaluation_contents\n",
+    "evaluation_contents.append(\n",
+    "    SourceFileContentContent(\n",
+    "        item={\n",
+    "            \"response\": response_perfect,\n",
+    "            \"ground_truth\": ground_truth_perfect\n",
+    "        }\n",
+    "    )\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "8a9c5e69",
+   "metadata": {},
+   "source": [
+    "#### Sample 2: Efficient Path with Extra Steps"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "f53dde31",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Test Case 2: Path with Extra Steps - Agent performs all required steps but with extra unnecessary step\n",
+    "response_extra = [\n",
+    "    {\n",
+    "        \"role\": \"assistant\",\n",
+    "        \"content\": [{\"type\": \"tool_call\", \"tool_call_id\": \"call_1\", \"name\": \"search\", \"arguments\": {}}],\n",
+    "    },\n",
+    "    {\n",
+    "        \"role\": \"assistant\",\n",
+    "        \"content\": [{\"type\": \"tool_call\", \"tool_call_id\": \"call_2\", \"name\": \"validate\", \"arguments\": {}}],\n",
+    "    },\n",
+    "    {\n",
+    "        \"role\": \"assistant\", \n",
+    "        \"content\": [{\"type\": \"tool_call\", \"tool_call_id\": \"call_3\", \"name\": \"analyze\", \"arguments\": {}}],\n",
+    "    },\n",
+    "    {\n",
+    "        \"role\": \"assistant\",\n",
+    "        \"content\": [{\"type\": \"tool_call\", \"tool_call_id\": \"call_4\", \"name\": \"report\", \"arguments\": {}}],\n",
+    "    },\n",
+    "]\n",
+    "\n",
+    "ground_truth_extra = [\"search\", \"analyze\", \"report\"]\n",
+    "\n",
+    "# Append to evaluation_contents\n",
+    "evaluation_contents.append(\n",
+    "    SourceFileContentContent(\n",
+    "        item={\n",
+    "            \"response\": response_extra,\n",
+    "            \"ground_truth\": ground_truth_extra\n",
+    "        }\n",
+    "    )\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "e4fa5757",
+   "metadata": {},
+   "source": [
+    "#### Sample 3: Inefficient Path (Wrong Order)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "c2226715",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Test Case 3: Wrong Order - Agent performs all required steps but in wrong order\n",
+    "response_wrong_order = [\n",
+    "    {\n",
+    "        \"role\": \"assistant\",\n",
+    "        \"content\": [{\"type\": \"tool_call\", \"tool_call_id\": \"call_1\", \"name\": \"report\", \"arguments\": {}}],\n",
+    "    },\n",
+    "    {\n",
+    "        \"role\": \"assistant\",\n",
+    "        \"content\": [{\"type\": \"tool_call\", \"tool_call_id\": \"call_2\", \"name\": \"search\", \"arguments\": {}}],\n",
+    "    },\n",
+    "    {\n",
+    "        \"role\": \"assistant\", \n",
+    "        \"content\": [{\"type\": \"tool_call\", \"tool_call_id\": \"call_3\", \"name\": \"analyze\", \"arguments\": {}}],\n",
+    "    },\n",
+    "]\n",
+    "\n",
+    "ground_truth_wrong_order = [\"search\", \"analyze\", \"report\"]\n",
+    "\n",
+    "# Append to evaluation_contents\n",
+    "evaluation_contents.append(\n",
+    "    SourceFileContentContent(\n",
+    "        item={\n",
+    "            \"response\": response_wrong_order,\n",
+    "            \"ground_truth\": ground_truth_wrong_order\n",
+    "        }\n",
+    "    )\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "ae74ac8e",
+   "metadata": {},
+   "source": [
+    "#### Sample 4: Incomplete Path (Missing Steps)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "ccd95640",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Test Case 4: Missing Steps - Agent performs only some of the required steps (incomplete)\n",
+    "response_missing = [\n",
+    "    {\n",
+    "        \"role\": \"assistant\",\n",
+    "        \"content\": [{\"type\": \"tool_call\", \"tool_call_id\": \"call_1\", \"name\": \"search\", \"arguments\": {}}],\n",
+    "    },\n",
+    "    {\n",
+    "        \"role\": \"assistant\",\n",
+    "        \"content\": [{\"type\": \"tool_call\", \"tool_call_id\": \"call_2\", \"name\": \"analyze\", \"arguments\": {}}],\n",
+    "    },\n",
+    "]\n",
+    "\n",
+    "ground_truth_missing = [\"search\", \"analyze\", \"report\"]\n",
+    "\n",
+    "# Append to evaluation_contents\n",
+    "evaluation_contents.append(\n",
+    "    SourceFileContentContent(\n",
+    "        item={\n",
+    "            \"response\": response_missing,\n",
+    "            \"ground_truth\": ground_truth_missing\n",
+    "        }\n",
+    "    )\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "2c9111ef",
+   "metadata": {},
+   "source": [
+    "#### Sample 5: Real-World Customer Service Scenario"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "b4bd7b47",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Test Case 5: Customer Service Scenario - Real-world example: Customer service agent handling a refund request\n",
+    "response_service = [\n",
+    "    {\n",
+    "        \"role\": \"assistant\",\n",
+    "        \"content\": [{\"type\": \"tool_call\", \"tool_call_id\": \"call_1\", \"name\": \"lookup_order\", \"arguments\": {\"order_id\": \"12345\"}}],\n",
+    "    },\n",
+    "    {\n",
+    "        \"role\": \"assistant\",\n",
+    "        \"content\": [{\"type\": \"tool_call\", \"tool_call_id\": \"call_2\", \"name\": \"check_inventory\", \"arguments\": {\"product_id\": \"ABC123\"}}],\n",
+    "    },\n",
+    "    {\n",
+    "        \"role\": \"assistant\", \n",
+    "        \"content\": [{\"type\": \"tool_call\", \"tool_call_id\": \"call_3\", \"name\": \"calculate_refund\", \"arguments\": {\"order_id\": \"12345\"}}],\n",
+    "    },\n",
+    "    {\n",
+    "        \"role\": \"assistant\",\n",
+    "        \"content\": [{\"type\": \"tool_call\", \"tool_call_id\": \"call_4\", \"name\": \"process_refund\", \"arguments\": {\"order_id\": \"12345\", \"amount\": \"29.99\"}}],\n",
+    "    },\n",
+    "]\n",
+    "\n",
+    "ground_truth_service = [\"lookup_order\", \"check_inventory\", \"calculate_refund\", \"process_refund\"]\n",
+    "\n",
+    "# Append to evaluation_contents\n",
+    "evaluation_contents.append(\n",
+    "    SourceFileContentContent(\n",
+    "        item={\n",
+    "            \"response\": response_service,\n",
+    "            \"ground_truth\": ground_truth_service\n",
+    "        }\n",
+    "    )\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "4bf2e0ee",
+   "metadata": {},
+   "source": [
+    "#### Sample 6: Complex Path with Duplicates"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "e4dcf585",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Test Case 6: Complex Path with Duplicates - Agent repeats some steps and includes extra ones\n",
+    "response_duplicates = [\n",
+    "    {\n",
+    "        \"role\": \"assistant\",\n",
+    "        \"content\": [{\"type\": \"tool_call\", \"tool_call_id\": \"call_1\", \"name\": \"search\", \"arguments\": {}}],\n",
+    "    },\n",
+    "    {\n",
+    "        \"role\": \"assistant\",\n",
+    "        \"content\": [{\"type\": \"tool_call\", \"tool_call_id\": \"call_2\", \"name\": \"search\", \"arguments\": {}}],  # duplicate\n",
+    "    },\n",
+    "    {\n",
+    "        \"role\": \"assistant\",\n",
+    "        \"content\": [{\"type\": \"tool_call\", \"tool_call_id\": \"call_4\", \"name\": \"analyze\", \"arguments\": {}}],\n",
+    "    },\n",
+    "    {\n",
+    "        \"role\": \"assistant\",\n",
+    "        \"content\": [{\"type\": \"tool_call\", \"tool_call_id\": \"call_5\", \"name\": \"report\", \"arguments\": {}}],\n",
+    "    },\n",
+    "]\n",
+    "\n",
+    "ground_truth_duplicates = [\"search\", \"analyze\", \"report\"]\n",
+    "\n",
+    "# Append to evaluation_contents\n",
+    "evaluation_contents.append(\n",
+    "    SourceFileContentContent(\n",
+    "        item={\n",
+    "            \"response\": response_duplicates,\n",
+    "            \"ground_truth\": ground_truth_duplicates\n",
+    "        }\n",
+    "    )\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "478c00fb",
+   "metadata": {},
+   "source": [
+    "#### Sample 7: Tuple Format with Parameters"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "7404f830",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Test Case 7: Tuple Format with Parameters - Supports exact parameter matching\n",
+    "response_params = [\n",
+    "    {\n",
+    "        \"role\": \"assistant\",\n",
+    "        \"content\": [{\"type\": \"tool_call\", \"tool_call_id\": \"call_1\", \"name\": \"search\", \"arguments\": {\"query\": \"test\"}}],\n",
+    "    },\n",
+    "]\n",
+    "\n",
+    "# Ground truth using tuple format: (tool_names, parameters_dict)\n",
+    "# Parameters must match exactly for tools to be considered matching\n",
+    "ground_truth_params = [[\"search\"], {\"search\": {\"query\": \"test\"}}]\n",
+    "\n",
+    "# Append to evaluation_contents\n",
+    "evaluation_contents.append(\n",
+    "    SourceFileContentContent(\n",
+    "        item={\n",
+    "            \"response\": response_params,\n",
+    "            \"ground_truth\": ground_truth_params\n",
+    "        }\n",
+    "    )\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "16a11522",
+   "metadata": {},
+   "source": [
+    "### Execute Batch Evaluation\n",
+    "Run all test cases together using the batch evaluation approach."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "1185d9f2",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "results = run_evaluator(\n",
+    "    evaluator_name=\"task_navigation_efficiency\",\n",
+    "    evaluation_contents=evaluation_contents,\n",
+    "    data_source_config=data_source_config,\n",
+    "    initialization_parameters=initialization_parameters,\n",
+    "    data_mapping=data_mapping\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "6ed16d6c",
+   "metadata": {},
+   "source": [
+    "### Results Display\n",
+    "Display the evaluation results for each test case."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "f9c0bd93",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "pprint(results)"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "samples",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.19"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
diff --git a/scenarios/evaluate/Supported_Evaluation_Metrics/Agent_Evaluation/AI_Judge_Evaluator_Tool_Call_Accuracy.ipynb b/scenarios/evaluate/Supported_Evaluation_Metrics/Agent_Evaluation/AI_Judge_Evaluator_Tool_Call_Accuracy.ipynb
index 0aa60d2d..fc77b828 100644
--- a/scenarios/evaluate/Supported_Evaluation_Metrics/Agent_Evaluation/AI_Judge_Evaluator_Tool_Call_Accuracy.ipynb
+++ b/scenarios/evaluate/Supported_Evaluation_Metrics/Agent_Evaluation/AI_Judge_Evaluator_Tool_Call_Accuracy.ipynb
@@ -16,18 +16,15 @@
     "You should expect to spend about 20 minutes running this notebook. \n",
     "\n",
     "## Before you begin\n",
-    "For quality evaluation, you need to deploy a `gpt` model supporting JSON mode. We recommend a model `gpt-4o` or `gpt-4o-mini` for their strong reasoning capabilities.    \n",
+    "For quality evaluation, you need to deploy a `gpt` model supporting JSON mode. We recommend using `gpt-4.1`.    \n",
     "\n",
     "### Prerequisite\n",
     "```bash\n",
-    "pip install azure-ai-projects azure-identity azure-ai-evaluation\n",
+    "pip install azure-ai-projects azure-identity openai\n",
     "```\n",
     "Set these environment variables with your own values:\n",
-    "1) **AZURE_AI_PROJECT** - The project connection string, as found in the overview page of your Azure AI Foundry project.\n",
-    "2) **MODEL_DEPLOYMENT_NAME** - The deployment name of the model for this AI-assisted evaluator, as found under the \"Name\" column in the \"Models + endpoints\" tab in your Azure AI Foundry project.\n",
-    "3) **AZURE_OPENAI_ENDPOINT** - Azure Open AI Endpoint to be used for evaluation.\n",
-    "4) **AZURE_OPENAI_API_KEY** - Azure Open AI Key to be used for evaluation.\n",
-    "5) **AZURE_OPENAI_API_VERSION** - Azure Open AI Api version to be used for evaluation.\n"
+    "1) **AZURE_AI_PROJECT_ENDPOINT** - Your Azure AI project endpoint in format: `https://<account_name>.services.ai.azure.com/api/projects/<project_name>`\n",
+    "2) **AZURE_AI_MODEL_DEPLOYMENT_NAME** - The deployment name of the model for this AI-assisted evaluator (e.g., gpt-4o-mini)"
    ]
   },
   {
@@ -75,18 +72,84 @@
    "outputs": [],
    "source": [
     "import os\n",
-    "from azure.ai.evaluation import ToolCallAccuracyEvaluator, AzureOpenAIModelConfiguration\n",
+    "from openai.types.evals.create_eval_jsonl_run_data_source_param import SourceFileContentContent\n",
     "from pprint import pprint\n",
+    "from agent_utils import run_evaluator\n",
     "\n",
-    "model_config = AzureOpenAIModelConfiguration(\n",
-    "    azure_endpoint=os.environ[\"AZURE_OPENAI_ENDPOINT\"],\n",
-    "    api_key=os.environ[\"AZURE_OPENAI_API_KEY\"],\n",
-    "    api_version=os.environ[\"AZURE_OPENAI_API_VERSION\"],\n",
-    "    azure_deployment=os.environ[\"MODEL_DEPLOYMENT_NAME\"],\n",
-    ")\n",
+    "# Get environment variables\n",
+    "deployment_name = os.environ[\"AZURE_AI_MODEL_DEPLOYMENT_NAME\"]\n",
     "\n",
+    "# Data source configuration (defines the schema for evaluation inputs)\n",
+    "data_source_config = {\n",
+    "    \"type\": \"custom\",\n",
+    "    \"item_schema\": {\n",
+    "        \"type\": \"object\",\n",
+    "        \"properties\": {\n",
+    "            \"query\": {\n",
+    "                \"anyOf\": [\n",
+    "                    {\"type\": \"string\"},\n",
+    "                    {\n",
+    "                        \"type\": \"array\",\n",
+    "                        \"items\": {\n",
+    "                            \"type\": \"object\"\n",
+    "                        }\n",
+    "                    }\n",
+    "                ]\n",
+    "            },\n",
+    "            \"tool_definitions\": {\n",
+    "                \"anyOf\": [\n",
+    "                    {\"type\": \"object\"},\n",
+    "                    {\n",
+    "                        \"type\": \"array\",\n",
+    "                        \"items\": {\n",
+    "                            \"type\": \"object\"\n",
+    "                        }\n",
+    "                    }\n",
+    "                ]\n",
+    "            },\n",
+    "            \"tool_calls\": {\n",
+    "                \"anyOf\": [\n",
+    "                    {\"type\": \"object\"},\n",
+    "                    {\n",
+    "                        \"type\": \"array\",\n",
+    "                        \"items\": {\n",
+    "                            \"type\": \"object\"\n",
+    "                        }\n",
+    "                    }\n",
+    "                ]\n",
+    "            },\n",
+    "            \"response\": {\n",
+    "                \"anyOf\": [\n",
+    "                    {\"type\": \"string\"},\n",
+    "                    {\n",
+    "                        \"type\": \"array\",\n",
+    "                        \"items\": {\n",
+    "                            \"type\": \"object\"\n",
+    "                        }\n",
+    "                    }\n",
+    "                ]\n",
+    "            }\n",
+    "        },\n",
+    "        \"required\": [\"query\", \"tool_definitions\"]\n",
+    "    },\n",
+    "    \"include_sample_schema\": True\n",
+    "}\n",
+    "\n",
+    "# Data mapping (maps evaluation inputs to evaluator parameters)\n",
+    "data_mapping = {\n",
+    "    \"query\": \"{{item.query}}\",\n",
+    "    \"tool_definitions\": \"{{item.tool_definitions}}\",\n",
+    "    \"tool_calls\": \"{{item.tool_calls}}\",\n",
+    "    \"response\": \"{{item.response}}\"\n",
+    "}\n",
+    "\n",
+    "# Initialization parameters for the evaluator\n",
+    "initialization_parameters = {\n",
+    "    \"deployment_name\": deployment_name\n",
+    "}\n",
     "\n",
-    "tool_call_accuracy = ToolCallAccuracyEvaluator(model_config=model_config)"
+    "# Initialize the evaluation_contents list - we'll append all test cases here\n",
+    "evaluation_contents = []"
    ]
   },
   {
@@ -109,33 +172,40 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "query = \"How is the weather in Seattle ?\"\n",
-    "tool_call = {\n",
-    "    \"type\": \"tool_call\",\n",
-    "    \"tool_call_id\": \"call_CUdbkBfvVBla2YP3p24uhElJ\",\n",
-    "    \"name\": \"fetch_weather\",\n",
-    "    \"arguments\": {\"location\": \"Seattle\"},\n",
-    "}\n",
+    "# Test Case 1: Single tool call\n",
+    "query_single = \"How is the weather in Seattle ?\"\n",
+    "tool_call_single = [\n",
+    "    {\n",
+    "        \"type\": \"tool_call\",\n",
+    "        \"tool_call_id\": \"call_CUdbkBfvVBla2YP3p24uhElJ\",\n",
+    "        \"name\": \"fetch_weather\",\n",
+    "        \"arguments\": {\"location\": \"Seattle\"},\n",
+    "    }\n",
+    "]\n",
     "\n",
-    "tool_definition = {\n",
-    "    \"id\": \"fetch_weather\",\n",
-    "    \"name\": \"fetch_weather\",\n",
-    "    \"description\": \"Fetches the weather information for the specified location.\",\n",
-    "    \"parameters\": {\n",
-    "        \"type\": \"object\",\n",
-    "        \"properties\": {\"location\": {\"type\": \"string\", \"description\": \"The location to fetch weather for.\"}},\n",
-    "    },\n",
-    "}"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "response = tool_call_accuracy(query=query, tool_calls=tool_call, tool_definitions=tool_definition)\n",
-    "pprint(response)"
+    "tool_definition_single = [\n",
+    "    {\n",
+    "        \"id\": \"fetch_weather\",\n",
+    "        \"name\": \"fetch_weather\",\n",
+    "        \"description\": \"Fetches the weather information for the specified location.\",\n",
+    "        \"parameters\": {\n",
+    "            \"type\": \"object\", \n",
+    "            \"properties\": {\"location\": {\"type\": \"string\", \"description\": \"The location to fetch weather for.\"}},\n",
+    "        },\n",
+    "    }\n",
+    "]\n",
+    "\n",
+    "# Append to evaluation_contents\n",
+    "evaluation_contents.append(\n",
+    "    SourceFileContentContent(\n",
+    "        item={\n",
+    "            \"query\": query_single,\n",
+    "            \"tool_calls\": tool_call_single,\n",
+    "            \"tool_definitions\": tool_definition_single,\n",
+    "            \"response\": None\n",
+    "        }\n",
+    "    )\n",
+    ")"
    ]
   },
   {
@@ -151,8 +221,9 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "query = \"How is the weather in Seattle ?\"\n",
-    "tool_calls = [\n",
+    "# Test Case 2: Multiple tool calls for single query\n",
+    "query_multiple = \"How is the weather in Seattle ?\"\n",
+    "tool_calls_multiple = [\n",
     "    {\n",
     "        \"type\": \"tool_call\",\n",
     "        \"tool_call_id\": \"call_CUdbkBfvVBla2YP3p24uhElJ\",\n",
@@ -167,25 +238,29 @@
     "    },\n",
     "]\n",
     "\n",
-    "tool_definition = {\n",
-    "    \"id\": \"fetch_weather\",\n",
-    "    \"name\": \"fetch_weather\",\n",
-    "    \"description\": \"Fetches the weather information for the specified location.\",\n",
-    "    \"parameters\": {\n",
-    "        \"type\": \"object\",\n",
-    "        \"properties\": {\"location\": {\"type\": \"string\", \"description\": \"The location to fetch weather for.\"}},\n",
-    "    },\n",
-    "}"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "response = tool_call_accuracy(query=query, tool_calls=tool_calls, tool_definitions=tool_definition)\n",
-    "pprint(response)"
+    "tool_definition_multiple = [\n",
+    "    {\n",
+    "        \"id\": \"fetch_weather\",\n",
+    "        \"name\": \"fetch_weather\",\n",
+    "        \"description\": \"Fetches the weather information for the specified location.\",\n",
+    "        \"parameters\": {\n",
+    "            \"type\": \"object\",\n",
+    "            \"properties\": {\"location\": {\"type\": \"string\", \"description\": \"The location to fetch weather for.\"}},\n",
+    "        },\n",
+    "    }\n",
+    "]\n",
+    "\n",
+    "# Append to evaluation_contents\n",
+    "evaluation_contents.append(\n",
+    "    SourceFileContentContent(\n",
+    "        item={\n",
+    "            \"query\": query_multiple,\n",
+    "            \"tool_calls\": tool_calls_multiple,\n",
+    "            \"tool_definitions\": tool_definition_multiple,\n",
+    "            \"response\": None\n",
+    "        }\n",
+    "    )\n",
+    ")"
    ]
   },
   {
@@ -202,8 +277,9 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "query = \"Can you send me an email with weather information for Seattle?\"\n",
-    "response = [\n",
+    "# Test Case 3: Tool calls extracted from response\n",
+    "query_response = \"Can you send me an email with weather information for Seattle?\"\n",
+    "response_messages = [\n",
     "    {\n",
     "        \"createdAt\": \"2025-03-26T17:27:35Z\",\n",
     "        \"run_id\": \"run_zblZyGCNyx6aOYTadmaqM4QN\",\n",
@@ -222,7 +298,7 @@
     "        \"run_id\": \"run_zblZyGCNyx6aOYTadmaqM4QN\",\n",
     "        \"tool_call_id\": \"call_CUdbkBfvVBla2YP3p24uhElJ\",\n",
     "        \"role\": \"tool\",\n",
-    "        \"content\": [{\"type\": \"tool_result\", \"tool_result\": {\"weather\": \"Rainy, 14\\u00b0C\"}}],\n",
+    "        \"content\": [{\"type\": \"tool_result\", \"tool_result\": {\"weather\": \"Rainy, 14°C\"}}],\n",
     "    },\n",
     "    {\n",
     "        \"createdAt\": \"2025-03-26T17:27:38Z\",\n",
@@ -236,7 +312,7 @@
     "                \"arguments\": {\n",
     "                    \"recipient\": \"your_email@example.com\",\n",
     "                    \"subject\": \"Weather Information for Seattle\",\n",
-    "                    \"body\": \"The current weather in Seattle is rainy with a temperature of 14\\u00b0C.\",\n",
+    "                    \"body\": \"The current weather in Seattle is rainy with a temperature of 14°C.\",\n",
     "                },\n",
     "            }\n",
     "        ],\n",
@@ -257,13 +333,13 @@
     "        \"content\": [\n",
     "            {\n",
     "                \"type\": \"text\",\n",
-    "                \"text\": \"I have successfully sent you an email with the weather information for Seattle. The current weather is rainy with a temperature of 14\\u00b0C.\",\n",
+    "                \"text\": \"I have successfully sent you an email with the weather information for Seattle. The current weather is rainy with a temperature of 14°C.\",\n",
     "            }\n",
     "        ],\n",
     "    },\n",
     "]\n",
     "\n",
-    "tool_definitions = [\n",
+    "tool_definitions_response = [\n",
     "    {\n",
     "        \"name\": \"fetch_weather\",\n",
     "        \"description\": \"Fetches the weather information for the specified location.\",\n",
@@ -284,7 +360,26 @@
     "            },\n",
     "        },\n",
     "    },\n",
-    "]"
+    "]\n",
+    "\n",
+    "# Append to evaluation_contents\n",
+    "evaluation_contents.append(\n",
+    "    SourceFileContentContent(\n",
+    "        item={\n",
+    "            \"query\": query_response,\n",
+    "            \"response\": response_messages,\n",
+    "            \"tool_definitions\": tool_definitions_response,\n",
+    "            \"tool_calls\": None\n",
+    "        }\n",
+    "    )\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "#### Response as String (str)"
    ]
   },
   {
@@ -293,18 +388,131 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "response = tool_call_accuracy(query=query, response=response, tool_definitions=tool_definitions)\n",
-    "pprint(response)"
+    "# Test Case 4: Response as string\n",
+    "query_str = \"Book a flight to New York for tomorrow\"\n",
+    "\n",
+    "# Response as a simple string instead of a list of messages\n",
+    "response_str = \"I've found several flight options to New York for tomorrow. I'll use the booking tool to reserve your seat.\"\n",
+    "\n",
+    "tool_calls_str = [\n",
+    "    {\n",
+    "        \"type\": \"tool_call\",\n",
+    "        \"tool_call_id\": \"call_book_flight_456\",\n",
+    "        \"name\": \"book_flight\",\n",
+    "        \"arguments\": {\n",
+    "            \"destination\": \"New York\",\n",
+    "            \"date\": \"tomorrow\"\n",
+    "        },\n",
+    "    }\n",
+    "]\n",
+    "\n",
+    "tool_definitions_str = [\n",
+    "    {\n",
+    "        \"name\": \"book_flight\",\n",
+    "        \"description\": \"Books a flight to the specified destination on the given date.\",\n",
+    "        \"parameters\": {\n",
+    "            \"type\": \"object\",\n",
+    "            \"properties\": {\n",
+    "                \"destination\": {\"type\": \"string\", \"description\": \"The destination city.\"},\n",
+    "                \"date\": {\"type\": \"string\", \"description\": \"The date of the flight.\"},\n",
+    "            },\n",
+    "        },\n",
+    "    }\n",
+    "]\n",
+    "\n",
+    "# Append to evaluation_contents\n",
+    "evaluation_contents.append(\n",
+    "    SourceFileContentContent(\n",
+    "        item={\n",
+    "            \"query\": query_str,\n",
+    "            \"response\": response_str,\n",
+    "            \"tool_calls\": tool_calls_str,\n",
+    "            \"tool_definitions\": tool_definitions_str\n",
+    "        }\n",
+    "    )\n",
+    ")"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## Batch evaluate and visualize results on Azure AI Foundry\n",
-    "Batch evaluate to leverage asynchronous evaluation on a dataset. \n",
+    "#### Response as List[dict] with Tool Definition as Single Dict"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Test Case 5: Tool definition as single dict\n",
+    "query_dict = \"What's the weather in San Francisco?\"\n",
     "\n",
-    "Optionally, you can go to AI Foundry URL for rich Azure AI Foundry data visualization. You can inspect the evaluation scores and reasoning to quickly identify bugs and issues of your agent to fix and improve. Make sure to authenticate to Azure using `az login` in your terminal before running this cell.\n"
+    "response_dict = [\n",
+    "    {\n",
+    "        \"createdAt\": \"2025-03-26T18:15:22Z\",\n",
+    "        \"run_id\": \"run_abc123\",\n",
+    "        \"role\": \"assistant\",\n",
+    "        \"content\": [\n",
+    "            {\n",
+    "                \"type\": \"tool_call\",\n",
+    "                \"tool_call_id\": \"call_sf_weather_789\",\n",
+    "                \"name\": \"fetch_weather\",\n",
+    "                \"arguments\": {\"location\": \"San Francisco\"},\n",
+    "            }\n",
+    "        ],\n",
+    "    },\n",
+    "    {\n",
+    "        \"createdAt\": \"2025-03-26T18:15:24Z\",\n",
+    "        \"run_id\": \"run_abc123\",\n",
+    "        \"tool_call_id\": \"call_sf_weather_789\",\n",
+    "        \"role\": \"tool\",\n",
+    "        \"content\": [{\"type\": \"tool_result\", \"tool_result\": {\"weather\": \"Foggy, 18°C\"}}],\n",
+    "    },\n",
+    "    {\n",
+    "        \"createdAt\": \"2025-03-26T18:15:25Z\",\n",
+    "        \"run_id\": \"run_abc123\",\n",
+    "        \"role\": \"assistant\",\n",
+    "        \"content\": [\n",
+    "            {\n",
+    "                \"type\": \"text\",\n",
+    "                \"text\": \"The weather in San Francisco is currently foggy with a temperature of 18°C.\",\n",
+    "            }\n",
+    "        ],\n",
+    "    },\n",
+    "]\n",
+    "\n",
+    "tool_definition_dict = [\n",
+    "    {\n",
+    "        \"name\": \"fetch_weather\",\n",
+    "        \"description\": \"Fetches the weather information for the specified location.\",\n",
+    "        \"parameters\": {\n",
+    "            \"type\": \"object\",\n",
+    "            \"properties\": {\"location\": {\"type\": \"string\", \"description\": \"The location to fetch weather for.\"}},\n",
+    "        },\n",
+    "    }\n",
+    "]\n",
+    "\n",
+    "# Append to evaluation_contents\n",
+    "evaluation_contents.append(\n",
+    "    SourceFileContentContent(\n",
+    "        item={\n",
+    "            \"query\": query_dict,\n",
+    "            \"response\": response_dict,\n",
+    "            \"tool_definitions\": tool_definition_dict,\n",
+    "            \"tool_calls\": None\n",
+    "        }\n",
+    "    )\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "#### Query as Conversation History (List of Messages)\n",
+    "The evaluator also supports query as a list of messages representing conversation history. This helps determine if the Agent used the information in the conversation history to make the right tool calls."
    ]
   },
   {
@@ -313,27 +521,121 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from azure.ai.evaluation import evaluate\n",
+    "# Test Case 6: Query as conversation history\n",
+    "query_conversation = [\n",
+    "    {\n",
+    "        \"role\": \"system\",\n",
+    "        \"content\": \"You are a helpful assistant that can fetch weather information and send emails.\"\n",
+    "    },\n",
+    "    {\n",
+    "        \"role\": \"user\", \n",
+    "        \"content\": \"Hi, can you check the weather in Seattle for me?\"\n",
+    "    },\n",
+    "    {\n",
+    "        \"role\": \"user\",\n",
+    "        \"content\": \"Actually, could you also send me an email with that weather information to john@example.com?\"\n",
+    "    }\n",
+    "]\n",
     "\n",
-    "# This sample files contains the evaluation data in JSONL format. Where each line is a run from agent.\n",
-    "# This was saved using agent thread and converter.\n",
-    "file_name = \"evaluation_data.jsonl\"\n",
+    "tool_calls_conversation = [\n",
+    "    {\n",
+    "        \"type\": \"tool_call\",\n",
+    "        \"tool_call_id\": \"call_weather_123\",\n",
+    "        \"name\": \"fetch_weather\",\n",
+    "        \"arguments\": {\"location\": \"Seattle\"},\n",
+    "    },\n",
+    "    {\n",
+    "        \"type\": \"tool_call\", \n",
+    "        \"tool_call_id\": \"call_email_456\",\n",
+    "        \"name\": \"send_email\",\n",
+    "        \"arguments\": {\n",
+    "            \"recipient\": \"john@example.com\",\n",
+    "            \"subject\": \"Weather Information for Seattle\",\n",
+    "            \"body\": \"Here is the weather information you requested.\"\n",
+    "        },\n",
+    "    },\n",
+    "]\n",
     "\n",
-    "response = evaluate(\n",
-    "    data=file_name,\n",
-    "    evaluation_name=\"Tool Call Accuracy Evaluation\",\n",
-    "    evaluators={\n",
-    "        \"tool_call_accuracy\": tool_call_accuracy,\n",
+    "tool_definitions_conversation = [\n",
+    "    {\n",
+    "        \"name\": \"fetch_weather\",\n",
+    "        \"description\": \"Fetches the weather information for the specified location.\",\n",
+    "        \"parameters\": {\n",
+    "            \"type\": \"object\",\n",
+    "            \"properties\": {\"location\": {\"type\": \"string\", \"description\": \"The location to fetch weather for.\"}},\n",
+    "        },\n",
+    "    },\n",
+    "    {\n",
+    "        \"name\": \"send_email\",\n",
+    "        \"description\": \"Sends an email with the specified subject and body to the recipient.\",\n",
+    "        \"parameters\": {\n",
+    "            \"type\": \"object\",\n",
+    "            \"properties\": {\n",
+    "                \"recipient\": {\"type\": \"string\", \"description\": \"Email address of the recipient.\"},\n",
+    "                \"subject\": {\"type\": \"string\", \"description\": \"Subject of the email.\"},\n",
+    "                \"body\": {\"type\": \"string\", \"description\": \"Body content of the email.\"},\n",
+    "            },\n",
+    "        },\n",
     "    },\n",
-    "    azure_ai_project=os.environ[\"AZURE_AI_PROJECT\"],\n",
-    ")\n",
-    "pprint(f'AI Foundary URL: {response.get(\"studio_url\")}')"
+    "]\n",
+    "\n",
+    "# Append to evaluation_contents\n",
+    "evaluation_contents.append(\n",
+    "    SourceFileContentContent(\n",
+    "        item={\n",
+    "            \"query\": query_conversation,\n",
+    "            \"tool_calls\": tool_calls_conversation,\n",
+    "            \"tool_definitions\": tool_definitions_conversation,\n",
+    "            \"response\": None\n",
+    "        }\n",
+    "    )\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Execute Batch Evaluation\n",
+    "Run all test cases together using the batch evaluation approach."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "results = run_evaluator(\n",
+    "    evaluator_name=\"tool_call_accuracy\",\n",
+    "    evaluation_contents=evaluation_contents,\n",
+    "    data_source_config=data_source_config,\n",
+    "    initialization_parameters=initialization_parameters,\n",
+    "    data_mapping=data_mapping\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Results Display\n",
+    "Display the evaluation results for each test case."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "pprint(results)"
    ]
   }
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "azureai-sample",
+   "display_name": "samples",
    "language": "python",
    "name": "python3"
   },
@@ -346,7 +648,8 @@
    "mimetype": "text/x-python",
    "name": "python",
    "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3"
+   "pygments_lexer": "ipython3",
+   "version": "3.10.19"
   }
  },
  "nbformat": 4,
diff --git a/scenarios/evaluate/Supported_Evaluation_Metrics/Agent_Evaluation/AI_Judge_Evaluator_Tool_Call_Success.ipynb b/scenarios/evaluate/Supported_Evaluation_Metrics/Agent_Evaluation/AI_Judge_Evaluator_Tool_Call_Success.ipynb
new file mode 100644
index 00000000..6020dd32
--- /dev/null
+++ b/scenarios/evaluate/Supported_Evaluation_Metrics/Agent_Evaluation/AI_Judge_Evaluator_Tool_Call_Success.ipynb
@@ -0,0 +1,604 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "cea11cae",
+   "metadata": {},
+   "source": [
+    "# Tool Call Success Evaluator\n",
+    "\n",
+    "## Objective\n",
+    "This sample demonstrates how to use tool success evaluator on agent data. The supported input formats include:\n",
+    "- simple data such as strings and `dict` describing agent responses;\n",
+    "- user-agent conversations in the form of list of agent messages. \n",
+    "\n",
+    "## Time\n",
+    "\n",
+    "You should expect to spend about 20 minutes running this notebook. \n",
+    "\n",
+    "## Before you begin\n",
+    "For quality evaluation, you need to deploy a `gpt` model supporting JSON mode. We recommend using `gpt-4.1-mini`.    \n",
+    "\n",
+    "### Prerequisite\n",
+    "```bash\n",
+    "pip install azure-ai-projects azure-identity openai\n",
+    "```\n",
+    "Set these environment variables with your own values:\n",
+    "1) **AZURE_AI_PROJECT_ENDPOINT** - Your Azure AI project endpoint in format: `https://<account_name>.services.ai.azure.com/api/projects/<project_name>`\n",
+    "2) **AZURE_AI_MODEL_DEPLOYMENT_NAME** - The deployment name of the model for this AI-assisted evaluator (e.g., gpt-4o-mini)\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "49404a44",
+   "metadata": {},
+   "source": [
+    "The Tool Call Success evaluator determines whether tool calls done by an AI agent includes failures or not.\n",
+    "\n",
+    "This evaluator focuses solely on tool call results and tool definitions, disregarding user's query to the agent, conversation history and agent's final response. Although tool definitions is optional, providing them can help the evaluator better understand the context of the tool calls made by the agent. Please note that this evaluator validates tool calls for potential technical failures like errors, exceptions, timeouts and empty results (only in cases where empty results could indicate a failure). It does not assess the correctness or the tool result itself, like mathematical errors and unrealistic field values like name=\"668656\".\n",
+    "\n",
+    "Scoring is binary:\n",
+    "- TRUE: All tool calls were successful\n",
+    "- FALSE: At least one tool call failed\n",
+    "\n",
+    "This evaluation focuses on measuring the technical success of tool execution, not the semantic correctness of the results."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "d91cb876",
+   "metadata": {},
+   "source": [
+    "Tool Call Success requires following input:\n",
+    "- Response - Response from Agent (or any GenAI App). This can be a single text response or a list of messages generated as part of Agent Response. The evaluator examines tool call results within the response.\n",
+    "- Tool Definitions - (Optional) Tool(s) definition used by Agent. Providing tool definitions helps the evaluator better understand the context of the tool calls made by the agent.\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "e07d293d",
+   "metadata": {},
+   "source": [
+    "### Initialize Tool Call Success Evaluator\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "919d1b9a",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import os\n",
+    "from openai.types.evals.create_eval_jsonl_run_data_source_param import SourceFileContentContent\n",
+    "from pprint import pprint\n",
+    "from agent_utils import run_evaluator\n",
+    "\n",
+    "# Get environment variables\n",
+    "deployment_name = os.environ[\"AZURE_AI_MODEL_DEPLOYMENT_NAME\"]\n",
+    "\n",
+    "# Data source configuration (defines the schema for evaluation inputs)\n",
+    "data_source_config = {\n",
+    "    \"type\": \"custom\",\n",
+    "    \"item_schema\": {\n",
+    "        \"type\": \"object\",\n",
+    "        \"properties\": {\n",
+    "            \"tool_definitions\": {\n",
+    "                \"anyOf\": [\n",
+    "                    {\"type\": \"object\"},\n",
+    "                    {\n",
+    "                        \"type\": \"array\",\n",
+    "                        \"items\": {\n",
+    "                            \"type\": \"object\"\n",
+    "                        }\n",
+    "                    }\n",
+    "                ]\n",
+    "            },\n",
+    "            \"response\": {\n",
+    "                \"anyOf\": [\n",
+    "                    {\"type\": \"string\"},\n",
+    "                    {\n",
+    "                        \"type\": \"array\",\n",
+    "                        \"items\": {\n",
+    "                            \"type\": \"object\"\n",
+    "                        }\n",
+    "                    }\n",
+    "                ]\n",
+    "            }\n",
+    "        },\n",
+    "        \"required\": [\"response\"]\n",
+    "    },\n",
+    "    \"include_sample_schema\": True\n",
+    "}\n",
+    "\n",
+    "# Data mapping (maps evaluation inputs to evaluator parameters)\n",
+    "data_mapping = {\n",
+    "    \"tool_definitions\": \"{{item.tool_definitions}}\",\n",
+    "    \"response\": \"{{item.response}}\"\n",
+    "}\n",
+    "\n",
+    "# Initialization parameters for the evaluator\n",
+    "initialization_parameters = {\n",
+    "    \"deployment_name\": deployment_name\n",
+    "}\n",
+    "\n",
+    "# Initialize the evaluation_contents list - we'll append all test cases here\n",
+    "evaluation_contents = []\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "4739878c",
+   "metadata": {},
+   "source": [
+    "### Samples\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "2352536d",
+   "metadata": {},
+   "source": [
+    "#### Evaluating Successful Tool Calls"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "9ddc2e22",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Test Case 1: Successful tool execution\n",
+    "successful_response = [\n",
+    "    {\n",
+    "        \"role\": \"assistant\",\n",
+    "        \"content\": [\n",
+    "            {\n",
+    "                \"type\": \"tool_call\",\n",
+    "                \"tool_call_id\": \"call_weather_123\",\n",
+    "                \"name\": \"fetch_weather\",\n",
+    "                \"arguments\": {\"location\": \"Seattle\"},\n",
+    "            }\n",
+    "        ],\n",
+    "    },\n",
+    "    {\n",
+    "        \"tool_call_id\": \"call_weather_123\",\n",
+    "        \"role\": \"tool\",\n",
+    "        \"content\": [{\"type\": \"tool_result\", \"tool_result\": {\"temperature\": \"15°C\", \"condition\": \"partly cloudy\", \"humidity\": \"68%\", \"wind\": \"8 mph NW\"}}],\n",
+    "    },\n",
+    "    {\n",
+    "        \"role\": \"assistant\",\n",
+    "        \"content\": [\n",
+    "            {\n",
+    "                \"type\": \"text\",\n",
+    "                \"text\": \"The current weather in Seattle is partly cloudy with a temperature of 15°C.\",\n",
+    "            }\n",
+    "        ],\n",
+    "    }\n",
+    "]\n",
+    "\n",
+    "tool_definitions = [\n",
+    "    {\n",
+    "        \"name\": \"fetch_weather\",\n",
+    "        \"description\": \"Fetches the weather information for the specified location.\",\n",
+    "        \"parameters\": {\n",
+    "            \"type\": \"object\",\n",
+    "            \"properties\": {\"location\": {\"type\": \"string\", \"description\": \"The location to fetch weather for.\"}},\n",
+    "        },\n",
+    "    }\n",
+    "]\n",
+    "\n",
+    "# Append to evaluation_contents\n",
+    "evaluation_contents.append(\n",
+    "    SourceFileContentContent(\n",
+    "        item={\n",
+    "            \"tool_definitions\": tool_definitions,\n",
+    "            \"response\": successful_response\n",
+    "        }\n",
+    "    )\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "c146de06",
+   "metadata": {},
+   "source": [
+    "#### Response as String (str)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "575d33eb",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Test Case 2: Response as a simple string containing tool call information\n",
+    "response_str = \"\"\"Tool call executed: fetch_weather(location=\"Paris\")\n",
+    "Tool result: {\"temperature\": \"18°C\", \"condition\": \"clear sky\", \"humidity\": \"55%\"}\n",
+    "The weather in Paris is currently clear with a temperature of 18°C.\"\"\"\n",
+    "\n",
+    "tool_definition = [\n",
+    "    {\n",
+    "        \"name\": \"fetch_weather\",\n",
+    "        \"description\": \"Fetches the weather information for the specified location.\",\n",
+    "        \"parameters\": {\n",
+    "            \"type\": \"object\",\n",
+    "            \"properties\": {\"location\": {\"type\": \"string\", \"description\": \"The location to fetch weather for.\"}},\n",
+    "        },\n",
+    "    }\n",
+    "]\n",
+    "\n",
+    "# Append to evaluation_contents\n",
+    "evaluation_contents.append(\n",
+    "    SourceFileContentContent(\n",
+    "        item={\n",
+    "            \"tool_definitions\": tool_definition,\n",
+    "            \"response\": response_str\n",
+    "        }\n",
+    "    )\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "b7217ddc",
+   "metadata": {},
+   "source": [
+    "#### Tool Definition as Single Dict"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "f89ca010",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Test Case 3: Successful tool execution with single tool definition dict\n",
+    "successful_response_nyc = [\n",
+    "    {\n",
+    "        \"role\": \"assistant\",\n",
+    "        \"content\": [\n",
+    "            {\n",
+    "                \"type\": \"tool_call\",\n",
+    "                \"tool_call_id\": \"call_nyc_weather\",\n",
+    "                \"name\": \"fetch_weather\",\n",
+    "                \"arguments\": {\"location\": \"New York\"},\n",
+    "            }\n",
+    "        ],\n",
+    "    },\n",
+    "    {\n",
+    "        \"tool_call_id\": \"call_nyc_weather\",\n",
+    "        \"role\": \"tool\",\n",
+    "        \"content\": [{\"type\": \"tool_result\", \"tool_result\": {\"temperature\": \"22°C\", \"condition\": \"sunny\", \"humidity\": \"45%\"}}],\n",
+    "    },\n",
+    "    {\n",
+    "        \"role\": \"assistant\",\n",
+    "        \"content\": [\n",
+    "            {\n",
+    "                \"type\": \"text\",\n",
+    "                \"text\": \"The weather in New York is currently sunny with a temperature of 22°C.\",\n",
+    "            }\n",
+    "        ],\n",
+    "    }\n",
+    "]\n",
+    "\n",
+    "tool_definition_dict = [\n",
+    "    {\n",
+    "        \"name\": \"fetch_weather\",\n",
+    "        \"description\": \"Fetches the weather information for the specified location.\",\n",
+    "        \"parameters\": {\n",
+    "            \"type\": \"object\",\n",
+    "            \"properties\": {\"location\": {\"type\": \"string\", \"description\": \"The location to fetch weather for.\"}},\n",
+    "        },\n",
+    "    }\n",
+    "]\n",
+    "\n",
+    "# Append to evaluation_contents\n",
+    "evaluation_contents.append(\n",
+    "    SourceFileContentContent(\n",
+    "        item={\n",
+    "            \"tool_definitions\": tool_definition_dict,\n",
+    "            \"response\": successful_response_nyc\n",
+    "        }\n",
+    "    )\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "1aeb8c11",
+   "metadata": {},
+   "source": [
+    "#### Multiple Successful Tool Calls"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "444ee59c",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Test Case 4: Multiple successful tool executions\n",
+    "multiple_successful_response = [\n",
+    "    {\n",
+    "        \"createdAt\": \"2025-03-26T17:27:35Z\",\n",
+    "        \"run_id\": \"run_zblZyGCNyx6aOYTadmaqM4QN\",\n",
+    "        \"role\": \"assistant\",\n",
+    "        \"content\": [\n",
+    "            {\n",
+    "                \"type\": \"tool_call\",\n",
+    "                \"tool_call_id\": \"call_CUdbkBfvVBla2YP3p24uhElJ\",\n",
+    "                \"name\": \"fetch_weather\",\n",
+    "                \"arguments\": {\"location\": \"Seattle\"},\n",
+    "            }\n",
+    "        ],\n",
+    "    },\n",
+    "    {\n",
+    "        \"createdAt\": \"2025-03-26T17:27:37Z\",\n",
+    "        \"run_id\": \"run_zblZyGCNyx6aOYTadmaqM4QN\",\n",
+    "        \"tool_call_id\": \"call_CUdbkBfvVBla2YP3p24uhElJ\",\n",
+    "        \"role\": \"tool\",\n",
+    "        \"content\": [{\"type\": \"tool_result\", \"tool_result\": {\"temperature\": \"14°C\", \"condition\": \"rainy\", \"precipitation\": \"light rain\"}}],\n",
+    "    },\n",
+    "    {\n",
+    "        \"createdAt\": \"2025-03-26T17:27:38Z\",\n",
+    "        \"run_id\": \"run_zblZyGCNyx6aOYTadmaqM4QN\",\n",
+    "        \"role\": \"assistant\",\n",
+    "        \"content\": [\n",
+    "            {\n",
+    "                \"type\": \"tool_call\",\n",
+    "                \"tool_call_id\": \"call_iq9RuPxqzykebvACgX8pqRW2\",\n",
+    "                \"name\": \"send_email\",\n",
+    "                \"arguments\": {\n",
+    "                    \"recipient\": \"your_email@example.com\",\n",
+    "                    \"subject\": \"Weather Information for Seattle\",\n",
+    "                    \"body\": \"Current weather: 14°C and rainy with light rain.\",\n",
+    "                },\n",
+    "            }\n",
+    "        ],\n",
+    "    },\n",
+    "    {\n",
+    "        \"createdAt\": \"2025-03-26T17:27:41Z\",\n",
+    "        \"run_id\": \"run_zblZyGCNyx6aOYTadmaqM4QN\",\n",
+    "        \"tool_call_id\": \"call_iq9RuPxqzykebvACgX8pqRW2\",\n",
+    "        \"role\": \"tool\",\n",
+    "        \"content\": [\n",
+    "            {\"type\": \"tool_result\", \"tool_result\": {\"message\": \"Email successfully sent to your_email@example.com.\", \"status\": \"delivered\"}}\n",
+    "        ],\n",
+    "    },\n",
+    "    {\n",
+    "        \"createdAt\": \"2025-03-26T17:27:42Z\",\n",
+    "        \"run_id\": \"run_zblZyGCNyx6aOYTadmaqM4QN\",\n",
+    "        \"role\": \"assistant\",\n",
+    "        \"content\": [\n",
+    "            {\n",
+    "                \"type\": \"text\",\n",
+    "                \"text\": \"I have successfully sent you an email with the weather information for Seattle.\",\n",
+    "            }\n",
+    "        ],\n",
+    "    },\n",
+    "]\n",
+    "\n",
+    "tool_definitions_multiple = [\n",
+    "    {\n",
+    "        \"name\": \"fetch_weather\",\n",
+    "        \"description\": \"Fetches the weather information for the specified location.\",\n",
+    "        \"parameters\": {\n",
+    "            \"type\": \"object\",\n",
+    "            \"properties\": {\"location\": {\"type\": \"string\", \"description\": \"The location to fetch weather for.\"}},\n",
+    "        },\n",
+    "    },\n",
+    "    {\n",
+    "        \"name\": \"send_email\",\n",
+    "        \"description\": \"Sends an email with the specified subject and body to the recipient.\",\n",
+    "        \"parameters\": {\n",
+    "            \"type\": \"object\",\n",
+    "            \"properties\": {\n",
+    "                \"recipient\": {\"type\": \"string\", \"description\": \"Email address of the recipient.\"},\n",
+    "                \"subject\": {\"type\": \"string\", \"description\": \"Subject of the email.\"},\n",
+    "                \"body\": {\"type\": \"string\", \"description\": \"Body content of the email.\"},\n",
+    "            },\n",
+    "        },\n",
+    "    },\n",
+    "]\n",
+    "\n",
+    "# Append to evaluation_contents\n",
+    "evaluation_contents.append(\n",
+    "    SourceFileContentContent(\n",
+    "        tool_definitions=tool_definitions_multiple,\n",
+    "        response=multiple_successful_response\n",
+    "    )\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "63e43736",
+   "metadata": {},
+   "source": [
+    "#### Tool Call Success without Tool Definitions"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "a0ca062e",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Test Case 5: Successful execution without providing tool definitions\n",
+    "simple_successful_response = [\n",
+    "    {\n",
+    "        \"role\": \"assistant\",\n",
+    "        \"content\": [\n",
+    "            {\n",
+    "                \"type\": \"tool_call\",\n",
+    "                \"tool_call_id\": \"call_calendar_123\",\n",
+    "                \"name\": \"get_calendar\",\n",
+    "                \"arguments\": {\"start_date\": \"2025-01-01\", \"end_date\": \"2025-01-31\"},\n",
+    "            }\n",
+    "        ],\n",
+    "    },\n",
+    "    {\n",
+    "        \"tool_call_id\": \"call_calendar_123\",\n",
+    "        \"role\": \"tool\",\n",
+    "        \"content\": [{\"type\": \"tool_result\", \"tool_result\": {\"events\": [{\"date\": \"2025-01-15\", \"title\": \"Team Meeting\"}, {\"date\": \"2025-01-20\", \"title\": \"Project Review\"}]}}],\n",
+    "    },\n",
+    "    {\n",
+    "        \"role\": \"assistant\",\n",
+    "        \"content\": [\n",
+    "            {\n",
+    "                \"type\": \"text\",\n",
+    "                \"text\": \"I found 2 events in your calendar for January: a Team Meeting on the 15th and a Project Review on the 20th.\",\n",
+    "            }\n",
+    "        ],\n",
+    "    }\n",
+    "]\n",
+    "\n",
+    "# Append without tool definitions - evaluator should still work\n",
+    "evaluation_contents.append(\n",
+    "    SourceFileContentContent(\n",
+    "        tool_definitions=None,\n",
+    "        response=simple_successful_response\n",
+    "    )\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "97c1af0f",
+   "metadata": {},
+   "source": [
+    "#### Example of Failed Tool Call"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "cafdee8e",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Test Case 6: Failed tool execution with error\n",
+    "failed_response = [\n",
+    "    {\n",
+    "        \"role\": \"assistant\",\n",
+    "        \"content\": [\n",
+    "            {\n",
+    "                \"type\": \"tool_call\",\n",
+    "                \"tool_call_id\": \"call_weather_456\",\n",
+    "                \"name\": \"fetch_weather\",\n",
+    "                \"arguments\": {\"location\": \"InvalidCity\"},\n",
+    "            }\n",
+    "        ],\n",
+    "    },\n",
+    "    {\n",
+    "        \"tool_call_id\": \"call_weather_456\",\n",
+    "        \"role\": \"tool\",\n",
+    "        \"content\": [{\"type\": \"tool_result\", \"tool_result\": {\"error\": \"Location not found\", \"status\": \"failed\", \"code\": 404}}],\n",
+    "    },\n",
+    "    {\n",
+    "        \"role\": \"assistant\",\n",
+    "        \"content\": [\n",
+    "            {\n",
+    "                \"type\": \"text\",\n",
+    "                \"text\": \"I'm sorry, I couldn't retrieve the weather information due to an error.\",\n",
+    "            }\n",
+    "        ],\n",
+    "    }\n",
+    "]\n",
+    "\n",
+    "tool_definitions_failed = [\n",
+    "    {\n",
+    "        \"name\": \"fetch_weather\",\n",
+    "        \"description\": \"Fetches the weather information for the specified location.\",\n",
+    "        \"parameters\": {\n",
+    "            \"type\": \"object\",\n",
+    "            \"properties\": {\"location\": {\"type\": \"string\", \"description\": \"The location to fetch weather for.\"}},\n",
+    "        },\n",
+    "    }\n",
+    "]\n",
+    "\n",
+    "# This should score FALSE due to tool failure\n",
+    "evaluation_contents.append(\n",
+    "    SourceFileContentContent(\n",
+    "        tool_definitions=tool_definitions_failed,\n",
+    "        response=failed_response\n",
+    "    )\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "4bdea268",
+   "metadata": {},
+   "source": [
+    "### Run Evaluation on All Test Cases\n",
+    "\n",
+    "Now that we've defined all test cases, let's run the evaluation once on all of them.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "9f9783c0",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "results = run_evaluator(\n",
+    "    evaluator_name=\"tool_call_success\",\n",
+    "    evaluation_contents=evaluation_contents,\n",
+    "    data_source_config=data_source_config,\n",
+    "    initialization_parameters=initialization_parameters,\n",
+    "    data_mapping=data_mapping\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "8d7fecd5",
+   "metadata": {},
+   "source": [
+    "### Display Results\n",
+    "\n",
+    "View the evaluation results for each test case.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "8e4fdb3a",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "pprint(results)"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "samples",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.19"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
diff --git a/scenarios/evaluate/Supported_Evaluation_Metrics/Agent_Evaluation/AI_Judge_Evaluator_Tool_Input_Accuracy.ipynb b/scenarios/evaluate/Supported_Evaluation_Metrics/Agent_Evaluation/AI_Judge_Evaluator_Tool_Input_Accuracy.ipynb
new file mode 100644
index 00000000..16d6e4b8
--- /dev/null
+++ b/scenarios/evaluate/Supported_Evaluation_Metrics/Agent_Evaluation/AI_Judge_Evaluator_Tool_Input_Accuracy.ipynb
@@ -0,0 +1,589 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "8d6b6cf9",
+   "metadata": {},
+   "source": [
+    "# Tool Input Accuracy Evaluator\n",
+    "\n",
+    "## Objective\n",
+    "This sample demonstrates how to use tool input accuracy evaluator on agent data. The supported input formats include:\n",
+    "- simple data such as strings and `dict` describing tool calls;\n",
+    "- user-agent conversations in the form of list of agent messages. \n",
+    "\n",
+    "## Time\n",
+    "\n",
+    "You should expect to spend about 20 minutes running this notebook. \n",
+    "\n",
+    "## Before you begin\n",
+    "For quality evaluation, you need to deploy a `gpt` model supporting JSON mode. We recommend using `gpt-4o`.    \n",
+    "\n",
+    "### Prerequisite\n",
+    "```bash\n",
+    "pip install azure-ai-projects azure-identity openai\n",
+    "```\n",
+    "Set these environment variables with your own values:\n",
+    "1) **AZURE_AI_PROJECT_ENDPOINT** - Your Azure AI project endpoint in format: `https://<account_name>.services.ai.azure.com/api/projects/<project_name>`\n",
+    "2) **AZURE_AI_MODEL_DEPLOYMENT_NAME** - The deployment name of the model for this AI-assisted evaluator (e.g., gpt-4o-mini)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "74ac1b5c",
+   "metadata": {},
+   "source": [
+    "The Tool Input Accuracy evaluator performs a strict binary evaluation (PASS/FAIL) of parameters passed to tool calls. It ensures that ALL parameters meet ALL criteria:\n",
+    "\n",
+    "- Parameter grounding: All parameters must be derived from conversation history/query\n",
+    "- Type compliance: All parameters must match exact types specified in tool definitions\n",
+    "- Format compliance: All parameters must follow exact format and structure requirements\n",
+    "- Completeness: All required parameters must be provided\n",
+    "- No unexpected parameters: Only defined parameters are allowed\n",
+    "\n",
+    "The evaluator uses strict binary evaluation:\n",
+    "\n",
+    "    - 1: Only when ALL criteria are satisfied perfectly for ALL parameters\n",
+    "    - 0: When ANY criterion fails for ANY parameter\n",
+    "\n",
+    "This evaluation focuses on ensuring tool call parameters are completely correct without any tolerance for partial correctness."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "17346632",
+   "metadata": {},
+   "source": [
+    "Tool Input Accuracy requires following input:\n",
+    "- Query - This can be a single query or a list of messages(conversation history with agent). The original task request from the user.\n",
+    "- Response - Response from Agent (or any GenAI App). This can be a single text response or a list of messages generated as part of Agent Response. The evaluator extracts tool calls from the response.\n",
+    "- Tool Definitions - Tool(s) definition used by Agent to answer the query. Required to validate parameter types and structures.\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "d78194ee",
+   "metadata": {},
+   "source": [
+    "### Initialize Tool Input Accuracy Evaluator\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "1663a6b1",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import os\n",
+    "from openai.types.evals.create_eval_jsonl_run_data_source_param import SourceFileContentContent\n",
+    "from pprint import pprint\n",
+    "from agent_utils import run_evaluator\n",
+    "\n",
+    "# Get environment variables\n",
+    "deployment_name = os.environ[\"AZURE_AI_MODEL_DEPLOYMENT_NAME\"]\n",
+    "\n",
+    "# Data source configuration (defines the schema for evaluation inputs)\n",
+    "data_source_config = {\n",
+    "    \"type\": \"custom\",\n",
+    "    \"item_schema\": {\n",
+    "        \"type\": \"object\",\n",
+    "        \"properties\": {\n",
+    "            \"query\": {\n",
+    "                \"anyOf\": [\n",
+    "                    {\"type\": \"string\"},\n",
+    "                    {\n",
+    "                        \"type\": \"array\",\n",
+    "                        \"items\": {\n",
+    "                            \"type\": \"object\"\n",
+    "                        }\n",
+    "                    }\n",
+    "                ]\n",
+    "            },\n",
+    "            \"response\": {\n",
+    "                \"anyOf\": [\n",
+    "                    {\"type\": \"string\"},\n",
+    "                    {\n",
+    "                        \"type\": \"array\",\n",
+    "                        \"items\": {\n",
+    "                            \"type\": \"object\"\n",
+    "                        }\n",
+    "                    }\n",
+    "                ]\n",
+    "            },\n",
+    "            \"tool_definitions\": {\n",
+    "                \"anyOf\": [\n",
+    "                    {\"type\": \"object\"},\n",
+    "                    {\n",
+    "                        \"type\": \"array\",\n",
+    "                        \"items\": {\n",
+    "                            \"type\": \"object\"\n",
+    "                        }\n",
+    "                    }\n",
+    "                ]\n",
+    "            }\n",
+    "        },\n",
+    "        \"required\": [\"query\", \"response\", \"tool_definitions\"]\n",
+    "    },\n",
+    "    \"include_sample_schema\": True\n",
+    "}\n",
+    "\n",
+    "# Data mapping (maps evaluation inputs to evaluator parameters)\n",
+    "data_mapping = {\n",
+    "    \"query\": \"{{item.query}}\",\n",
+    "    \"response\": \"{{item.response}}\",\n",
+    "    \"tool_definitions\": \"{{item.tool_definitions}}\"\n",
+    "}\n",
+    "\n",
+    "# Initialization parameters for the evaluator\n",
+    "initialization_parameters = {\n",
+    "    \"deployment_name\": deployment_name\n",
+    "}\n",
+    "\n",
+    "# Initialize the evaluation_contents list - we'll append all test cases here\n",
+    "evaluation_contents = []"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "f92e8378",
+   "metadata": {},
+   "source": [
+    "### Samples"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "ba5f6441",
+   "metadata": {},
+   "source": [
+    "#### Evaluating Correct Tool Input Parameters"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "486057ca",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Test Case 1: Correct tool input parameters\n",
+    "query = \"How is the weather in Seattle?\"\n",
+    "response = [\n",
+    "    {\n",
+    "        \"role\": \"assistant\",\n",
+    "        \"content\": [\n",
+    "            {\n",
+    "                \"type\": \"tool_call\",\n",
+    "                \"tool_call_id\": \"call_weather_123\",\n",
+    "                \"name\": \"fetch_weather\",\n",
+    "                \"arguments\": {\"location\": \"Seattle\"},\n",
+    "            }\n",
+    "        ],\n",
+    "    }\n",
+    "]\n",
+    "\n",
+    "tool_definitions = [\n",
+    "    {\n",
+    "        \"name\": \"fetch_weather\",\n",
+    "        \"description\": \"Fetches the weather information for the specified location.\",\n",
+    "        \"parameters\": {\n",
+    "            \"type\": \"object\",\n",
+    "            \"properties\": {\"location\": {\"type\": \"string\", \"description\": \"The location to fetch weather for.\"}},\n",
+    "            \"required\": [\"location\"]\n",
+    "        },\n",
+    "    }\n",
+    "]\n",
+    "\n",
+    "# Append to evaluation_contents\n",
+    "evaluation_contents.append(\n",
+    "    SourceFileContentContent(\n",
+    "        item={\n",
+    "            \"query\": query,\n",
+    "            \"response\": response,\n",
+    "            \"tool_definitions\": tool_definitions\n",
+    "        }\n",
+    "    )\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "80fc6e82",
+   "metadata": {},
+   "source": [
+    "#### Response as String (str)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "626462d3",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Test Case 2: Response as string\n",
+    "query_str = \"Check the weather in Miami\"\n",
+    "\n",
+    "# Response as a simple string containing tool call information\n",
+    "response_str = \"I'll check the weather for you. Calling fetch_weather with location=\\\"Miami\\\".\"\n",
+    "\n",
+    "tool_definition = [{\n",
+    "    \"name\": \"fetch_weather\",\n",
+    "    \"description\": \"Fetches the weather information for the specified location.\",\n",
+    "    \"parameters\": {\n",
+    "        \"type\": \"object\",\n",
+    "        \"properties\": {\"location\": {\"type\": \"string\", \"description\": \"The location to fetch weather for.\"}},\n",
+    "        \"required\": [\"location\"]\n",
+    "    },\n",
+    "}]\n",
+    "\n",
+    "# Append to evaluation_contents\n",
+    "evaluation_contents.append(\n",
+    "    SourceFileContentContent(\n",
+    "        item={\n",
+    "            \"query\": query_str,\n",
+    "            \"response\": response_str,\n",
+    "            \"tool_definitions\": tool_definition\n",
+    "        }\n",
+    "    )\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "e9275f1c",
+   "metadata": {},
+   "source": [
+    "#### Response as a List and Tool Definition as Single Dict"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "91e1e4d3",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Test Case 3: Tool definition as single dict\n",
+    "query_dict = \"What's the temperature in Boston?\"\n",
+    "response_dict = [\n",
+    "    {\n",
+    "        \"role\": \"assistant\",\n",
+    "        \"content\": [\n",
+    "            {\n",
+    "                \"type\": \"tool_call\",\n",
+    "                \"tool_call_id\": \"call_boston_weather\",\n",
+    "                \"name\": \"fetch_weather\",\n",
+    "                \"arguments\": {\"location\": \"Boston\"},\n",
+    "            }\n",
+    "        ],\n",
+    "    }\n",
+    "]\n",
+    "\n",
+    "tool_definition_dict = [{\n",
+    "    \"name\": \"fetch_weather\",\n",
+    "    \"description\": \"Fetches the weather information for the specified location.\",\n",
+    "    \"parameters\": {\n",
+    "        \"type\": \"object\",\n",
+    "        \"properties\": {\"location\": {\"type\": \"string\", \"description\": \"The location to fetch weather for.\"}},\n",
+    "        \"required\": [\"location\"]\n",
+    "    },\n",
+    "}]\n",
+    "\n",
+    "# Append to evaluation_contents\n",
+    "evaluation_contents.append(\n",
+    "    SourceFileContentContent(\n",
+    "        item={\n",
+    "            \"query\": query_dict,\n",
+    "            \"response\": response_dict,\n",
+    "            \"tool_definitions\": tool_definition_dict\n",
+    "        }\n",
+    "    )\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "c351c61a",
+   "metadata": {},
+   "source": [
+    "#### Complex Tool Parameters with Multiple Fields"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "6b1147ab",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Test Case 4: Complex tool parameters with multiple fields\n",
+    "query_complex = \"Can you send an email to john@example.com with the subject 'Weather Update' and tell him the weather in Seattle is 15°C and partly cloudy?\"\n",
+    "response_complex = [\n",
+    "    {\n",
+    "        \"role\": \"assistant\",\n",
+    "        \"content\": [\n",
+    "            {\n",
+    "                \"type\": \"tool_call\",\n",
+    "                \"tool_call_id\": \"call_email_456\",\n",
+    "                \"name\": \"send_email\",\n",
+    "                \"arguments\": {\n",
+    "                    \"recipient\": \"john@example.com\",\n",
+    "                    \"subject\": \"Weather Update\",\n",
+    "                    \"body\": \"The weather in Seattle is 15°C and partly cloudy.\"\n",
+    "                },\n",
+    "            }\n",
+    "        ],\n",
+    "    }\n",
+    "]\n",
+    "\n",
+    "tool_definitions_complex = [\n",
+    "    {\n",
+    "        \"name\": \"send_email\",\n",
+    "        \"description\": \"Sends an email with the specified subject and body to the recipient.\",\n",
+    "        \"parameters\": {\n",
+    "            \"type\": \"object\",\n",
+    "            \"properties\": {\n",
+    "                \"recipient\": {\"type\": \"string\", \"description\": \"Email address of the recipient.\"},\n",
+    "                \"subject\": {\"type\": \"string\", \"description\": \"Subject of the email.\"},\n",
+    "                \"body\": {\"type\": \"string\", \"description\": \"Body content of the email.\"},\n",
+    "            },\n",
+    "            \"required\": [\"recipient\", \"subject\", \"body\"]\n",
+    "        },\n",
+    "    }\n",
+    "]\n",
+    "\n",
+    "# Append to evaluation_contents\n",
+    "evaluation_contents.append(\n",
+    "    SourceFileContentContent(\n",
+    "        item={\n",
+    "            \"query\": query_complex,\n",
+    "            \"response\": response_complex,\n",
+    "            \"tool_definitions\": tool_definitions_complex\n",
+    "        }\n",
+    "    )\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "2b274da2",
+   "metadata": {},
+   "source": [
+    "#### Query as Conversation History (List of Messages)\n",
+    "The evaluator also supports query as a list of messages representing conversation history. This helps validate parameters are grounded in the conversation context."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "a90e8827",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Test Case 5: Query as conversation history with multiple tools\n",
+    "query_conversation = [\n",
+    "    {\n",
+    "        \"role\": \"system\",\n",
+    "        \"content\": \"You are a helpful assistant that can fetch weather information and send emails.\"\n",
+    "    },\n",
+    "    {\n",
+    "        \"role\": \"user\", \n",
+    "        \"content\": \"Hi, can you check the weather in Seattle for me?\"\n",
+    "    },\n",
+    "    {\n",
+    "        \"role\": \"user\",\n",
+    "        \"content\": \"Please send the results to admin@company.com with the subject 'Daily Weather Report'.\"\n",
+    "    }\n",
+    "]\n",
+    "\n",
+    "response_conversation = [\n",
+    "    {\n",
+    "        \"role\": \"assistant\",\n",
+    "        \"content\": [\n",
+    "            {\n",
+    "                \"type\": \"tool_call\",\n",
+    "                \"tool_call_id\": \"call_weather_123\",\n",
+    "                \"name\": \"fetch_weather\", \n",
+    "                \"arguments\": {\"location\": \"Seattle\"},\n",
+    "            }\n",
+    "        ],\n",
+    "    },\n",
+    "    {\n",
+    "        \"role\": \"assistant\",\n",
+    "        \"content\": [\n",
+    "            {\n",
+    "                \"type\": \"tool_call\",\n",
+    "                \"tool_call_id\": \"call_email_456\",\n",
+    "                \"name\": \"send_email\",\n",
+    "                \"arguments\": {\n",
+    "                    \"recipient\": \"admin@company.com\",\n",
+    "                    \"subject\": \"Daily Weather Report\",\n",
+    "                    \"body\": \"Weather information for Seattle as requested.\"\n",
+    "                },\n",
+    "            }\n",
+    "        ],\n",
+    "    }\n",
+    "]\n",
+    "\n",
+    "tool_definitions_conversation = [\n",
+    "    {\n",
+    "        \"name\": \"fetch_weather\",\n",
+    "        \"description\": \"Fetches the weather information for the specified location.\",\n",
+    "        \"parameters\": {\n",
+    "            \"type\": \"object\",\n",
+    "            \"properties\": {\"location\": {\"type\": \"string\", \"description\": \"The location to fetch weather for.\"}},\n",
+    "            \"required\": [\"location\"]\n",
+    "        },\n",
+    "    },\n",
+    "    {\n",
+    "        \"name\": \"send_email\",\n",
+    "        \"description\": \"Sends an email with the specified subject and body to the recipient.\",\n",
+    "        \"parameters\": {\n",
+    "            \"type\": \"object\",\n",
+    "            \"properties\": {\n",
+    "                \"recipient\": {\"type\": \"string\", \"description\": \"Email address of the recipient.\"},\n",
+    "                \"subject\": {\"type\": \"string\", \"description\": \"Subject of the email.\"},\n",
+    "                \"body\": {\"type\": \"string\", \"description\": \"Body content of the email.\"},\n",
+    "            },\n",
+    "            \"required\": [\"recipient\", \"subject\", \"body\"]\n",
+    "        },\n",
+    "    },\n",
+    "]\n",
+    "\n",
+    "# Append to evaluation_contents\n",
+    "evaluation_contents.append(\n",
+    "    SourceFileContentContent(\n",
+    "        item={\n",
+    "            \"query\": query_conversation,\n",
+    "            \"response\": response_conversation,\n",
+    "            \"tool_definitions\": tool_definitions_conversation\n",
+    "        }\n",
+    "    )\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "14c41e90",
+   "metadata": {},
+   "source": [
+    "#### Example of Incorrect Tool Parameters"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "59810a22",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Test Case 6: Incorrect tool parameters (should score 0)\n",
+    "query_incorrect = \"How is the weather in Seattle?\"\n",
+    "# Missing required parameter or wrong type\n",
+    "response_incorrect = [\n",
+    "    {\n",
+    "        \"role\": \"assistant\",\n",
+    "        \"content\": [\n",
+    "            {\n",
+    "                \"type\": \"tool_call\",\n",
+    "                \"tool_call_id\": \"call_weather_123\",\n",
+    "                \"name\": \"fetch_weather\",\n",
+    "                \"arguments\": {\"city\": \"Seattle\"},  # Wrong parameter name (should be \"location\")\n",
+    "            }\n",
+    "        ],\n",
+    "    }\n",
+    "]\n",
+    "\n",
+    "tool_definitions_incorrect = [\n",
+    "    {\n",
+    "        \"name\": \"fetch_weather\",\n",
+    "        \"description\": \"Fetches the weather information for the specified location.\",\n",
+    "        \"parameters\": {\n",
+    "            \"type\": \"object\",\n",
+    "            \"properties\": {\"location\": {\"type\": \"string\", \"description\": \"The location to fetch weather for.\"}},\n",
+    "            \"required\": [\"location\"]\n",
+    "        },\n",
+    "    }\n",
+    "]\n",
+    "\n",
+    "# Append to evaluation_contents - This should score 0 due to incorrect parameter name\n",
+    "evaluation_contents.append(\n",
+    "    SourceFileContentContent(\n",
+    "        item={\n",
+    "            \"query\": query_incorrect,\n",
+    "            \"response\": response_incorrect,\n",
+    "            \"tool_definitions\": tool_definitions_incorrect\n",
+    "        }\n",
+    "    )\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "e4c7dd3f",
+   "metadata": {},
+   "source": [
+    "### Execute Batch Evaluation\n",
+    "Run all test cases together using the batch evaluation approach."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "fffe6f9f",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "results = run_evaluator(\n",
+    "    evaluator_name=\"tool_input_accuracy\",\n",
+    "    evaluation_contents=evaluation_contents,\n",
+    "    data_source_config=data_source_config,\n",
+    "    initialization_parameters=initialization_parameters,\n",
+    "    data_mapping=data_mapping\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "16b2b2ea",
+   "metadata": {},
+   "source": [
+    "### Results Display\n",
+    "Display the evaluation results for each test case."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "1f7c5195",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "pprint(results)"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "samples",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.19"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
diff --git a/scenarios/evaluate/Supported_Evaluation_Metrics/Agent_Evaluation/AI_Judge_Evaluator_Tool_Output_Utilization.ipynb b/scenarios/evaluate/Supported_Evaluation_Metrics/Agent_Evaluation/AI_Judge_Evaluator_Tool_Output_Utilization.ipynb
new file mode 100644
index 00000000..18dd5b48
--- /dev/null
+++ b/scenarios/evaluate/Supported_Evaluation_Metrics/Agent_Evaluation/AI_Judge_Evaluator_Tool_Output_Utilization.ipynb
@@ -0,0 +1,693 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "d638561f",
+   "metadata": {},
+   "source": [
+    "# Tool Output Utilization Evaluator\n",
+    "\n",
+    "## Objective\n",
+    "This sample demonstrates how to use tool output utilization evaluator on agent data. The supported input formats include:\n",
+    "- simple data such as strings and `dict` describing agent responses;\n",
+    "- user-agent conversations in the form of list of agent messages. \n",
+    "\n",
+    "## Time\n",
+    "\n",
+    "You should expect to spend about 20 minutes running this notebook. \n",
+    "\n",
+    "## Before you begin\n",
+    "For quality evaluation, you need to deploy a `gpt` model supporting JSON mode. We recommend using `gpt-4o`.    \n",
+    "\n",
+    "### Prerequisite\n",
+    "```bash\n",
+    "pip install azure-ai-projects azure-identity openai\n",
+    "```\n",
+    "Set these environment variables with your own values:\n",
+    "1) **AZURE_AI_PROJECT_ENDPOINT** - Your Azure AI project endpoint in format: `https://<account_name>.services.ai.azure.com/api/projects/<project_name>`\n",
+    "2) **AZURE_AI_MODEL_DEPLOYMENT_NAME** - The deployment name of the model for this AI-assisted evaluator (e.g., gpt-4o-mini)\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "41d3647c",
+   "metadata": {},
+   "source": [
+    "The Tool Output Utilization Evaluator assesses how effectively an AI agent utilizes the outputs from tools and whether it accurately incorporates this information into its responses.\n",
+    "\n",
+    "Scoring is based on two levels:\n",
+    "1. Pass - The agent effectively utilizes tool outputs and accurately incorporates the information into its response.\n",
+    "2. Fail - The agent fails to properly utilize tool outputs or incorrectly incorporates the information into its response.\n",
+    "\n",
+    "The evaluation includes the score, a brief explanation, and a final pass/fail result.\n",
+    "\n",
+    "This evaluation focuses on measuring whether the agent properly uses the information returned by tools to provide accurate and complete responses to users."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "246c7bcd",
+   "metadata": {},
+   "source": [
+    "Tool Output Utilization requires following input:\n",
+    "- Query - This can be a single query or a list of messages(conversation history with agent). The original task request from the user.\n",
+    "- Response - Response from Agent (or any GenAI App). This can be a single text response or a list of messages generated as part of Agent Response.\n",
+    "- Tool Definitions - Tool(s) definition used by Agent to answer the query. Required to understand the context and expected outputs of the tools.\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "d386d576",
+   "metadata": {},
+   "source": [
+    "### Initialize Tool Output Utilization Evaluator\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "087dd96d",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import os\n",
+    "from openai.types.evals.create_eval_jsonl_run_data_source_param import SourceFileContentContent\n",
+    "from pprint import pprint\n",
+    "from agent_utils import run_evaluator\n",
+    "\n",
+    "# Get environment variables\n",
+    "deployment_name = os.environ[\"AZURE_AI_MODEL_DEPLOYMENT_NAME\"]\n",
+    "\n",
+    "# Data source configuration (defines the schema for evaluation inputs)\n",
+    "data_source_config = {\n",
+    "    \"type\": \"custom\",\n",
+    "    \"item_schema\": {\n",
+    "        \"type\": \"object\",\n",
+    "        \"properties\": {\n",
+    "            \"query\": {\n",
+    "                \"anyOf\": [\n",
+    "                    {\"type\": \"string\"},\n",
+    "                    {\n",
+    "                        \"type\": \"array\",\n",
+    "                        \"items\": {\n",
+    "                            \"type\": \"object\"\n",
+    "                        }\n",
+    "                    }\n",
+    "                ]\n",
+    "            },\n",
+    "            \"response\": {\n",
+    "                \"anyOf\": [\n",
+    "                    {\"type\": \"string\"},\n",
+    "                    {\n",
+    "                        \"type\": \"array\",\n",
+    "                        \"items\": {\n",
+    "                            \"type\": \"object\"\n",
+    "                        }\n",
+    "                    }\n",
+    "                ]\n",
+    "            },\n",
+    "            \"tool_definitions\": {\n",
+    "                \"anyOf\": [\n",
+    "                    {\"type\": \"object\"},\n",
+    "                    {\n",
+    "                        \"type\": \"array\",\n",
+    "                        \"items\": {\n",
+    "                            \"type\": \"object\"\n",
+    "                        }\n",
+    "                    }\n",
+    "                ]\n",
+    "            }\n",
+    "        },\n",
+    "        \"required\": [\"query\", \"response\"]\n",
+    "    },\n",
+    "    \"include_sample_schema\": True\n",
+    "}\n",
+    "\n",
+    "# Data mapping (maps evaluation inputs to evaluator parameters)\n",
+    "data_mapping = {\n",
+    "    \"query\": \"{{item.query}}\",\n",
+    "    \"response\": \"{{item.response}}\",\n",
+    "    \"tool_definitions\": \"{{item.tool_definitions}}\"\n",
+    "}\n",
+    "\n",
+    "# Initialization parameters for the evaluator\n",
+    "initialization_parameters = {\n",
+    "    \"deployment_name\": deployment_name\n",
+    "}\n",
+    "\n",
+    "# Initialize the evaluation_contents list - we'll append all test cases here\n",
+    "evaluation_contents = []"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "c7b4f98a",
+   "metadata": {},
+   "source": [
+    "### Samples"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "9ee00e9d",
+   "metadata": {},
+   "source": [
+    "#### Evaluating Good Tool Output Utilization"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "017a7afe",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Test Case 1: Good tool output utilization\n",
+    "query = \"How is the weather in Seattle?\"\n",
+    "response = [\n",
+    "    {\n",
+    "        \"role\": \"assistant\",\n",
+    "        \"content\": [\n",
+    "            {\n",
+    "                \"type\": \"tool_call\",\n",
+    "                \"tool_call_id\": \"call_weather_123\",\n",
+    "                \"name\": \"fetch_weather\",\n",
+    "                \"arguments\": {\"location\": \"Seattle\"},\n",
+    "            }\n",
+    "        ],\n",
+    "    },\n",
+    "    {\n",
+    "        \"tool_call_id\": \"call_weather_123\",\n",
+    "        \"role\": \"tool\",\n",
+    "        \"content\": [{\"type\": \"tool_result\", \"tool_result\": {\"temperature\": \"15°C\", \"condition\": \"partly cloudy\", \"humidity\": \"68%\", \"wind\": \"8 mph NW\"}}],\n",
+    "    },\n",
+    "    {\n",
+    "        \"role\": \"assistant\",\n",
+    "        \"content\": [\n",
+    "            {\n",
+    "                \"type\": \"text\",\n",
+    "                \"text\": \"The current weather in Seattle is partly cloudy with a temperature of 15°C. The humidity is at 68% and there's a light wind from the northwest at 8 mph.\",\n",
+    "            }\n",
+    "        ],\n",
+    "    }\n",
+    "]\n",
+    "\n",
+    "tool_definitions = [\n",
+    "    {\n",
+    "        \"name\": \"fetch_weather\",\n",
+    "        \"description\": \"Fetches the weather information for the specified location.\",\n",
+    "        \"parameters\": {\n",
+    "            \"type\": \"object\",\n",
+    "            \"properties\": {\"location\": {\"type\": \"string\", \"description\": \"The location to fetch weather for.\"}},\n",
+    "        },\n",
+    "    }\n",
+    "]\n",
+    "\n",
+    "# Append to evaluation_contents\n",
+    "evaluation_contents.append(\n",
+    "    SourceFileContentContent(\n",
+    "        item={\n",
+    "            \"query\": query,\n",
+    "            \"response\": response,\n",
+    "            \"tool_definitions\": tool_definitions\n",
+    "        }\n",
+    "    )\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "78270b9b",
+   "metadata": {},
+   "source": [
+    "#### Complex Scenario with Multiple Tool Outputs"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "e8543c52",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Test Case 2: Complex scenario with multiple tool outputs\n",
+    "query_complex = \"Can you send me an email with weather information for Seattle?\"\n",
+    "response_complex = [\n",
+    "    {\n",
+    "        \"createdAt\": \"2025-03-26T17:27:35Z\",\n",
+    "        \"run_id\": \"run_zblZyGCNyx6aOYTadmaqM4QN\",\n",
+    "        \"role\": \"assistant\",\n",
+    "        \"content\": [\n",
+    "            {\n",
+    "                \"type\": \"tool_call\",\n",
+    "                \"tool_call_id\": \"call_CUdbkBfvVBla2YP3p24uhElJ\",\n",
+    "                \"name\": \"fetch_weather\",\n",
+    "                \"arguments\": {\"location\": \"Seattle\"},\n",
+    "            }\n",
+    "        ],\n",
+    "    },\n",
+    "    {\n",
+    "        \"createdAt\": \"2025-03-26T17:27:37Z\",\n",
+    "        \"run_id\": \"run_zblZyGCNyx6aOYTadmaqM4QN\",\n",
+    "        \"tool_call_id\": \"call_CUdbkBfvVBla2YP3p24uhElJ\",\n",
+    "        \"role\": \"tool\",\n",
+    "        \"content\": [{\"type\": \"tool_result\", \"tool_result\": {\"temperature\": \"14°C\", \"condition\": \"rainy\", \"precipitation\": \"light rain\", \"wind\": \"12 mph SW\"}}],\n",
+    "    },\n",
+    "    {\n",
+    "        \"createdAt\": \"2025-03-26T17:27:38Z\",\n",
+    "        \"run_id\": \"run_zblZyGCNyx6aOYTadmaqM4QN\",\n",
+    "        \"role\": \"assistant\",\n",
+    "        \"content\": [\n",
+    "            {\n",
+    "                \"type\": \"tool_call\",\n",
+    "                \"tool_call_id\": \"call_iq9RuPxqzykebvACgX8pqRW2\",\n",
+    "                \"name\": \"send_email\",\n",
+    "                \"arguments\": {\n",
+    "                    \"recipient\": \"your_email@example.com\",\n",
+    "                    \"subject\": \"Weather Information for Seattle\",\n",
+    "                    \"body\": \"Current weather in Seattle: 14°C and rainy with light rain. Wind is from the southwest at 12 mph. Stay dry!\",\n",
+    "                },\n",
+    "            }\n",
+    "        ],\n",
+    "    },\n",
+    "    {\n",
+    "        \"createdAt\": \"2025-03-26T17:27:41Z\",\n",
+    "        \"run_id\": \"run_zblZyGCNyx6aOYTadmaqM4QN\",\n",
+    "        \"tool_call_id\": \"call_iq9RuPxqzykebvACgX8pqRW2\",\n",
+    "        \"role\": \"tool\",\n",
+    "        \"content\": [\n",
+    "            {\"type\": \"tool_result\", \"tool_result\": {\"message\": \"Email successfully sent to your_email@example.com.\", \"status\": \"delivered\"}}\n",
+    "        ],\n",
+    "    },\n",
+    "    {\n",
+    "        \"createdAt\": \"2025-03-26T17:27:42Z\",\n",
+    "        \"run_id\": \"run_zblZyGCNyx6aOYTadmaqM4QN\",\n",
+    "        \"role\": \"assistant\",\n",
+    "        \"content\": [\n",
+    "            {\n",
+    "                \"type\": \"text\",\n",
+    "                \"text\": \"I have successfully sent you an email with the weather information for Seattle. The email includes the current conditions: 14°C and rainy with light rain, plus wind details. The email was delivered successfully to your inbox.\",\n",
+    "            }\n",
+    "        ],\n",
+    "    },\n",
+    "]\n",
+    "\n",
+    "tool_definitions_multiple = [\n",
+    "    {\n",
+    "        \"name\": \"fetch_weather\",\n",
+    "        \"description\": \"Fetches the weather information for the specified location.\",\n",
+    "        \"parameters\": {\n",
+    "            \"type\": \"object\",\n",
+    "            \"properties\": {\"location\": {\"type\": \"string\", \"description\": \"The location to fetch weather for.\"}},\n",
+    "        },\n",
+    "    },\n",
+    "    {\n",
+    "        \"name\": \"send_email\",\n",
+    "        \"description\": \"Sends an email with the specified subject and body to the recipient.\",\n",
+    "        \"parameters\": {\n",
+    "            \"type\": \"object\",\n",
+    "            \"properties\": {\n",
+    "                \"recipient\": {\"type\": \"string\", \"description\": \"Email address of the recipient.\"},\n",
+    "                \"subject\": {\"type\": \"string\", \"description\": \"Subject of the email.\"},\n",
+    "                \"body\": {\"type\": \"string\", \"description\": \"Body content of the email.\"},\n",
+    "            },\n",
+    "        },\n",
+    "    },\n",
+    "]\n",
+    "\n",
+    "# Append to evaluation_contents\n",
+    "evaluation_contents.append(\n",
+    "    SourceFileContentContent(\n",
+    "        item={\n",
+    "            \"query\": query_complex,\n",
+    "            \"response\": response_complex,\n",
+    "            \"tool_definitions\": tool_definitions_multiple\n",
+    "        }\n",
+    "    )\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "1c52f24c",
+   "metadata": {},
+   "source": [
+    "#### Query as Conversation History"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "b3b9bef5",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Test Case 3: Query as conversation history instead of a single string\n",
+    "query_as_conversation = [\n",
+    "    {\n",
+    "        \"role\": \"system\",\n",
+    "        \"content\": \"You are a helpful assistant that can fetch weather information and send emails.\"\n",
+    "    },\n",
+    "    {\n",
+    "        \"role\": \"user\", \n",
+    "        \"content\": \"Hi, I need to know the weather in Seattle for my trip planning.\"\n",
+    "    },\n",
+    "    {\n",
+    "        \"role\": \"user\",\n",
+    "        \"content\": \"Can you also email me a summary so I can reference it later when packing?\"\n",
+    "    }\n",
+    "]\n",
+    "\n",
+    "response_conversation = [\n",
+    "    {\n",
+    "        \"role\": \"assistant\",\n",
+    "        \"content\": [\n",
+    "            {\n",
+    "                \"type\": \"tool_call\",\n",
+    "                \"tool_call_id\": \"call_weather_123\",\n",
+    "                \"name\": \"fetch_weather\",\n",
+    "                \"arguments\": {\"location\": \"Seattle\"},\n",
+    "            }\n",
+    "        ],\n",
+    "    },\n",
+    "    {\n",
+    "        \"tool_call_id\": \"call_weather_123\",\n",
+    "        \"role\": \"tool\",\n",
+    "        \"content\": [{\"type\": \"tool_result\", \"tool_result\": {\"temperature\": \"15°C\", \"condition\": \"partly cloudy\", \"humidity\": \"68%\", \"wind\": \"8 mph NW\", \"forecast\": \"no rain expected\"}}],\n",
+    "    },\n",
+    "    {\n",
+    "        \"role\": \"assistant\",\n",
+    "        \"content\": [\n",
+    "            {\n",
+    "                \"type\": \"tool_call\",\n",
+    "                \"tool_call_id\": \"call_email_456\",\n",
+    "                \"name\": \"send_email\",\n",
+    "                \"arguments\": {\n",
+    "                    \"recipient\": \"user@example.com\",\n",
+    "                    \"subject\": \"Seattle Weather Summary for Trip Planning\",\n",
+    "                    \"body\": \"Weather summary for Seattle: Temperature is 15°C, partly cloudy conditions, humidity at 68%, with northwest winds at 8 mph. No rain is expected, so light layers should be sufficient for your trip.\"\n",
+    "                },\n",
+    "            }\n",
+    "        ],\n",
+    "    },\n",
+    "    {\n",
+    "        \"tool_call_id\": \"call_email_456\",\n",
+    "        \"role\": \"tool\",\n",
+    "        \"content\": [{\"type\": \"tool_result\", \"tool_result\": {\"message\": \"Email sent successfully\", \"delivery_status\": \"delivered\"}}],\n",
+    "    },\n",
+    "    {\n",
+    "        \"role\": \"assistant\",\n",
+    "        \"content\": [\n",
+    "            {\n",
+    "                \"type\": \"text\",\n",
+    "                \"text\": \"Perfect! I've checked the weather in Seattle for your trip planning. It's currently 15°C and partly cloudy with light northwest winds and no rain expected. I've also sent you an email summary with all these details including packing recommendations based on the weather conditions. The email was delivered successfully, so you can reference it when preparing for your trip.\",\n",
+    "            }\n",
+    "        ],\n",
+    "    }\n",
+    "]\n",
+    "\n",
+    "tool_definitions_conversation = [\n",
+    "    {\n",
+    "        \"name\": \"fetch_weather\",\n",
+    "        \"description\": \"Fetches the weather information for the specified location.\",\n",
+    "        \"parameters\": {\n",
+    "            \"type\": \"object\",\n",
+    "            \"properties\": {\"location\": {\"type\": \"string\", \"description\": \"The location to fetch weather for.\"}},\n",
+    "        },\n",
+    "    },\n",
+    "    {\n",
+    "        \"name\": \"send_email\",\n",
+    "        \"description\": \"Sends an email with the specified subject and body to the recipient.\",\n",
+    "        \"parameters\": {\n",
+    "            \"type\": \"object\",\n",
+    "            \"properties\": {\n",
+    "                \"recipient\": {\"type\": \"string\", \"description\": \"Email address of the recipient.\"},\n",
+    "                \"subject\": {\"type\": \"string\", \"description\": \"Subject of the email.\"},\n",
+    "                \"body\": {\"type\": \"string\", \"description\": \"Body content of the email.\"},\n",
+    "            },\n",
+    "        },\n",
+    "    },\n",
+    "]\n",
+    "\n",
+    "# Append to evaluation_contents\n",
+    "evaluation_contents.append(\n",
+    "    SourceFileContentContent(\n",
+    "        item={\n",
+    "            \"query\": query_as_conversation,\n",
+    "            \"response\": response_conversation,\n",
+    "            \"tool_definitions\": tool_definitions_conversation\n",
+    "        }\n",
+    "    )\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "199a11bc",
+   "metadata": {},
+   "source": [
+    "#### Example of Poor Tool Output Utilization"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "6be39269",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Test Case 4: Example of poor tool output utilization\n",
+    "query_poor = \"How is the weather in Seattle?\"\n",
+    "# Poor utilization - agent doesn't use the detailed tool output effectively\n",
+    "poor_response = [\n",
+    "    {\n",
+    "        \"role\": \"assistant\",\n",
+    "        \"content\": [\n",
+    "            {\n",
+    "                \"type\": \"tool_call\",\n",
+    "                \"tool_call_id\": \"call_weather_123\",\n",
+    "                \"name\": \"fetch_weather\",\n",
+    "                \"arguments\": {\"location\": \"Seattle\"},\n",
+    "            }\n",
+    "        ],\n",
+    "    },\n",
+    "    {\n",
+    "        \"tool_call_id\": \"call_weather_123\",\n",
+    "        \"role\": \"tool\",\n",
+    "        \"content\": [{\"type\": \"tool_result\", \"tool_result\": {\"temperature\": \"15°C\", \"condition\": \"partly cloudy\", \"humidity\": \"68%\", \"wind\": \"8 mph NW\", \"UV_index\": \"3\", \"visibility\": \"10 km\"}}],\n",
+    "    },\n",
+    "    {\n",
+    "        \"role\": \"assistant\",\n",
+    "        \"content\": [\n",
+    "            {\n",
+    "                \"type\": \"text\",\n",
+    "                \"text\": \"The weather in Seattle is okay. It's not too bad outside.\",  # Poor utilization of detailed weather data\n",
+    "            }\n",
+    "        ],\n",
+    "    }\n",
+    "]\n",
+    "\n",
+    "tool_definitions_poor = [\n",
+    "    {\n",
+    "        \"name\": \"fetch_weather\",\n",
+    "        \"description\": \"Fetches the weather information for the specified location.\",\n",
+    "        \"parameters\": {\n",
+    "            \"type\": \"object\",\n",
+    "            \"properties\": {\"location\": {\"type\": \"string\", \"description\": \"The location to fetch weather for.\"}},\n",
+    "        },\n",
+    "    }\n",
+    "]\n",
+    "\n",
+    "# This should score poorly due to inadequate use of tool output\n",
+    "evaluation_contents.append(\n",
+    "    SourceFileContentContent(\n",
+    "        item={\n",
+    "            \"query\": query_poor,\n",
+    "            \"response\": poor_response,\n",
+    "            \"tool_definitions\": tool_definitions_poor\n",
+    "        }\n",
+    "    )\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "bcb8a2cf",
+   "metadata": {},
+   "source": [
+    "#### Response as String (str)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "0845822f",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Test Case 5: Response as string\n",
+    "query_str = \"What's the weather like in Seattle?\"\n",
+    "# Response as a simple string (not a list of messages)\n",
+    "response_str = \"The weather in Seattle is currently 15°C and partly cloudy with light winds from the northwest.\"\n",
+    "\n",
+    "tool_definitions_str = [\n",
+    "    {\n",
+    "        \"name\": \"fetch_weather\",\n",
+    "        \"description\": \"Fetches the weather information for the specified location.\",\n",
+    "        \"parameters\": {\n",
+    "            \"type\": \"object\",\n",
+    "            \"properties\": {\"location\": {\"type\": \"string\", \"description\": \"The location to fetch weather for.\"}},\n",
+    "        },\n",
+    "    }\n",
+    "]\n",
+    "\n",
+    "# Append to evaluation_contents\n",
+    "evaluation_contents.append(\n",
+    "    SourceFileContentContent(\n",
+    "        item={\n",
+    "            \"query\": query_str,\n",
+    "            \"response\": response_str,\n",
+    "            \"tool_definitions\": tool_definitions_str\n",
+    "        }\n",
+    "    )\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "b4cb9ddd",
+   "metadata": {},
+   "source": [
+    "#### Tool Definitions as Single Dict"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "6424b3bb",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Test Case 6: Tool definitions as single dict\n",
+    "query_dict = \"Check the weather in Seattle\"\n",
+    "response_dict = [\n",
+    "    {\n",
+    "        \"role\": \"assistant\",\n",
+    "        \"content\": [\n",
+    "            {\n",
+    "                \"type\": \"tool_call\",\n",
+    "                \"tool_call_id\": \"call_weather_456\",\n",
+    "                \"name\": \"fetch_weather\",\n",
+    "                \"arguments\": {\"location\": \"Seattle\"},\n",
+    "            }\n",
+    "        ],\n",
+    "    },\n",
+    "    {\n",
+    "        \"tool_call_id\": \"call_weather_456\",\n",
+    "        \"role\": \"tool\",\n",
+    "        \"content\": [{\"type\": \"tool_result\", \"tool_result\": {\"temperature\": \"14°C\", \"condition\": \"rainy\"}}],\n",
+    "    },\n",
+    "    {\n",
+    "        \"role\": \"assistant\",\n",
+    "        \"content\": [\n",
+    "            {\n",
+    "                \"type\": \"text\",\n",
+    "                \"text\": \"The current weather in Seattle is rainy with a temperature of 14°C.\",\n",
+    "            }\n",
+    "        ],\n",
+    "    }\n",
+    "]\n",
+    "\n",
+    "# Tool definition as a single dict (not a list)\n",
+    "tool_definition_dict = [\n",
+    "    {\n",
+    "    \"name\": \"fetch_weather\",\n",
+    "    \"description\": \"Fetches the weather information for the specified location.\",\n",
+    "    \"parameters\": {\n",
+    "        \"type\": \"object\",\n",
+    "        \"properties\": {\"location\": {\"type\": \"string\", \"description\": \"The location to fetch weather for.\"}},\n",
+    "    },\n",
+    "}\n",
+    "]\n",
+    "\n",
+    "# Append to evaluation_contents\n",
+    "evaluation_contents.append(\n",
+    "    SourceFileContentContent(\n",
+    "        item={\n",
+    "            \"query\": query_dict,\n",
+    "            \"response\": response_dict,\n",
+    "            \"tool_definitions\": tool_definition_dict\n",
+    "        }\n",
+    "    )\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "9dcbf1cd",
+   "metadata": {},
+   "source": [
+    "### Run Evaluation on All Test Cases\n",
+    "\n",
+    "Now that we've defined all test cases, let's run the evaluation once on all of them."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "2f6af4c2",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "results = run_evaluator(\n",
+    "    evaluator_name=\"tool_output_utilization\",\n",
+    "    evaluation_contents=evaluation_contents,\n",
+    "    data_source_config=data_source_config,\n",
+    "    initialization_parameters=initialization_parameters,\n",
+    "    data_mapping=data_mapping\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "abfb12c7",
+   "metadata": {},
+   "source": [
+    "### Display Results\n",
+    "\n",
+    "View the evaluation results for each test case."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "ce47c4cb",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "pprint(results)"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "samples",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.19"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
diff --git a/scenarios/evaluate/Supported_Evaluation_Metrics/Agent_Evaluation/AI_Judge_Evaluator_Tool_Selection.ipynb b/scenarios/evaluate/Supported_Evaluation_Metrics/Agent_Evaluation/AI_Judge_Evaluator_Tool_Selection.ipynb
new file mode 100644
index 00000000..83cc295a
--- /dev/null
+++ b/scenarios/evaluate/Supported_Evaluation_Metrics/Agent_Evaluation/AI_Judge_Evaluator_Tool_Selection.ipynb
@@ -0,0 +1,767 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "862760b0",
+   "metadata": {},
+   "source": [
+    "# Tool Selection Evaluator\n",
+    "\n",
+    "## Objective\n",
+    "This sample demonstrates how to use tool selection evaluator on agent data. The supported input formats include:\n",
+    "- simple data such as strings and `dict` describing tool calls;\n",
+    "- user-agent conversations in the form of list of agent messages. \n",
+    "\n",
+    "## Time\n",
+    "\n",
+    "You should expect to spend about 20 minutes running this notebook. \n",
+    "\n",
+    "## Before you begin\n",
+    "For quality evaluation, you need to deploy a `gpt` model supporting JSON mode. We recommend using `gpt-4.1` or `gpt-4.1-mini`.\n",
+    "\n",
+    "### Prerequisite\n",
+    "```bash\n",
+    "pip install azure-ai-projects azure-identity openai\n",
+    "```\n",
+    "Set these environment variables with your own values:\n",
+    "1) **AZURE_AI_PROJECT_ENDPOINT** - Your Azure AI project endpoint in format: `https://<account_name>.services.ai.azure.com/api/projects/<project_name>`\n",
+    "2) **AZURE_AI_MODEL_DEPLOYMENT_NAME** - The deployment name of the model for this AI-assisted evaluator (e.g., gpt-4o-mini)\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "31a75ee6",
+   "metadata": {},
+   "source": [
+    "The Tool Selection evaluator assesses the appropriateness and efficiency of tool choices made by an AI agent by examining:\n",
+    "- Relevance of selected tools to the conversation\n",
+    "- Completeness of tool selection according to task requirements\n",
+    "- Efficiency in avoiding unnecessary or redundant tools\n",
+    "\n",
+    "The evaluator uses a binary scoring system:\n",
+    "\n",
+    "    - Score 0 (Fail): Tools selected are irrelevant, incorrect, or missing essential tools\n",
+    "    - Score 1 (Pass): All needed tools are selected, even if there are redundant tools\n",
+    "\n",
+    "This evaluation focuses on measuring whether the right tools were chosen for the task, regardless of how those tools were executed or their parameter correctness."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "353f2304",
+   "metadata": {},
+   "source": [
+    "Tool Selection requires following input:\n",
+    "- Query - This can be a single query or a list of messages(conversation history with agent). The original task request from the user.\n",
+    "- Tool Calls - Tool Call(s) made by Agent to answer the query. Optional - if response has tool calls, if not provided evaluator will look for tool calls in response.\n",
+    "- Response - (Optional) Response from Agent (or any GenAI App). This can be a single text response or a list or messages generated as part of Agent Response. If tool calls are not provided, Tool Selection Evaluator will look at response for tool calls.\n",
+    "- Tool Definitions - Tool(s) definition used by Agent to answer the query. Required to understand available tools and their purposes.\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "75125e94",
+   "metadata": {},
+   "source": [
+    "### Initialize Tool Selection Evaluator\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "b5f31144",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import os\n",
+    "from openai.types.evals.create_eval_jsonl_run_data_source_param import SourceFileContentContent\n",
+    "from pprint import pprint\n",
+    "from agent_utils import run_evaluator\n",
+    "\n",
+    "# Get environment variables\n",
+    "deployment_name = os.environ[\"AZURE_AI_MODEL_DEPLOYMENT_NAME\"]\n",
+    "\n",
+    "# Data source configuration (defines the schema for evaluation inputs)\n",
+    "data_source_config = {\n",
+    "    \"type\": \"custom\",\n",
+    "    \"item_schema\": {\n",
+    "        \"type\": \"object\",\n",
+    "        \"properties\": {\n",
+    "            \"query\": {\n",
+    "                \"anyOf\": [\n",
+    "                    {\"type\": \"string\"},\n",
+    "                    {\n",
+    "                        \"type\": \"array\",\n",
+    "                        \"items\": {\n",
+    "                            \"type\": \"object\"\n",
+    "                        }\n",
+    "                    }\n",
+    "                ]\n",
+    "            },\n",
+    "            \"response\": {\n",
+    "                \"anyOf\": [\n",
+    "                    {\"type\": \"string\"},\n",
+    "                    {\n",
+    "                        \"type\": \"array\",\n",
+    "                        \"items\": {\n",
+    "                            \"type\": \"object\"\n",
+    "                        }\n",
+    "                    }\n",
+    "                ]\n",
+    "            },\n",
+    "            \"tool_calls\": {\n",
+    "                \"anyOf\": [\n",
+    "                    {\"type\": \"object\"},\n",
+    "                    {\n",
+    "                        \"type\": \"array\",\n",
+    "                        \"items\": {\n",
+    "                            \"type\": \"object\"\n",
+    "                        }\n",
+    "                    }\n",
+    "                ]\n",
+    "            },\n",
+    "            \"tool_definitions\": {\n",
+    "                \"anyOf\": [\n",
+    "                    {\"type\": \"object\"},\n",
+    "                    {\n",
+    "                        \"type\": \"array\",\n",
+    "                        \"items\": {\n",
+    "                            \"type\": \"object\"\n",
+    "                        }\n",
+    "                    }\n",
+    "                ]\n",
+    "            }\n",
+    "        },\n",
+    "        \"required\": [\"query\", \"response\", \"tool_definitions\"]\n",
+    "    },\n",
+    "    \"include_sample_schema\": True\n",
+    "}\n",
+    "\n",
+    "# Data mapping (maps evaluation inputs to evaluator parameters)\n",
+    "data_mapping = {\n",
+    "    \"query\": \"{{item.query}}\",\n",
+    "    \"response\": \"{{item.response}}\",\n",
+    "    \"tool_calls\": \"{{item.tool_calls}}\",\n",
+    "    \"tool_definitions\": \"{{item.tool_definitions}}\"\n",
+    "}\n",
+    "\n",
+    "# Initialization parameters for the evaluator\n",
+    "initialization_parameters = {\n",
+    "    \"deployment_name\": deployment_name\n",
+    "}\n",
+    "\n",
+    "# Initialize the evaluation_contents list - we'll append all test cases here\n",
+    "evaluation_contents = []"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "fb435367",
+   "metadata": {},
+   "source": [
+    "### Samples"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "344fe68e",
+   "metadata": {},
+   "source": [
+    "#### Evaluating Single Tool Selection"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "dcb71257",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Test Case 1: Single tool selection for weather query\n",
+    "query = \"How is the weather in Seattle?\"\n",
+    "tool_call = [\n",
+    "    {\n",
+    "        \"type\": \"tool_call\",\n",
+    "        \"tool_call_id\": \"call_CUdbkBfvVBla2YP3p24uhElJ\",\n",
+    "        \"name\": \"fetch_weather\",\n",
+    "        \"arguments\": {\"location\": \"Seattle\"},\n",
+    "    }\n",
+    "]\n",
+    "\n",
+    "tool_definitions = [\n",
+    "    {\n",
+    "        \"id\": \"fetch_weather\",\n",
+    "        \"name\": \"fetch_weather\",\n",
+    "        \"description\": \"Fetches the weather information for the specified location.\",\n",
+    "        \"parameters\": {\n",
+    "            \"type\": \"object\",\n",
+    "            \"properties\": {\"location\": {\"type\": \"string\", \"description\": \"The location to fetch weather for.\"}},\n",
+    "        },\n",
+    "    },\n",
+    "    {\n",
+    "        \"id\": \"send_email\",\n",
+    "        \"name\": \"send_email\",\n",
+    "        \"description\": \"Sends an email with the specified subject and body to the recipient.\",\n",
+    "        \"parameters\": {\n",
+    "            \"type\": \"object\",\n",
+    "            \"properties\": {\n",
+    "                \"recipient\": {\"type\": \"string\", \"description\": \"Email address of the recipient.\"},\n",
+    "                \"subject\": {\"type\": \"string\", \"description\": \"Subject of the email.\"},\n",
+    "                \"body\": {\"type\": \"string\", \"description\": \"Body content of the email.\"},\n",
+    "            },\n",
+    "        },\n",
+    "    },\n",
+    "]\n",
+    "\n",
+    "# Append to evaluation_contents\n",
+    "evaluation_contents.append(\n",
+    "    SourceFileContentContent(\n",
+    "        item={\n",
+    "            \"query\": query,\n",
+    "            \"response\": None,\n",
+    "            \"tool_calls\": tool_call,\n",
+    "            \"tool_definitions\": tool_definitions\n",
+    "        }\n",
+    "    )\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "e078b1fa",
+   "metadata": {},
+   "source": [
+    "#### Multiple Tool Selections for Complex Task"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "322efe28",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Test Case 2: Multiple tool selections for complex task\n",
+    "query = \"Can you send me an email with weather information for Seattle?\"\n",
+    "tool_calls = [\n",
+    "    {\n",
+    "        \"type\": \"tool_call\",\n",
+    "        \"tool_call_id\": \"call_CUdbkBfvVBla2YP3p24uhElJ\",\n",
+    "        \"name\": \"fetch_weather\",\n",
+    "        \"arguments\": {\"location\": \"Seattle\"},\n",
+    "    },\n",
+    "    {\n",
+    "        \"type\": \"tool_call\",\n",
+    "        \"tool_call_id\": \"call_iq9RuPxqzykebvACgX8pqRW2\",\n",
+    "        \"name\": \"send_email\",\n",
+    "        \"arguments\": {\n",
+    "            \"recipient\": \"user@example.com\",\n",
+    "            \"subject\": \"Weather Information for Seattle\",\n",
+    "            \"body\": \"Weather data will be included here.\",\n",
+    "        },\n",
+    "    },\n",
+    "]\n",
+    "\n",
+    "tool_definitions_multiple = [\n",
+    "    {\n",
+    "        \"id\": \"fetch_weather\",\n",
+    "        \"name\": \"fetch_weather\",\n",
+    "        \"description\": \"Fetches the weather information for the specified location.\",\n",
+    "        \"parameters\": {\n",
+    "            \"type\": \"object\",\n",
+    "            \"properties\": {\"location\": {\"type\": \"string\", \"description\": \"The location to fetch weather for.\"}},\n",
+    "        },\n",
+    "    },\n",
+    "    {\n",
+    "        \"id\": \"send_email\",\n",
+    "        \"name\": \"send_email\",\n",
+    "        \"description\": \"Sends an email with the specified subject and body to the recipient.\",\n",
+    "        \"parameters\": {\n",
+    "            \"type\": \"object\",\n",
+    "            \"properties\": {\n",
+    "                \"recipient\": {\"type\": \"string\", \"description\": \"Email address of the recipient.\"},\n",
+    "                \"subject\": {\"type\": \"string\", \"description\": \"Subject of the email.\"},\n",
+    "                \"body\": {\"type\": \"string\", \"description\": \"Body content of the email.\"},\n",
+    "            },\n",
+    "        },\n",
+    "    },\n",
+    "    {\n",
+    "        \"id\": \"get_calendar\",\n",
+    "        \"name\": \"get_calendar\",\n",
+    "        \"description\": \"Retrieves calendar events for a specified date range.\",\n",
+    "        \"parameters\": {\n",
+    "            \"type\": \"object\",\n",
+    "            \"properties\": {\n",
+    "                \"start_date\": {\"type\": \"string\", \"description\": \"Start date for calendar events.\"},\n",
+    "                \"end_date\": {\"type\": \"string\", \"description\": \"End date for calendar events.\"},\n",
+    "            },\n",
+    "        },\n",
+    "    },\n",
+    "]\n",
+    "\n",
+    "# Append to evaluation_contents\n",
+    "evaluation_contents.append(\n",
+    "    SourceFileContentContent(\n",
+    "        item={\n",
+    "            \"query\": query,\n",
+    "            \"response\": None,\n",
+    "            \"tool_calls\": tool_calls,\n",
+    "            \"tool_definitions\": tool_definitions_multiple\n",
+    "        }\n",
+    "    )\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "19b3c8de",
+   "metadata": {},
+   "source": [
+    "#### Tool Calls passed as part of `Response` (common for agent case)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "629f3cc7",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Test Case 3: Tool calls passed as part of Response (common for agent case)\n",
+    "query = \"Can you send me an email with weather information for Seattle?\"\n",
+    "response = [\n",
+    "    {\n",
+    "        \"createdAt\": \"2025-03-26T17:27:35Z\",\n",
+    "        \"run_id\": \"run_zblZyGCNyx6aOYTadmaqM4QN\",\n",
+    "        \"role\": \"assistant\",\n",
+    "        \"content\": [\n",
+    "            {\n",
+    "                \"type\": \"tool_call\",\n",
+    "                \"tool_call_id\": \"call_CUdbkBfvVBla2YP3p24uhElJ\",\n",
+    "                \"name\": \"fetch_weather\",\n",
+    "                \"arguments\": {\"location\": \"Seattle\"},\n",
+    "            }\n",
+    "        ],\n",
+    "    },\n",
+    "    {\n",
+    "        \"createdAt\": \"2025-03-26T17:27:37Z\",\n",
+    "        \"run_id\": \"run_zblZyGCNyx6aOYTadmaqM4QN\",\n",
+    "        \"tool_call_id\": \"call_CUdbkBfvVBla2YP3p24uhElJ\",\n",
+    "        \"role\": \"tool\",\n",
+    "        \"content\": [{\"type\": \"tool_result\", \"tool_result\": {\"weather\": \"Rainy, 14°C\"}}],\n",
+    "    },\n",
+    "    {\n",
+    "        \"createdAt\": \"2025-03-26T17:27:38Z\",\n",
+    "        \"run_id\": \"run_zblZyGCNyx6aOYTadmaqM4QN\",\n",
+    "        \"role\": \"assistant\",\n",
+    "        \"content\": [\n",
+    "            {\n",
+    "                \"type\": \"tool_call\",\n",
+    "                \"tool_call_id\": \"call_iq9RuPxqzykebvACgX8pqRW2\",\n",
+    "                \"name\": \"send_email\",\n",
+    "                \"arguments\": {\n",
+    "                    \"recipient\": \"your_email@example.com\",\n",
+    "                    \"subject\": \"Weather Information for Seattle\",\n",
+    "                    \"body\": \"The current weather in Seattle is rainy with a temperature of 14°C.\",\n",
+    "                },\n",
+    "            }\n",
+    "        ],\n",
+    "    },\n",
+    "    {\n",
+    "        \"createdAt\": \"2025-03-26T17:27:41Z\",\n",
+    "        \"run_id\": \"run_zblZyGCNyx6aOYTadmaqM4QN\",\n",
+    "        \"tool_call_id\": \"call_iq9RuPxqzykebvACgX8pqRW2\",\n",
+    "        \"role\": \"tool\",\n",
+    "        \"content\": [\n",
+    "            {\"type\": \"tool_result\", \"tool_result\": {\"message\": \"Email successfully sent to your_email@example.com.\"}}\n",
+    "        ],\n",
+    "    },\n",
+    "    {\n",
+    "        \"createdAt\": \"2025-03-26T17:27:42Z\",\n",
+    "        \"run_id\": \"run_zblZyGCNyx6aOYTadmaqM4QN\",\n",
+    "        \"role\": \"assistant\",\n",
+    "        \"content\": [\n",
+    "            {\n",
+    "                \"type\": \"text\",\n",
+    "                \"text\": \"I have successfully sent you an email with the weather information for Seattle. The current weather is rainy with a temperature of 14°C.\",\n",
+    "            }\n",
+    "        ],\n",
+    "    },\n",
+    "]\n",
+    "\n",
+    "tool_definitions_agent = [\n",
+    "    {\n",
+    "        \"name\": \"fetch_weather\",\n",
+    "        \"description\": \"Fetches the weather information for the specified location.\",\n",
+    "        \"parameters\": {\n",
+    "            \"type\": \"object\",\n",
+    "            \"properties\": {\"location\": {\"type\": \"string\", \"description\": \"The location to fetch weather for.\"}},\n",
+    "        },\n",
+    "    },\n",
+    "    {\n",
+    "        \"name\": \"send_email\",\n",
+    "        \"description\": \"Sends an email with the specified subject and body to the recipient.\",\n",
+    "        \"parameters\": {\n",
+    "            \"type\": \"object\",\n",
+    "            \"properties\": {\n",
+    "                \"recipient\": {\"type\": \"string\", \"description\": \"Email address of the recipient.\"},\n",
+    "                \"subject\": {\"type\": \"string\", \"description\": \"Subject of the email.\"},\n",
+    "                \"body\": {\"type\": \"string\", \"description\": \"Body content of the email.\"},\n",
+    "            },\n",
+    "        },\n",
+    "    },\n",
+    "    {\n",
+    "        \"name\": \"get_calendar\",\n",
+    "        \"description\": \"Retrieves calendar events for a specified date range.\",\n",
+    "        \"parameters\": {\n",
+    "            \"type\": \"object\",\n",
+    "            \"properties\": {\n",
+    "                \"start_date\": {\"type\": \"string\", \"description\": \"Start date for calendar events.\"},\n",
+    "                \"end_date\": {\"type\": \"string\", \"description\": \"End date for calendar events.\"},\n",
+    "            },\n",
+    "        },\n",
+    "    },\n",
+    "]\n",
+    "\n",
+    "# Append to evaluation_contents (no separate tool_calls since they're in response)\n",
+    "evaluation_contents.append(\n",
+    "    SourceFileContentContent(\n",
+    "        item={\n",
+    "            \"query\": query,\n",
+    "            \"response\": response,\n",
+    "            \"tool_calls\": None,\n",
+    "            \"tool_definitions\": tool_definitions_agent\n",
+    "        }\n",
+    "    )\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "682839b8",
+   "metadata": {},
+   "source": [
+    "#### Query as Conversation History (List of Messages)\n",
+    "The evaluator also supports query as a list of messages representing conversation history. This helps determine if the Agent selected appropriate tools based on the conversation context."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "218ade7a",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Test Case 4: Query as conversation history instead of a single string\n",
+    "query_as_conversation = [\n",
+    "    {\n",
+    "        \"role\": \"system\",\n",
+    "        \"content\": \"You are a helpful assistant that can fetch weather information and send emails.\"\n",
+    "    },\n",
+    "    {\n",
+    "        \"role\": \"user\", \n",
+    "        \"content\": \"Hi, can you check the weather in Seattle for me?\"\n",
+    "    },\n",
+    "    {\n",
+    "        \"role\": \"user\",\n",
+    "        \"content\": \"Actually, could you also send me an email with that weather information to john@example.com?\"\n",
+    "    }\n",
+    "]\n",
+    "\n",
+    "tool_calls_conversation = [\n",
+    "    {\n",
+    "        \"type\": \"tool_call\",\n",
+    "        \"tool_call_id\": \"call_weather_123\",\n",
+    "        \"name\": \"fetch_weather\",\n",
+    "        \"arguments\": {\"location\": \"Seattle\"},\n",
+    "    },\n",
+    "    {\n",
+    "        \"type\": \"tool_call\", \n",
+    "        \"tool_call_id\": \"call_email_456\",\n",
+    "        \"name\": \"send_email\",\n",
+    "        \"arguments\": {\n",
+    "            \"recipient\": \"john@example.com\",\n",
+    "            \"subject\": \"Weather Information for Seattle\",\n",
+    "            \"body\": \"Here is the weather information you requested.\"\n",
+    "        },\n",
+    "    },\n",
+    "]\n",
+    "\n",
+    "tool_definitions_conversation = [\n",
+    "    {\n",
+    "        \"name\": \"fetch_weather\",\n",
+    "        \"description\": \"Fetches the weather information for the specified location.\",\n",
+    "        \"parameters\": {\n",
+    "            \"type\": \"object\",\n",
+    "            \"properties\": {\"location\": {\"type\": \"string\", \"description\": \"The location to fetch weather for.\"}},\n",
+    "        },\n",
+    "    },\n",
+    "    {\n",
+    "        \"name\": \"send_email\",\n",
+    "        \"description\": \"Sends an email with the specified subject and body to the recipient.\",\n",
+    "        \"parameters\": {\n",
+    "            \"type\": \"object\",\n",
+    "            \"properties\": {\n",
+    "                \"recipient\": {\"type\": \"string\", \"description\": \"Email address of the recipient.\"},\n",
+    "                \"subject\": {\"type\": \"string\", \"description\": \"Subject of the email.\"},\n",
+    "                \"body\": {\"type\": \"string\", \"description\": \"Body content of the email.\"},\n",
+    "            },\n",
+    "        },\n",
+    "    },\n",
+    "    {\n",
+    "        \"name\": \"get_calendar\",\n",
+    "        \"description\": \"Retrieves calendar events for a specified date range.\",\n",
+    "        \"parameters\": {\n",
+    "            \"type\": \"object\",\n",
+    "            \"properties\": {\n",
+    "                \"start_date\": {\"type\": \"string\", \"description\": \"Start date for calendar events.\"},\n",
+    "                \"end_date\": {\"type\": \"string\", \"description\": \"End date for calendar events.\"},\n",
+    "            },\n",
+    "        },\n",
+    "    },\n",
+    "]\n",
+    "\n",
+    "# Append to evaluation_contents\n",
+    "evaluation_contents.append(\n",
+    "    SourceFileContentContent(\n",
+    "        item={\n",
+    "            \"query\": query_as_conversation,\n",
+    "            \"response\": None,\n",
+    "            \"tool_calls\": tool_calls_conversation,\n",
+    "            \"tool_definitions\": tool_definitions_conversation\n",
+    "        }\n",
+    "    )\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "406bc90c",
+   "metadata": {},
+   "source": [
+    "#### Example of Poor Tool Selection"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "2c451e4e",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Test Case 5: Example of poor tool selection\n",
+    "query_poor = \"How is the weather in Seattle?\"\n",
+    "# Using irrelevant tool for the task\n",
+    "poor_tool_calls = [\n",
+    "    {\n",
+    "        \"type\": \"tool_call\",\n",
+    "        \"tool_call_id\": \"call_calendar_123\",\n",
+    "        \"name\": \"get_calendar\",\n",
+    "        \"arguments\": {\"start_date\": \"2025-01-01\", \"end_date\": \"2025-01-31\"},\n",
+    "    },\n",
+    "]\n",
+    "\n",
+    "# This should score poorly as get_calendar is not relevant for weather queries\n",
+    "evaluation_contents.append(\n",
+    "    SourceFileContentContent(\n",
+    "        item={\n",
+    "            \"query\": query_poor,\n",
+    "            \"response\": None,\n",
+    "            \"tool_calls\": poor_tool_calls,\n",
+    "            \"tool_definitions\": tool_definitions_conversation\n",
+    "        }\n",
+    "    )\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "ab4a46c7",
+   "metadata": {},
+   "source": [
+    "#### Response as String (str)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "e34bec52",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Test Case 6: Response as string\n",
+    "query_str = \"What's the weather in New York?\"\n",
+    "\n",
+    "# Response as a simple string (not commonly used, but supported by the API)\n",
+    "# When using string response, tool_calls should be provided separately\n",
+    "response_str = \"The current weather in New York is sunny with a temperature of 22°C.\"\n",
+    "\n",
+    "tool_calls_str = [\n",
+    "    {\n",
+    "        \"type\": \"tool_call\",\n",
+    "        \"tool_call_id\": \"call_ny_weather_789\",\n",
+    "        \"name\": \"fetch_weather\",\n",
+    "        \"arguments\": {\"location\": \"New York\"},\n",
+    "    }\n",
+    "]\n",
+    "\n",
+    "tool_definitions_str = [\n",
+    "    {\n",
+    "        \"name\": \"fetch_weather\",\n",
+    "        \"description\": \"Fetches the weather information for the specified location.\",\n",
+    "        \"parameters\": {\n",
+    "            \"type\": \"object\",\n",
+    "            \"properties\": {\"location\": {\"type\": \"string\", \"description\": \"The location to fetch weather for.\"}},\n",
+    "        },\n",
+    "    },\n",
+    "    {\n",
+    "        \"name\": \"send_email\",\n",
+    "        \"description\": \"Sends an email with the specified subject and body to the recipient.\",\n",
+    "        \"parameters\": {\n",
+    "            \"type\": \"object\",\n",
+    "            \"properties\": {\n",
+    "                \"recipient\": {\"type\": \"string\", \"description\": \"Email address of the recipient.\"},\n",
+    "                \"subject\": {\"type\": \"string\", \"description\": \"Subject of the email.\"},\n",
+    "                \"body\": {\"type\": \"string\", \"description\": \"Body content of the email.\"},\n",
+    "            },\n",
+    "        },\n",
+    "    },\n",
+    "]\n",
+    "\n",
+    "# Append to evaluation_contents\n",
+    "evaluation_contents.append(\n",
+    "    SourceFileContentContent(\n",
+    "        item={\n",
+    "            \"query\": query_str,\n",
+    "            \"response\": response_str,\n",
+    "            \"tool_calls\": tool_calls_str,\n",
+    "            \"tool_definitions\": tool_definitions_str\n",
+    "        }\n",
+    "    )\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "cc857725",
+   "metadata": {},
+   "source": [
+    "#### Tool Definition as Single Dict"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "d94822e1",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Test Case 7: Tool definition as single dict\n",
+    "query_dict = \"How is the weather in Seattle?\"\n",
+    "\n",
+    "tool_definition_dict = [\n",
+    "    {\n",
+    "        \"name\": \"get_weather\",\n",
+    "        \"description\": \"Get the current weather for a location\",\n",
+    "        \"parameters\": {\n",
+    "            \"type\": \"object\",\n",
+    "            \"properties\": {\n",
+    "                \"location\": {\"type\": \"string\", \"description\": \"The city and state, e.g. San Francisco, CA\"},\n",
+    "                \"unit\": {\"type\": \"string\", \"enum\": [\"celsius\", \"fahrenheit\"]}\n",
+    "            },\n",
+    "            \"required\": [\"location\"]\n",
+    "        }\n",
+    "    }\n",
+    "]\n",
+    "\n",
+    "tool_calls_dict = [\n",
+    "    {\n",
+    "        \"type\": \"tool_call\",\n",
+    "        \"tool_call_id\": \"call_abc123\",\n",
+    "        \"name\": \"get_weather\",\n",
+    "        \"arguments\": {\"location\": \"Seattle, WA\", \"unit\": \"celsius\"}\n",
+    "    }\n",
+    "]\n",
+    "\n",
+    "# Append to evaluation_contents\n",
+    "evaluation_contents.append(\n",
+    "    SourceFileContentContent(\n",
+    "        item={\n",
+    "            \"query\": query_dict,\n",
+    "            \"response\": None,\n",
+    "            \"tool_calls\": tool_calls_dict,\n",
+    "            \"tool_definitions\": tool_definition_dict\n",
+    "        }\n",
+    "    )\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "922df035",
+   "metadata": {},
+   "source": [
+    "### Run Evaluation on All Test Cases\n",
+    "\n",
+    "Now that we've defined all test cases, let's run the evaluation once on all of them."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "f0266598",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "results = run_evaluator(\n",
+    "    evaluator_name=\"tool_selection\",\n",
+    "    evaluation_contents=evaluation_contents,\n",
+    "    data_source_config=data_source_config,\n",
+    "    initialization_parameters=initialization_parameters,\n",
+    "    data_mapping=data_mapping\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "eb2dc7d1",
+   "metadata": {},
+   "source": [
+    "### Display Results\n",
+    "\n",
+    "View the evaluation results for each test case."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "7929cda3",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "pprint(results)"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "samples",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.19"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
diff --git a/scenarios/evaluate/Supported_Evaluation_Metrics/Agent_Evaluation/AI_Judge_Evaluators_Response_Completeness.ipynb b/scenarios/evaluate/Supported_Evaluation_Metrics/Agent_Evaluation/AI_Judge_Evaluators_Response_Completeness.ipynb
deleted file mode 100644
index 06142292..00000000
--- a/scenarios/evaluate/Supported_Evaluation_Metrics/Agent_Evaluation/AI_Judge_Evaluators_Response_Completeness.ipynb
+++ /dev/null
@@ -1,231 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "### Response Completeness Evaluator"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "### Getting Started\n",
-    "\n",
-    "This sample demonstrates how to use Response Completeness Evaluator on agent's response when ground truth is provided. This evaluator is helpful when you have ground truth to assess the quality of the agent's final response. \n",
-    "\n",
-    "## Time\n",
-    "\n",
-    "You should expect to spend about 20 minutes running this notebook. \n",
-    "\n",
-    "## Before you begin\n",
-    "For quality evaluation, you need to deploy a `gpt` model supporting JSON mode. We recommend a model `gpt-4o` or `gpt-4o-mini` for their strong reasoning capabilities.    \n",
-    "\n",
-    "### Prerequisite\n",
-    "```bash\n",
-    "pip install azure-ai-projects azure-identity azure-ai-evaluation\n",
-    "```\n",
-    "Set these environment variables with your own values:\n",
-    "1) **AZURE_AI_PROJECT** - The project connection string, as found in the overview page of your Azure AI Foundry project.\n",
-    "2) **MODEL_DEPLOYMENT_NAME** - The deployment name of the model for this AI-assisted evaluator, as found under the \"Name\" column in the \"Models + endpoints\" tab in your Azure AI Foundry project.\n",
-    "3) **AZURE_OPENAI_ENDPOINT** - Azure Open AI Endpoint to be used for evaluation.\n",
-    "4) **AZURE_OPENAI_API_KEY** - Azure Open AI Key to be used for evaluation.\n",
-    "5) **AZURE_OPENAI_API_VERSION** - Azure Open AI Api version to be used for evaluation.\n"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "The Response Completeness evaluator assesses the quality of an agent response by examining how well it aligns with the provided ground truth. The evaluation is based on the following scoring system:\n",
-    "\n",
-    "<pre>\n",
-    "Score 1: Fully incomplete: The response misses all necessary and relevant information compared to the ground truth.\n",
-    "Score 2: Barely complete: The response contains only a small percentage of the necessary information.\n",
-    "Score 3: Moderately complete: The response includes about half of the necessary information.\n",
-    "Score 4: Mostly complete: The response contains most of the necessary information, with only minor omissions.\n",
-    "Score 5: Fully complete: The response perfectly matches all necessary and relevant information from the ground truth.\n",
-    "</pre>\n",
-    "\n",
-    "The evaluation requires the following inputs:\n",
-    "\n",
-    "- Response: The response to be evaluated. (string)\n",
-    "- Ground Truth: The correct and complete information against which the response is compared. (string)\n",
-    "\n",
-    "The evaluator uses these inputs to determine the completeness score, ensuring that the response meaningfully addresses the query while adhering to the provided definitions and data."
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "### Initialize Completeness Evaluator\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from azure.ai.evaluation import ResponseCompletenessEvaluator, AzureOpenAIModelConfiguration\n",
-    "from pprint import pprint\n",
-    "import os\n",
-    "\n",
-    "model_config = AzureOpenAIModelConfiguration(\n",
-    "    azure_endpoint=os.environ[\"AZURE_OPENAI_ENDPOINT\"],\n",
-    "    api_key=os.environ[\"AZURE_OPENAI_API_KEY\"],\n",
-    "    api_version=os.environ[\"AZURE_OPENAI_API_VERSION\"],\n",
-    "    azure_deployment=os.environ[\"MODEL_DEPLOYMENT_NAME\"],\n",
-    ")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from azure.ai.evaluation import AzureOpenAIModelConfiguration\n",
-    "\n",
-    "response_completeness_evaluator = ResponseCompletenessEvaluator(model_config=model_config)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "### Samples"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "#### Evaluating for a ground_truth and response"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# agent response is complete\n",
-    "result = response_completeness_evaluator(\n",
-    "    response=\"Itinery: Day 1 check out the downtown district of the city on train; for Day 2, we can rest in hotel.\",\n",
-    "    ground_truth=\"Itinery: Day 1 take a train to visit the downtown area for city sightseeing; Day 2 rests in hotel.\",\n",
-    ")\n",
-    "result"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# agent response is incomplete\n",
-    "result = response_completeness_evaluator(\n",
-    "    response=\"The order with ID 123 has been shipped and is expected to be delivered on March 15, 2025. However, the order with ID 124 is delayed and should now arrive by March 20, 2025.\",\n",
-    "    ground_truth=\"The order with ID 124 is delayed and should now arrive by March 20, 2025.\",\n",
-    ")\n",
-    "result"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "### Prepare ground truth for agent response\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "import json\n",
-    "from pathlib import Path\n",
-    "\n",
-    "data = [\n",
-    "    {\n",
-    "        \"response\": \"The temperature of Seattle now is 70 degrees. Based on the temperature, having an outdoor office party is recommended.\",\n",
-    "        \"ground_truth\": \"The temperature of Seattle now is 50 degrees. It will be recommended to bring a jacket in the evening.\",\n",
-    "    },\n",
-    "    {\n",
-    "        \"response\": 'The email draft \"Project Plan\" is attached. Please review and provide feedback.',\n",
-    "        \"ground_truth\": 'The email draft \"Project Plan\" is attached. Please review and provide feedback by EOD.',\n",
-    "    },\n",
-    "    {\n",
-    "        \"response\": \"Based on the retrieved documents, the shareholder meeting discussed the operational efficiency of the company and financing options.\",\n",
-    "        \"ground_truth\": \"The shareholder meeting discussed the compensation package of the company CEO.\",\n",
-    "    },\n",
-    "    {\n",
-    "        \"response\": \"The calendar API returns an error code 500. Please check the server logs for more details.\",\n",
-    "        \"ground_truth\": \"The meeting is scheduled for 2 PM tomorrow. Please confirm your availability by EOD.\",\n",
-    "    },\n",
-    "]\n",
-    "\n",
-    "file_path = \"response_completeness_data.jsonl\"\n",
-    "\n",
-    "with Path.open(file_path, \"w\") as file:\n",
-    "    for line in data:\n",
-    "        file.write(json.dumps(line) + \"\\n\")"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Batch evaluate and visualize results on Azure AI Foundry\n",
-    "Batch evaluate to leverage asynchronous evaluation on a dataset. \n",
-    "\n",
-    "Optionally, you can go to AI Foundry URL for rich Azure AI Foundry data visualization. You can inspect the evaluation scores and reasoning to quickly identify bugs and issues of your agent to fix and improve. Make sure to authenticate to Azure using `az login` in your terminal before running this cell."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from azure.ai.evaluation import evaluate\n",
-    "\n",
-    "azure_ai_project = os.environ[\"AZURE_AI_PROJECT\"]\n",
-    "\n",
-    "response = evaluate(\n",
-    "    data=file_path,\n",
-    "    evaluators={\n",
-    "        \"response_completeness\": response_completeness_evaluator,\n",
-    "    },\n",
-    "    azure_ai_project=azure_ai_project,\n",
-    ")\n",
-    "\n",
-    "pprint(f'AI Foundry URL: {response.get(\"studio_url\")}')"
-   ]
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "azureai-sample",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 2
-}
diff --git a/scenarios/evaluate/Supported_Evaluation_Metrics/Agent_Evaluation/agent_utils.py b/scenarios/evaluate/Supported_Evaluation_Metrics/Agent_Evaluation/agent_utils.py
new file mode 100644
index 00000000..4003784c
--- /dev/null
+++ b/scenarios/evaluate/Supported_Evaluation_Metrics/Agent_Evaluation/agent_utils.py
@@ -0,0 +1,84 @@
+# pylint: disable=line-too-long,useless-suppression
+# ------------------------------------
+# Copyright (c) Microsoft Corporation.
+# Licensed under the MIT License.
+# ------------------------------------
+
+import os
+import time
+from pprint import pprint
+
+from azure.identity import DefaultAzureCredential
+from azure.ai.projects import AIProjectClient
+from openai.types.evals.create_eval_jsonl_run_data_source_param import (
+    CreateEvalJSONLRunDataSourceParam,
+    SourceFileContent,
+    SourceFileContentContent,
+)
+
+
+def run_evaluator(
+    evaluator_name: str,
+    evaluation_contents: list[SourceFileContentContent],
+    data_source_config: dict,
+    initialization_parameters: dict[str, str],
+    data_mapping: dict[str, str],
+) -> list:#list of type
+    """
+    Run an evaluator on the provided evaluation contents.
+    
+    Args:
+        evaluator_name: Name of the evaluator (e.g., "tool-success")
+        evaluation_contents: List of test cases to evaluate
+        data_source_config: Configuration for the data source schema
+        initialization_parameters: Parameters for initializing the evaluator
+        data_mapping: Mapping of data fields to evaluator inputs
+        
+    Returns:
+        List of evaluation output items with results
+    """
+    endpoint = os.environ["AZURE_AI_PROJECT_ENDPOINT"]
+    
+    with DefaultAzureCredential() as credential:
+        with AIProjectClient(endpoint=endpoint, credential=credential) as project_client:
+            client = project_client.get_openai_client()
+
+            testing_criteria = [
+                {
+                    "type": "azure_ai_evaluator",
+                    "name": f"{evaluator_name}",
+                    "evaluator_name": f"builtin.{evaluator_name}",
+                    "initialization_parameters": initialization_parameters,
+                    "data_mapping": data_mapping,
+                }
+            ]
+
+            eval_object = client.evals.create(
+                name=f"Test {evaluator_name} Evaluator with inline data",
+                data_source_config=data_source_config,
+                testing_criteria=testing_criteria,
+            )
+
+            eval_run_object = client.evals.runs.create(
+                eval_id=eval_object.id,
+                name="inline_data_run",
+                metadata={"team": "eval-exp", "scenario": "inline-data-v1"},
+                data_source=CreateEvalJSONLRunDataSourceParam(
+                    type="jsonl",
+                    source=SourceFileContent(type="file_content", content=evaluation_contents)
+                ),
+            )
+
+            print(f"Eval Run created with ID: {eval_run_object.id}")
+            print("Waiting for eval run to complete...")
+
+            while True:
+                run = client.evals.runs.retrieve(run_id=eval_run_object.id, eval_id=eval_object.id)
+                if run.status == "completed" or run.status == "failed":
+                    output_items = list(client.evals.runs.output_items.list(run_id=run.id, eval_id=eval_object.id))
+                    print(f"✓ Eval Run Status: {run.status}")
+                    print(f"📊 Eval Run Report URL: {run.report_url}")
+                    
+                    return output_items
+                    
+                time.sleep(5)