ContentHarmEvaluator throws JsonReaderException when evaluating plain text response

[avijayarajait]

Description

When using Microsoft.Extensions.AI.Evaluation.Safety (version 9.10.0-preview.1.25513.3), calling EvaluateAsync with ContentHarmEvaluator fails with:
System.Text.Json.JsonReaderException: 'T' is an invalid start of a value. LineNumber: 0 | BytePositionInLine: 0

Scenario:

Evaluator: ContentHarmEvaluator
Input: ChatMessage and ChatResponse with plain text (e.g., "This very short distance.")
ChatConfiguration built with Azure OpenAI client (AzureOpenAIClient)

Expected:

Evaluator should either return metrics or clearly indicate missing configuration.

Actual:

Throws JSON parsing error because the evaluator expects structured JSON or Foundry configuration.

Minimal Repro

using Microsoft.Extensions.AI.Evaluation;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.AI.Evaluation.Safety;
using Azure;
using Microsoft.Extensions.AI.Evaluation.Quality;
using Microsoft.Extensions.AI;

 var azureClient = new Azure.AI.OpenAI.AzureOpenAIClient(new Uri(endpoint), new AzureKeyCredential(AzureOpenAiKey));
 var aoaiChatClient = azureClient.GetChatClient(model);
 IChatClient client = aoaiChatClient.AsIChatClient();
s_chatConfiguration = new ChatConfiguration(client);

var s_messages = new List<ChatMessage>
{
    new ChatMessage(ChatRole.User, "Hello"),
    new ChatMessage(ChatRole.Assistant, "This very short distance.")
};

var s_response = new ChatResponse(new ChatMessage(ChatRole.Assistant, "This very short distance."));

var contentHarmEvaluator = new ContentHarmEvaluator();
var evaluator = new CompositeEvaluator(new[] { contentHarmEvaluator });

var result = await evaluator.EvaluateAsync(
    messages: s_messages,
    modelResponse: s_response,
    chatConfiguration: s_chatConfiguration // built with AzureOpenAIClient
);

Environment:

• .NET 8 / 9
• Packages:
  1. Microsoft.Extensions.AI.Evaluation.Safety 9.10.0-preview.1.25513.3
  2. Microsoft.Extensions.AI.OpenAI 9.10.0-preview.1.25513.3
  3. Azure.AI.OpenAI 2.1.0

Question:

What am I missing in my setup? How can I correctly configure this evaluator to get an evaluation score without this exception?