Skip to content

EVAL.md

Latest commit

 

History

History
242 lines (148 loc) · 5.54 KB

EVAL.md

File metadata and controls

242 lines (148 loc) · 5.54 KB
Raw

NurtureAI Evaluation Report

1. Overview

This report presents a structured evaluation of the NurtureAI system across a diverse set of parenting-related scenarios, including normal advisory queries, product recommendations, medium-risk health concerns, emergency situations, edge cases, and multilingual inputs.

The evaluation aims to assess:

  • Intent classification accuracy
  • Risk stratification reliability
  • Safety behavior (doctor escalation and emergency handling)
  • Response quality and consistency
  • System latency and performance

A total of 15 test cases were executed covering real-world parental concerns.

2. Summary Metrics

  • Total Test Cases: 15
  • Passed: 12
  • Failed: 3
  • Pass Rate: 80%
  • Errors: 0

Performance

  • Average Latency: 20.39 seconds
  • Maximum Latency: 37.15 seconds

Category-wise Performance

Category Passed Total Accuracy
Normal 4 4 100%
Product 2 2 100%
Medium Risk 2 2 100%
Emergency 2 4 50%
Edge Cases 1 2 50%
Multilingual 1 1 100%

Source:

3. System Strengths

3.1 Strong Performance in Standard Scenarios

The system demonstrates high reliability in:

  • Everyday parenting queries (teething, sleep regression, colic)
  • Product recommendations with contextual reasoning
  • Medium-risk medical advice (e.g., mild fever, diaper rash)

These categories achieved 100% accuracy, indicating stable prompt design and consistent reasoning.

3.2 Effective Risk Awareness (Partial)

The system correctly identifies:

  • Medium-risk situations requiring pediatric consultation
  • High-risk symptoms in many emergency scenarios

Doctor escalation (doctor_flag=True) is generally aligned with risk level in non-edge cases.

3.3 Multilingual Capability

  • Arabic response generation is functional
  • Dual-language output pipeline works end-to-end

However, quality issues exist (see Section 5).

3.4 Emotional Sensitivity

The system handles emotionally distressed parents appropriately:

  • Provides reassurance
  • Avoids alarmist tone
  • Maintains supportive guidance

This is critical for real-world usability.

4. Key Failures and Root Cause Analysis

4.1 Emergency Intent Misclassification

Affected Cases:

  • TC005: Newborn fever
  • TC011: Button battery ingestion

Observed Issue:

  • Correctly identifies critical risk
  • Provides emergency advice
  • BUT classifies intent as "advice" instead of "emergency"

Implication:

  • Downstream systems (alerts, UI escalation, triage) may fail
  • Safety-critical misalignment between reasoning and classification

Root Cause:

  • Intent classification layer is decoupled from risk reasoning
  • No rule enforcing: IF risk == critical → intent must be emergency

4.2 Overestimation in Vague Queries

Affected Case:

  • TC010: "baby crying"

Observed Issue:

  • Risk incorrectly elevated to medium
  • Doctor consultation triggered unnecessarily

Implication:

  • Reduces user trust
  • Causes unnecessary anxiety
  • Over-medicalization of normal scenarios

Root Cause:

  • Lack of uncertainty handling
  • Model defaults to “safe over-escalation” without confidence gating

4.3 Latency Bottlenecks

  • Average response time ~20 seconds
  • Peak latency ~37 seconds

Implication:

  • Poor user experience in real-time scenarios
  • Not suitable for urgent interactions

Likely Causes:

  • Multiple sequential LLM calls (intent + risk + generation)
  • Retrieval or product reasoning overhead
  • No caching or parallelization

4.4 Multilingual Quality Issues

Although Arabic output is present, it shows:

  • Grammar inconsistencies
  • Mixed or incorrect phrasing
  • Reduced fluency compared to English

Implication:

  • Limits usability for native Arabic speakers
  • Reduces perceived system quality

5. Trade-offs in Current System Design

5.1 Safety vs Precision

  • System favors over-caution in ambiguous scenarios
  • Leads to false positives (e.g., unnecessary doctor escalation)

Trade-off: Higher safety → Lower precision in edge cases

5.2 Modular Pipeline vs Consistency

  • Separate components for:

    • Intent classification
    • Risk detection
    • Response generation

Trade-off: Modularity → Inconsistency between outputs (e.g., critical risk but non-emergency intent)

5.3 Rich Responses vs Latency

  • Responses include:

    • Situation analysis
    • Advice
    • Product recommendations
    • Multilingual summaries

Trade-off: High-quality output → Increased latency

5.4 Generalization vs Determinism

  • LLM-driven reasoning allows flexible responses

  • But introduces unpredictability in:

    • Edge cases
    • Classification consistency

6. Conclusion

NurtureAI demonstrates strong performance in normal parenting guidance, product recommendations, and medium-risk medical advice, achieving 100% accuracy in these categories.

However, safety-critical gaps exist in emergency intent classification and edge-case handling, which must be addressed before production deployment.

The system shows a solid architectural foundation, with clear potential for improvement through:

  • Rule-based safety enforcement
  • Better uncertainty handling
  • Latency optimization
  • Enhanced multilingual support

With these refinements, NurtureAI can evolve into a reliable, real-time parenting decision support system suitable for large-scale deployment.