Fix/pin dspy version compatibility

.amazonq/docs/baseline-evaluation-fix.md

@@ -0,0 +1,160 @@
+# Baseline Evaluation Fix - Major Milestone
+
+## Issue Summary
+The Nova Prompt Optimizer frontend was failing to generate baseline evaluation scores, consistently returning `None` instead of actual performance metrics. This prevented proper comparison between baseline and optimized prompts.
+
+## Root Cause Analysis
+
+### Primary Issues Identified:
+
+1. **Database Schema Mismatch**
+   - `create_optimization()` was storing `prompt["name"]` instead of `prompt_id`
+   - Caused "prompt not found" errors during optimization runs
+
+2. **Dataset Structure Incompatibility**
+   - Frontend used nested structure: `{'inputs': {'input': '...'}, 'outputs': {'answer': '...'}}`
+   - SDK expected flat structure: `{'input': '...', 'answer': '...'}`
+   - SDK's inference engine couldn't process nested data
+
+3. **Missing Metric Aggregation**
+   - Custom metric classes had empty `batch_apply()` methods (`pass`)
+   - SDK's `aggregate_score()` relied on `batch_apply()` for final score calculation
+   - Individual scores calculated correctly, but final aggregation returned `None`
+
+4. **Import Scoping Conflicts**
+   - Duplicate imports inside functions shadowed global imports
+   - Caused `UnboundLocalError` for `json` and `JSONDatasetAdapter`
+
+## Technical Details
+
+### Error Symptoms:
+```
+🔍 DEBUG - Baseline score from SDK Evaluator: None
+Parameter validation failed: Invalid type for parameter messages[0].content, 
+value: , type: <class 'str'>, valid types: <class 'list'>, <class 'tuple'>
+```
+
+### SDK Workflow Expected:
+```
+TextPromptAdapter → JSONDatasetAdapter → MetricAdapter → BedrockInferenceAdapter → Evaluator.aggregate_score()
+```
+
+## Solution Implementation
+
+### 1. Fixed Database Schema Bug
+**File**: `database.py`
+```python
+# BEFORE (❌ Bug):
+conn.execute("""
+    INSERT INTO optimizations (id, name, prompt, dataset, ...)
+    VALUES (?, ?, ?, ?, ...)
+""", (optimization_id, name, prompt["name"], dataset["name"], ...))
+
+# AFTER (✅ Fixed):
+conn.execute("""
+    INSERT INTO optimizations (id, name, prompt, dataset, ...)
+    VALUES (?, ?, ?, ?, ...)
+""", (optimization_id, name, prompt_id, dataset["name"], ...))
+```
+
+### 2. Implemented Dataset Structure Flattening
+**File**: `sdk_worker.py`
+```python
+# Create flattened JSONL file for baseline evaluation
+flattened_data = []
+for sample in test_dataset.standardized_dataset:
+    flattened_sample = {
+        'input': sample['inputs']['input'],      # Extract from nested
+        'answer': sample['outputs']['answer']    # Extract from nested
+    }
+    flattened_data.append(flattened_sample)
+
+# Write to temporary file and create new dataset adapter
+with tempfile.NamedTemporaryFile(mode='w', suffix='.jsonl', delete=False) as f:
+    for sample in flattened_data:
+        f.write(json.dumps(sample) + '\n')
+    temp_baseline_file = f.name
+
+baseline_dataset_adapter = JSONDatasetAdapter({"input"}, {"answer"})
+baseline_dataset_adapter.adapt(data_source=temp_baseline_file)
+```
+
+### 3. Implemented Proper Metric Aggregation
+**File**: `sdk_worker.py`
+```python
+# BEFORE (❌ Empty):
+def batch_apply(self, y_preds, y_trues):
+    pass  # Not needed for Nova SDK
+
+# AFTER (✅ Proper aggregation):
+def batch_apply(self, y_preds, y_trues):
+    # Calculate average of individual scores
+    scores = [self.apply(pred, true) for pred, true in zip(y_preds, y_trues)]
+    return sum(scores) / len(scores) if scores else 0.0
+```
+
+### 4. Fixed Import Scoping
+**File**: `sdk_worker.py`
+- Removed duplicate `import json` and `import JSONDatasetAdapter` from functions
+- Used global imports consistently
+
+## Results
+
+### Before Fix:
+```
+🔍 DEBUG - Baseline score from SDK Evaluator: None
+❌ No baseline comparison possible
+❌ Frontend showed "Baseline: 0%" 
+```
+
+### After Fix:
+```
+🔍 DEBUG - Baseline score from SDK Evaluator: 0.36206666666666665
+✅ Real baseline score calculated
+✅ Proper baseline vs optimized comparison
+✅ Individual scores: [0.113, 0.44, 1.0, 0.44, 0.113, ...]
+✅ Aggregated score: 0.362 (36.2%)
+```
+
+## Impact
+
+1. **Functional Baseline Evaluation**: SDK now properly evaluates baseline prompts
+2. **Accurate Performance Metrics**: Real scores instead of None/0%
+3. **Proper Optimization Comparison**: Can compare baseline vs optimized performance
+4. **SDK Compatibility**: Frontend now follows official SDK workflow patterns
+5. **Improved Debugging**: Comprehensive logging for troubleshooting
+
+## Key Learnings
+
+1. **SDK Integration**: Always follow official SDK patterns rather than custom implementations
+2. **Data Structure Compatibility**: Ensure data formats match SDK expectations exactly
+3. **Metric Implementation**: Both `apply()` and `batch_apply()` methods are required
+4. **Import Management**: Avoid duplicate imports that can cause scoping issues
+5. **Debugging Strategy**: Layer-by-layer debugging revealed multiple interconnected issues
+
+## Files Modified
+
+- `database.py` - Fixed prompt ID storage in optimizations
+- `sdk_worker.py` - Dataset flattening, metric aggregation, import fixes
+- `app.py` - JSON parsing fix for prompt retrieval
+
+## Testing
+
+Verified with multiple optimization runs:
+- ✅ Baseline evaluation returns real scores (0.362, 0.445, etc.)
+- ✅ No more "Parameter validation failed" in baseline evaluation
+- ✅ Proper dataset structure handling
+- ✅ Successful metric aggregation
+
+## Future Considerations
+
+1. **Performance**: Temporary file creation adds overhead - consider in-memory flattening
+2. **Error Handling**: Add more robust error handling for edge cases
+3. **Validation**: Add dataset structure validation before processing
+4. **Monitoring**: Add metrics to track baseline evaluation success rates
+
+---
+
+**Status**: ✅ **RESOLVED** - Baseline evaluation now fully functional
+**Date**: 2025-08-13
+**Impact**: High - Core functionality restored