diff --git a/.amazonq/docs/baseline-evaluation-fix.md b/.amazonq/docs/baseline-evaluation-fix.md new file mode 100644 index 0000000..e456577 --- /dev/null +++ b/.amazonq/docs/baseline-evaluation-fix.md @@ -0,0 +1,160 @@ +# Baseline Evaluation Fix - Major Milestone + +## Issue Summary +The Nova Prompt Optimizer frontend was failing to generate baseline evaluation scores, consistently returning `None` instead of actual performance metrics. This prevented proper comparison between baseline and optimized prompts. + +## Root Cause Analysis + +### Primary Issues Identified: + +1. **Database Schema Mismatch** + - `create_optimization()` was storing `prompt["name"]` instead of `prompt_id` + - Caused "prompt not found" errors during optimization runs + +2. **Dataset Structure Incompatibility** + - Frontend used nested structure: `{'inputs': {'input': '...'}, 'outputs': {'answer': '...'}}` + - SDK expected flat structure: `{'input': '...', 'answer': '...'}` + - SDK's inference engine couldn't process nested data + +3. **Missing Metric Aggregation** + - Custom metric classes had empty `batch_apply()` methods (`pass`) + - SDK's `aggregate_score()` relied on `batch_apply()` for final score calculation + - Individual scores calculated correctly, but final aggregation returned `None` + +4. **Import Scoping Conflicts** + - Duplicate imports inside functions shadowed global imports + - Caused `UnboundLocalError` for `json` and `JSONDatasetAdapter` + +## Technical Details + +### Error Symptoms: +``` +๐Ÿ” DEBUG - Baseline score from SDK Evaluator: None +Parameter validation failed: Invalid type for parameter messages[0].content, +value: , type: , valid types: , +``` + +### SDK Workflow Expected: +``` +TextPromptAdapter โ†’ JSONDatasetAdapter โ†’ MetricAdapter โ†’ BedrockInferenceAdapter โ†’ Evaluator.aggregate_score() +``` + +## Solution Implementation + +### 1. Fixed Database Schema Bug +**File**: `database.py` +```python +# BEFORE (โŒ Bug): +conn.execute(""" + INSERT INTO optimizations (id, name, prompt, dataset, ...) + VALUES (?, ?, ?, ?, ...) +""", (optimization_id, name, prompt["name"], dataset["name"], ...)) + +# AFTER (โœ… Fixed): +conn.execute(""" + INSERT INTO optimizations (id, name, prompt, dataset, ...) + VALUES (?, ?, ?, ?, ...) +""", (optimization_id, name, prompt_id, dataset["name"], ...)) +``` + +### 2. Implemented Dataset Structure Flattening +**File**: `sdk_worker.py` +```python +# Create flattened JSONL file for baseline evaluation +flattened_data = [] +for sample in test_dataset.standardized_dataset: + flattened_sample = { + 'input': sample['inputs']['input'], # Extract from nested + 'answer': sample['outputs']['answer'] # Extract from nested + } + flattened_data.append(flattened_sample) + +# Write to temporary file and create new dataset adapter +with tempfile.NamedTemporaryFile(mode='w', suffix='.jsonl', delete=False) as f: + for sample in flattened_data: + f.write(json.dumps(sample) + '\n') + temp_baseline_file = f.name + +baseline_dataset_adapter = JSONDatasetAdapter({"input"}, {"answer"}) +baseline_dataset_adapter.adapt(data_source=temp_baseline_file) +``` + +### 3. Implemented Proper Metric Aggregation +**File**: `sdk_worker.py` +```python +# BEFORE (โŒ Empty): +def batch_apply(self, y_preds, y_trues): + pass # Not needed for Nova SDK + +# AFTER (โœ… Proper aggregation): +def batch_apply(self, y_preds, y_trues): + # Calculate average of individual scores + scores = [self.apply(pred, true) for pred, true in zip(y_preds, y_trues)] + return sum(scores) / len(scores) if scores else 0.0 +``` + +### 4. Fixed Import Scoping +**File**: `sdk_worker.py` +- Removed duplicate `import json` and `import JSONDatasetAdapter` from functions +- Used global imports consistently + +## Results + +### Before Fix: +``` +๐Ÿ” DEBUG - Baseline score from SDK Evaluator: None +โŒ No baseline comparison possible +โŒ Frontend showed "Baseline: 0%" +``` + +### After Fix: +``` +๐Ÿ” DEBUG - Baseline score from SDK Evaluator: 0.36206666666666665 +โœ… Real baseline score calculated +โœ… Proper baseline vs optimized comparison +โœ… Individual scores: [0.113, 0.44, 1.0, 0.44, 0.113, ...] +โœ… Aggregated score: 0.362 (36.2%) +``` + +## Impact + +1. **Functional Baseline Evaluation**: SDK now properly evaluates baseline prompts +2. **Accurate Performance Metrics**: Real scores instead of None/0% +3. **Proper Optimization Comparison**: Can compare baseline vs optimized performance +4. **SDK Compatibility**: Frontend now follows official SDK workflow patterns +5. **Improved Debugging**: Comprehensive logging for troubleshooting + +## Key Learnings + +1. **SDK Integration**: Always follow official SDK patterns rather than custom implementations +2. **Data Structure Compatibility**: Ensure data formats match SDK expectations exactly +3. **Metric Implementation**: Both `apply()` and `batch_apply()` methods are required +4. **Import Management**: Avoid duplicate imports that can cause scoping issues +5. **Debugging Strategy**: Layer-by-layer debugging revealed multiple interconnected issues + +## Files Modified + +- `database.py` - Fixed prompt ID storage in optimizations +- `sdk_worker.py` - Dataset flattening, metric aggregation, import fixes +- `app.py` - JSON parsing fix for prompt retrieval + +## Testing + +Verified with multiple optimization runs: +- โœ… Baseline evaluation returns real scores (0.362, 0.445, etc.) +- โœ… No more "Parameter validation failed" in baseline evaluation +- โœ… Proper dataset structure handling +- โœ… Successful metric aggregation + +## Future Considerations + +1. **Performance**: Temporary file creation adds overhead - consider in-memory flattening +2. **Error Handling**: Add more robust error handling for edge cases +3. **Validation**: Add dataset structure validation before processing +4. **Monitoring**: Add metrics to track baseline evaluation success rates + +--- + +**Status**: โœ… **RESOLVED** - Baseline evaluation now fully functional +**Date**: 2025-08-13 +**Impact**: High - Core functionality restored diff --git a/.amazonq/docs/bedrock-calls-analysis.md b/.amazonq/docs/bedrock-calls-analysis.md new file mode 100644 index 0000000..3867fd8 --- /dev/null +++ b/.amazonq/docs/bedrock-calls-analysis.md @@ -0,0 +1,243 @@ +# Bedrock API Calls Analysis for Nova Prompt Optimization + +## Overview + +This document traces all Amazon Bedrock API calls that occur during a Nova prompt optimization job, using a 5-record dataset as an example. + +## Optimization Flow and Bedrock Calls + +### Phase 1: Optimization Process (Nova SDK) + +#### 1. Meta-Prompter Calls +- **File**: `/src/amzn_nova_prompt_optimizer/core/optimizers/nova_meta_prompter/nova_prompt_template.py` +- **Purpose**: Convert user prompt into structured system/user prompts +- **Bedrock Calls**: 1-2 calls +- **Model**: Nova Premier (as selected) +- **Template Used**: "You are tasked with translating the Original Prompt into a..." + +#### 2. Instruction Generation +- **File**: `/src/amzn_nova_prompt_optimizer/core/optimizers/nova_prompt_optimizer/nova_grounded_proposer.py` +- **Purpose**: Generate optimized instruction candidates using tips (like "high_stakes") +- **Bedrock Calls**: 3-5 calls per instruction candidate +- **Model**: Nova Premier +- **Why**: Creates multiple prompt variations to test +- **Tips Used**: From `NOVA_TIPS` dictionary (high_stakes, creative, simple, etc.) + +#### 3. Few-shot Demo Selection +- **File**: `/.venv/lib/python3.13/site-packages/dspy/propose/grounded_proposer.py` +- **Purpose**: Select best examples from your 5 records to include in prompts +- **Bedrock Calls**: 2-3 calls to evaluate which examples work best +- **Model**: Nova Premier + +#### 4. Prompt Candidate Testing +- **File**: MiPro optimizer (DSPy) +- **Purpose**: Test each prompt candidate against your dataset +- **Bedrock Calls**: ~10-15 calls (3-5 prompt candidates ร— 5 records each) +- **Model**: Nova Premier +- **Why**: Evaluate performance of each optimized prompt + +### Phase 2: Evaluation Process (SDK Worker) + +#### 5. Baseline Evaluation +- **File**: `/frontend/sdk_worker.py` (line 308) +- **Purpose**: Test original prompt against dataset +- **Bedrock Calls**: 5 calls (1 per record) - **ORIGINAL** +- **Bedrock Calls**: 1 call (all records batched) - **OPTIMIZED** +- **Model**: Nova Premier +- **Code**: `baseline_evaluator.aggregate_score()` +- **API**: Uses `BedrockInferenceAdapter.call_model()` + +#### 6. Optimized Evaluation +- **File**: `/frontend/sdk_worker.py` (line 312) +- **Purpose**: Test best optimized prompt against dataset +- **Bedrock Calls**: 5 calls (1 per record) - **ORIGINAL** +- **Bedrock Calls**: 1 call (all records batched) - **OPTIMIZED** +- **Model**: Nova Premier +- **Code**: `optimized_evaluator.aggregate_score()` +- **API**: Uses `BedrockInferenceAdapter.call_model()` + +## Bedrock API Call Locations + +### Direct Bedrock Clients + +1. **BedrockInferenceAdapter** (`/src/amzn_nova_prompt_optimizer/core/inference/adapter.py`) + - **Line 70-73**: Creates `bedrock-runtime` client + - **Line 76-79**: `call_model()` method (main entry point) + - **Line 86**: Calls `converse_client.call_model()` + +2. **BedrockConverseHandler** (`/src/amzn_nova_prompt_optimizer/core/inference/bedrock_converse.py`) + - **Line 48-54**: `client.converse()` call with system prompt + - **Line 56-62**: `client.converse()` call without system prompt + +3. **MetricService** (`/frontend/metric_service.py`) + - **Line 15**: Creates `bedrock-runtime` client + - **Line 36**: `bedrock.invoke_model()` call for metric generation + +4. **BatchedEvaluator** (`/frontend/batched_evaluator.py`) - **NEW** + - **Purpose**: Reduce evaluation API calls by batching multiple records + - **Method**: Combines all dataset records into single prompt + - **Fallback**: Reverts to individual calls if batching fails + +5. **Rate-limited calls through DSPy/LiteLLM** (indirect) + - DSPy uses LiteLLM which calls Bedrock for optimization prompts + - Includes meta-prompter template calls + +## Total Estimated Bedrock Calls + +For a 5-record dataset optimization: + +### Original Implementation +| Phase | Calls | Purpose | +|-------|-------|---------| +| Meta-prompting | 1-2 | Restructure user prompt | +| Instruction generation | 3-5 | Create prompt candidates | +| Demo selection | 2-3 | Select best examples | +| Candidate testing | 10-15 | Test prompt variations | +| Baseline evaluation | 5 | Test original prompt | +| Optimized evaluation | 5 | Test best prompt | +| **Total** | **26-35** | **Complete optimization** | + +### Optimized Implementation (With Batching) +| Phase | Calls | Purpose | Improvement | +|-------|-------|---------|-------------| +| Meta-prompting | 1-2 | Restructure user prompt | No change | +| Instruction generation | 3-5 | Create prompt candidates | No change | +| Demo selection | 2-3 | Select best examples | No change | +| Candidate testing | 10-15 | Test prompt variations | No change | +| Baseline evaluation | 1 | Test original prompt (batched) | **-4 calls** | +| Optimized evaluation | 1 | Test best prompt (batched) | **-4 calls** | +| **Total** | **18-27** | **Complete optimization** | **-8 calls (23% reduction)** | + +## Rate Limiting Analysis + +### Why Rate Limiting Occurs + +With a 20 RPM limit and ~30 calls happening rapidly: + +1. **Burst Pattern**: Calls happen in concentrated bursts during: + - Instruction generation (5-10 calls quickly) + - Candidate testing (10-15 calls quickly) + - Final evaluation (10 calls quickly) + +2. **Rate Limit Exceeded**: 30 calls in ~2-3 minutes exceeds 20 RPM + +### Solutions + +1. **Lower Rate Limit**: Set to 5-10 RPM to spread calls over time +2. **Use Nova Lite**: Higher rate limits than Premier +3. **Reduce Dataset Size**: Fewer records = fewer evaluation calls +4. **Wait Between Runs**: Allow rate limits to reset +5. **Batched Evaluation**: **NEW** - Reduces evaluation calls by 80% + +## Call Flow Diagram + +### Original Flow +``` +User Starts Optimization + โ†“ +Meta-Prompter (1-2 calls) + โ†“ +Instruction Generation (3-5 calls) + โ†“ +Demo Selection (2-3 calls) + โ†“ +Candidate Testing (10-15 calls) + โ†“ +Baseline Evaluation (5 calls) + โ†“ +Optimized Evaluation (5 calls) + โ†“ +Results Displayed +``` + +### Optimized Flow (With Batching) +``` +User Starts Optimization + โ†“ +Meta-Prompter (1-2 calls) + โ†“ +Instruction Generation (3-5 calls) + โ†“ +Demo Selection (2-3 calls) + โ†“ +Candidate Testing (10-15 calls) + โ†“ +Baseline Evaluation (1 batched call) โ† IMPROVED + โ†“ +Optimized Evaluation (1 batched call) โ† IMPROVED + โ†“ +Results Displayed +``` + +## Batching Implementation Details + +### BatchedEvaluator Class (`/frontend/batched_evaluator.py`) + +**Key Features:** +- **Single API Call**: Combines all dataset records into one prompt +- **JSON Response Parsing**: Expects structured array response +- **Fallback Mechanism**: Reverts to individual calls if batching fails +- **Configurable Batch Size**: Default 5 records per batch + +**Batched Prompt Format:** +``` +Process the following inputs and provide responses in the exact format shown: + +Format your response as a JSON array with one response per input: +[{"response": "your_response_1"}, {"response": "your_response_2"}, ...] + +Inputs to process: +1. Hello, I need help with my order +2. Thank you for your excellent service +3. I want to cancel my subscription +4. How do I track my package? +5. What are your business hours? + +Provide exactly 5 responses in JSON array format. +``` + +**Response Parsing:** +- Extracts JSON array from model response +- Falls back to numbered response parsing +- Handles malformed responses gracefully + +### Integration Points + +**Modified Files:** +- `/frontend/sdk_worker.py`: Updated to use `BatchedEvaluator` +- `/frontend/batched_evaluator.py`: New batching implementation + +**Usage:** +```python +# Before +baseline_evaluator = Evaluator(prompt_adapter, test_dataset, metric_adapter, inference_adapter) + +# After +baseline_evaluator = BatchedEvaluator(prompt_adapter, test_dataset, metric_adapter, inference_adapter) +``` + +## Performance Impact + +### API Call Reduction +- **Evaluation Phase**: 80% reduction (10 calls โ†’ 2 calls) +- **Overall Optimization**: 23% reduction (26-35 calls โ†’ 18-27 calls) +- **Rate Limiting**: Significantly reduced due to fewer burst calls + +### Trade-offs +- **Pros**: Fewer API calls, reduced rate limiting, faster evaluation +- **Cons**: More complex response parsing, potential for batch failures +- **Mitigation**: Automatic fallback to individual calls on failure + +## Monitoring Calls + +The optimization monitor page shows meta-prompter calls because they're captured during the optimization process. These are internal Nova SDK calls, not your actual job prompts. + +To see your actual prompts, check the prompt candidates generated after optimization completes. + +## Future Optimizations + +Potential areas for further API call reduction: +1. **Batch Candidate Testing**: Combine multiple prompt candidates in single calls +2. **Parallel Processing**: Use async calls where possible +3. **Caching**: Cache similar prompt evaluations +4. **Smart Sampling**: Use subset of dataset for initial candidate filtering diff --git a/.amazonq/docs/nova-sdk-workflow.md b/.amazonq/docs/nova-sdk-workflow.md new file mode 100644 index 0000000..2e00fa3 --- /dev/null +++ b/.amazonq/docs/nova-sdk-workflow.md @@ -0,0 +1,207 @@ +# Nova Prompt Optimizer SDK - Official Workflow + +## Overview +The Nova Prompt Optimizer SDK follows a structured workflow for prompt optimization using the Nova Meta Prompter + MIPROv2 with Nova Model Tips. + +## Step-by-Step Workflow + +### 1. Initial Setup +- Upload dataset +- Create prompt (system + user prompts) +- Define custom metrics +- Initialize optimization job + +### 2. Dataset Adapter Initialization +```python +# Dataset adapter handles training/test data +dataset_adapter = JSONDatasetAdapter(dataset_path) +``` + +### 3. Prompt Adapter Setup +```python +# Prompt adapter processes system and user prompts +prompt_adapter = TextPromptAdapter() +prompt_adapter.set_system_prompt(content=system_prompt) +prompt_adapter.set_user_prompt(content=user_prompt, variables={"input"}) +``` + +### 4. Metric Adapter Initialization +```python +# Custom metric adapter for evaluation (inherits from MetricAdapter abstract class) +metric_adapter = CustomMetricAdapter() +``` + +### 5. Inference Adapter Setup +```python +# Handles Bedrock API calls with rate limiting +inference_adapter = BedrockInferenceAdapter(region_name="us-east-1", rate_limit=rate_limit) +``` + +### 6. Baseline Evaluation +```python +from amzn_nova_prompt_optimizer.core.evaluation import Evaluator + +# Evaluator uses InferenceRunner internally to generate results then evaluate +evaluator = Evaluator(prompt_adapter, test_set, metric_adapter, inference_adapter) +original_prompt_score = evaluator.aggregate_score(model_id=NOVA_MODEL_ID) + +print(f"Original Prompt Evaluation Score = {original_prompt_score}") +``` + +**Key Components:** +- **Evaluator**: Orchestrates evaluation process +- **InferenceRunner**: Generates inference results using model_id +- **Metric Adapter**: Evaluates output against custom metrics +- **Process**: Generate inference โ†’ Evaluate against metrics โ†’ Aggregate score + +### 7. Optimization Process +```python +from amzn_nova_prompt_optimizer.core.optimizers import NovaPromptOptimizer + +# Nova Prompt Optimizer = Nova Meta Prompter + MIPROv2 + Nova Model Tips +nova_prompt_optimizer = NovaPromptOptimizer( + prompt_adapter=prompt_adapter, + inference_adapter=inference_adapter, + dataset_adapter=train_set, + metric_adapter=metric_adapter +) + +# Run optimization (mode: "lite", "pro", "premier") +optimized_prompt_adapter = nova_prompt_optimizer.optimize(mode="pro") +``` + +### 8. Extract Optimized Results +```python +# Access optimized components +print(optimized_prompt_adapter.system_prompt) # Optimized system prompt +print(optimized_prompt_adapter.user_prompt) # Optimized user prompt +print(optimized_prompt_adapter.few_shot_examples) # Few-shot examples +``` + +### 9. Evaluate Optimized Prompt +```python +# Evaluate optimized prompt using same process as baseline +evaluator = Evaluator(optimized_prompt_adapter, test_set, metric_adapter, inference_adapter) +nova_prompt_optimizer_evaluation_score = evaluator.aggregate_score(model_id=NOVA_MODEL_ID) + +print(f"Nova Prompt Optimizer = {nova_prompt_optimizer_evaluation_score}") +``` + +### 10. Save Results +```python +# Save optimized prompt adapter +optimized_prompt_adapter.save("optimized_prompt/") +``` + +## Key Architecture Components + +### Core Adapters +1. **DatasetAdapter**: Handles training/test data formatting +2. **PromptAdapter**: Manages system/user prompts and variables +3. **MetricAdapter**: Custom evaluation logic (abstract class implementation) +4. **InferenceAdapter**: Bedrock API interface with rate limiting + +### Evaluation System +- **Evaluator**: Main evaluation orchestrator +- **InferenceRunner**: Internal component for generating model responses +- **Process Flow**: Input โ†’ InferenceRunner โ†’ Model Response โ†’ MetricAdapter โ†’ Score + +### Optimization Engine +- **NovaPromptOptimizer**: Meta-optimizer combining: + - Nova Meta Prompter + - MIPROv2 algorithm + - Nova Model Tips +- **Modes**: lite, pro, premier (different optimization intensities) + +## Critical Implementation Notes + +1. **Evaluation Flow**: Evaluator โ†’ InferenceRunner โ†’ Bedrock API โ†’ MetricAdapter โ†’ Aggregate Score +2. **Optimization**: Uses DSPy MIPROv2 internally with Nova-specific enhancements +3. **Rate Limiting**: InferenceAdapter handles Bedrock rate limits +4. **Custom Metrics**: Must inherit from MetricAdapter abstract class +5. **Model Support**: Designed specifically for Nova models (lite, pro, premier) + +## Expected Outputs +- **Baseline Score**: Original prompt performance +- **Optimized Score**: Improved prompt performance +- **Optimized Prompts**: Enhanced system/user prompts +- **Few-shot Examples**: Generated training examples +- **Saved Artifacts**: Serialized optimized prompt adapter + +This workflow ensures systematic prompt optimization using Nova's advanced capabilities while maintaining evaluation consistency between baseline and optimized versions. +--- + +## Frontend Implementation Architecture + +### Two Distinct Bedrock Usage Patterns + +Our frontend implementation separates concerns between enhancement features and core optimization workflow: + +#### 1. Frontend Enhancement Features (Direct Bedrock Calls) +**Purpose**: Productivity features that enhance user experience +**Implementation**: Direct `boto3.client('bedrock-runtime')` calls + +**Use Cases:** +- โœ… **Metric Generation**: AI-powered custom metric creation +- โœ… **Dataset Enhancement**: AI-powered dataset expansion/validation +- โœ… **Prompt Suggestions**: AI-powered initial prompt recommendations +- โœ… **Database Operations**: Store/retrieve user data +- โœ… **UI Intelligence**: Any AI features we add to enhance UX + +**Example:** +```python +def generate_custom_metric(description): + bedrock = boto3.client('bedrock-runtime') # Our direct call + response = bedrock.converse( + modelId="us.amazon.nova-premier-v1:0", + messages=[{"role": "user", "content": f"Generate metric code for: {description}"}], + system=[{"text": "You are a Python code generator..."}] + ) + return response['output']['message']['content'][0]['text'] +``` + +#### 2. Core Optimization Workflow (Official SDK) +**Purpose**: The actual prompt optimization that users expect +**Implementation**: Official Nova SDK classes and methods + +**Use Cases:** +- โœ… **Baseline Evaluation**: Use `Evaluator` class +- โœ… **Optimization Process**: Use `NovaPromptOptimizer` +- โœ… **Final Evaluation**: Use `Evaluator` class again +- โœ… **Result Storage**: Use `optimized_prompt_adapter.save()` + +**Corrected Implementation:** +```python +# Baseline evaluation (OFFICIAL WAY) +baseline_evaluator = Evaluator(prompt_adapter, test_dataset, metric_adapter, inference_adapter) +baseline_score = baseline_evaluator.aggregate_score(model_id=NOVA_MODEL_ID) + +# Optimization (already correct) +nova_optimizer = NovaPromptOptimizer(prompt_adapter, inference_adapter, dataset_adapter, metric_adapter) +optimized_prompt_adapter = nova_optimizer.optimize(mode=model_mode) + +# Final evaluation (OFFICIAL WAY) +optimized_evaluator = Evaluator(optimized_prompt_adapter, test_dataset, metric_adapter, inference_adapter) +optimized_score = optimized_evaluator.aggregate_score(model_id=NOVA_MODEL_ID) + +# Save results (OFFICIAL WAY) +optimized_prompt_adapter.save(f"optimized_prompts/{optimization_id}/") +``` + +### Architecture Benefits + +- โœ… **Clear Separation**: Frontend features vs core optimization +- โœ… **Official Workflow**: Baseline/optimization evaluation uses SDK properly +- โœ… **Enhanced UX**: Keep our AI-powered frontend improvements +- โœ… **Maintainable**: Each system handles what it's designed for +- โœ… **Rate Limiting**: Separate rate limits for different use cases + +### Implementation Guidelines + +1. **Use Direct Bedrock Calls For**: Frontend productivity and enhancement features +2. **Use Official SDK For**: Core prompt optimization workflow (baseline โ†’ optimize โ†’ evaluate) +3. **Separate Rate Limits**: Frontend features get minimal allocation, SDK gets majority +4. **Proper Error Handling**: Each system handles its own error patterns +5. **Data Flow**: Frontend features enhance inputs โ†’ SDK processes optimization โ†’ Frontend displays results + +This dual approach ensures we maintain the official SDK workflow integrity while providing enhanced user experience through AI-powered frontend features. diff --git a/.amazonq/docs/optimized-prompt-builder-design.md b/.amazonq/docs/optimized-prompt-builder-design.md new file mode 100644 index 0000000..13d66c3 --- /dev/null +++ b/.amazonq/docs/optimized-prompt-builder-design.md @@ -0,0 +1,577 @@ +# Optimized Prompt Builder - Design & Implementation Plan + +## ๐Ÿ“‹ **Overview** + +The Optimized Prompt Builder is a declarative prompt construction feature that leverages the Nova SDK's built-in best practices and optimization capabilities. It provides a user-friendly interface for creating high-quality prompts that follow Nova's proven patterns. + +## ๐ŸŽฏ **Objectives** + +- **Democratize prompt engineering** - Enable non-technical users to create optimized prompts +- **Leverage Nova SDK best practices** - Use proven patterns from the SDK's template system +- **Integrate seamlessly** - Work with existing optimization and evaluation workflows +- **Provide real-time feedback** - Show prompt structure and validation as users build + +## ๐Ÿ—๏ธ **Architecture Design** + +### **Core Components** + +``` +โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” +โ”‚ Frontend UI Layer โ”‚ +โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค +โ”‚ Builder Form โ”‚ Preview Panel โ”‚ Validation Panel โ”‚ +โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค +โ”‚ Prompt Builder Service โ”‚ +โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค +โ”‚ Nova SDK Integration Layer โ”‚ +โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค +โ”‚ PromptAdapter โ”‚ Nova Meta Prompter โ”‚ Optimization โ”‚ +โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ +``` + +### **Data Flow** + +1. **User Input** โ†’ Builder Form (Task, Context, Instructions, Format) +2. **Real-time Preview** โ†’ Generated System/User Prompts +3. **Validation** โ†’ Best Practice Compliance Check +4. **Build** โ†’ PromptAdapter Creation +5. **Optimize** โ†’ Nova Meta Prompter Enhancement +6. **Save** โ†’ Database Storage for Reuse + +## ๐Ÿ”ง **Implementation Plan** + +### **Phase 1: Core Builder Service (Week 1)** + +#### **New Files** +``` +frontend/services/prompt_builder.py +``` + +**Key Classes:** +```python +class OptimizedPromptBuilder: + """Declarative prompt builder using Nova best practices""" + + def __init__(self): + self.task: str = "" + self.context: List[str] = [] + self.instructions: List[str] = [] + self.response_format: List[str] = [] + self.variables: Set[str] = set() + self.metadata: Dict[str, Any] = {} + + def set_task(self, description: str) -> 'OptimizedPromptBuilder' + def add_context(self, context: str) -> 'OptimizedPromptBuilder' + def add_instruction(self, instruction: str) -> 'OptimizedPromptBuilder' + def set_response_format(self, format_spec: str) -> 'OptimizedPromptBuilder' + def add_variable(self, name: str) -> 'OptimizedPromptBuilder' + def validate(self) -> Dict[str, List[str]] + def build(self) -> PromptAdapter + def preview(self) -> Dict[str, str] + +class NovaPromptTemplate: + """Nova SDK template integration""" + + @staticmethod + def apply_best_practices(builder: OptimizedPromptBuilder) -> Dict[str, str] + @staticmethod + def validate_structure(prompt_dict: Dict[str, str]) -> List[str] +``` + +#### **Modified Files** +``` +frontend/config.py # Add prompt builder settings +``` + +### **Phase 2: Database Integration (Week 1)** + +#### **New Files** +``` +frontend/migrations/add_prompt_builder.py +``` + +**Database Schema:** +```sql +CREATE TABLE prompt_templates ( + id TEXT PRIMARY KEY, + name TEXT NOT NULL, + description TEXT, + task TEXT NOT NULL, + context_items TEXT, -- JSON array + instructions TEXT, -- JSON array + response_format TEXT, -- JSON array + variables TEXT, -- JSON array + metadata TEXT, -- JSON object + created_date TEXT, + last_modified TEXT +); + +CREATE TABLE prompt_builder_sessions ( + id TEXT PRIMARY KEY, + template_id TEXT, + current_state TEXT, -- JSON object + created_date TEXT, + FOREIGN KEY (template_id) REFERENCES prompt_templates(id) +); +``` + +#### **Modified Files** +``` +frontend/database.py # Add prompt builder methods +``` + +**New Database Methods:** +```python +def create_prompt_template(self, name: str, builder_data: Dict) -> str +def get_prompt_template(self, template_id: str) -> Dict +def list_prompt_templates(self) -> List[Dict] +def update_prompt_template(self, template_id: str, builder_data: Dict) -> bool +def delete_prompt_template(self, template_id: str) -> bool +def save_builder_session(self, session_data: Dict) -> str +def load_builder_session(self, session_id: str) -> Dict +``` + +### **Phase 3: UI Components (Week 2)** + +#### **New Files** +``` +frontend/components/prompt_builder.py +frontend/templates/prompt_builder/builder_form.py +frontend/templates/prompt_builder/preview_panel.py +frontend/templates/prompt_builder/validation_panel.py +``` + +**UI Component Structure:** +```python +# Builder Form Components +def task_input_section() -> Div +def context_builder_section() -> Div +def instructions_builder_section() -> Div +def response_format_section() -> Div +def variables_manager_section() -> Div + +# Preview Components +def system_prompt_preview(prompt_text: str) -> Div +def user_prompt_preview(prompt_text: str) -> Div +def combined_preview(system: str, user: str) -> Div + +# Validation Components +def validation_results(issues: List[str]) -> Div +def best_practices_checklist(checks: Dict[str, bool]) -> Div +def improvement_suggestions(suggestions: List[str]) -> Div +``` + +#### **Modified Files** +``` +frontend/components/layout.py # Add prompt builder navigation +``` + +### **Phase 4: Web Routes & Integration (Week 2)** + +#### **New Files** +``` +frontend/routes/prompt_builder.py +``` + +**Route Structure:** +```python +@app.get("/prompt-builder") +async def prompt_builder_page() + +@app.post("/prompt-builder/preview") +async def preview_prompt(request) + +@app.post("/prompt-builder/validate") +async def validate_prompt(request) + +@app.post("/prompt-builder/build") +async def build_prompt(request) + +@app.post("/prompt-builder/save-template") +async def save_template(request) + +@app.get("/prompt-builder/templates") +async def list_templates() + +@app.get("/prompt-builder/template/{template_id}") +async def load_template(template_id: str) +``` + +#### **Modified Files** +``` +frontend/app.py # Add prompt builder routes +frontend/simple_routes.py # Add "Build Optimized Prompt" option +``` + +### **Phase 5: SDK Integration & Optimization (Week 3)** + +#### **Modified Files** +``` +frontend/sdk_worker.py # Integrate OptimizedPromptBuilder +``` + +**Integration Points:** +```python +def create_prompt_from_builder(builder_data: Dict) -> PromptAdapter +def optimize_built_prompt(prompt_adapter: PromptAdapter, mode: str = "pro") -> PromptAdapter +def evaluate_built_prompt(prompt_adapter: PromptAdapter, dataset_id: str, metric_id: str) -> Dict +``` + +## ๐Ÿ“Š **User Experience Flow** + +### **1. Builder Interface** +``` +โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” +โ”‚ Optimized Prompt Builder โ”‚ +โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค +โ”‚ Task Description: [Text Area] โ”‚ +โ”‚ Context Items: [+ Add Context] [List of contexts] โ”‚ +โ”‚ Instructions: [+ Add Rule] [List of instructions] โ”‚ +โ”‚ Response Format: [Format Builder] [Preview] โ”‚ +โ”‚ Variables: [+ Add Variable] [Variable list] โ”‚ +โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค +โ”‚ [Preview] [Validate] [Build Prompt] [Save Template] โ”‚ +โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ +``` + +### **2. Real-time Preview** +``` +โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” +โ”‚ Generated Prompt Preview โ”‚ +โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค +โ”‚ System Prompt: โ”‚ +โ”‚ โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”‚ +โ”‚ โ”‚ Task: [Generated task description] โ”‚ โ”‚ +โ”‚ โ”‚ Context: [Context items formatted] โ”‚ โ”‚ +โ”‚ โ”‚ Instructions: [Instructions with MUST/DO NOT] โ”‚ โ”‚ +โ”‚ โ”‚ Response Format: [Structured format requirements] โ”‚ โ”‚ +โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚ +โ”‚ โ”‚ +โ”‚ User Prompt: โ”‚ +โ”‚ โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”‚ +โ”‚ โ”‚ [User-facing prompt with variables] โ”‚ โ”‚ +โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚ +โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ +``` + +### **3. Validation Panel** +``` +โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” +โ”‚ Best Practices Validation โ”‚ +โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค +โ”‚ โœ… Task clearly defined โ”‚ +โ”‚ โœ… Context provides sufficient information โ”‚ +โ”‚ โœ… Instructions use strong directive language โ”‚ +โ”‚ โœ… Response format is specific โ”‚ +โ”‚ โš ๏ธ Consider adding more context for complex tasks โ”‚ +โ”‚ โŒ Missing required variables in user prompt โ”‚ +โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค +โ”‚ Suggestions: โ”‚ +โ”‚ โ€ข Add examples to clarify expected output โ”‚ +โ”‚ โ€ข Use "MUST" instead of "should" for requirements โ”‚ +โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ +``` + +## ๐Ÿ”„ **Integration with Existing Features** + +### **Flexible Generator Enhancement** +- Add "Build Optimized Prompt" button alongside existing options +- Allow users to start with builder or import existing prompts +- Seamless transition from builder to optimization workflow + +### **Optimization Workflow** +- Built prompts automatically compatible with existing optimization +- Support for "Optimize Further" with builder-created prompts +- Evaluation integration with custom metrics + +### **Dataset Integration** +- Builder prompts work with existing dataset management +- Variable mapping to dataset columns +- Automatic validation against dataset structure + +## ๐Ÿ“ˆ **Success Metrics** + +### **User Adoption** +- Number of prompts created via builder vs manual creation +- User retention and repeat usage of builder feature +- Time reduction in prompt creation process + +### **Quality Metrics** +- Optimization score improvements for builder-created prompts +- Validation compliance rates +- User satisfaction with generated prompts + +### **Technical Metrics** +- Builder session completion rates +- Template reuse frequency +- Integration success with existing workflows + +## ๐Ÿš€ **Deployment Strategy** + +### **Phase 1 Rollout (Week 1)** +- Core builder service with basic validation +- Simple database integration +- Internal testing with sample prompts + +### **Phase 2 Rollout (Week 2)** +- Full UI implementation +- Real-time preview and validation +- Template management system + +### **Phase 3 Rollout (Week 3)** +- Complete SDK integration +- Optimization workflow integration +- Production deployment with monitoring + +## ๐Ÿ”ง **Technical Considerations** + +### **Performance** +- Real-time preview updates with debouncing +- Efficient template storage and retrieval +- Minimal impact on existing optimization workflows + +### **Scalability** +- Template sharing and collaboration features +- Bulk template operations +- Export/import capabilities + +### **Maintenance** +- Automated testing for prompt generation +- Version control for template changes +- Monitoring and error tracking + +## ๐Ÿงช **Testing Strategy** + +### **Unit Tests** + +#### **New Test Files** +``` +frontend/tests/unit/test_prompt_builder.py +frontend/tests/unit/test_nova_prompt_template.py +frontend/tests/unit/test_prompt_builder_database.py +frontend/tests/unit/test_prompt_builder_routes.py +``` + +**Test Coverage:** +```python +# test_prompt_builder.py +class TestOptimizedPromptBuilder: + def test_set_task_updates_task_field() + def test_add_context_appends_to_list() + def test_add_instruction_with_validation() + def test_set_response_format_validates_structure() + def test_add_variable_updates_set() + def test_validate_returns_issues_for_incomplete_prompt() + def test_build_creates_valid_prompt_adapter() + def test_preview_generates_system_and_user_prompts() + +# test_nova_prompt_template.py +class TestNovaPromptTemplate: + def test_apply_best_practices_formats_correctly() + def test_validate_structure_catches_missing_sections() + def test_variable_injection_preserves_all_variables() + def test_system_user_prompt_separation() + +# test_prompt_builder_database.py +class TestPromptBuilderDatabase: + def test_create_prompt_template_returns_id() + def test_get_prompt_template_returns_correct_data() + def test_list_prompt_templates_pagination() + def test_update_prompt_template_modifies_existing() + def test_delete_prompt_template_removes_record() + def test_save_builder_session_stores_state() + +# test_prompt_builder_routes.py +class TestPromptBuilderRoutes: + def test_prompt_builder_page_renders() + def test_preview_prompt_returns_formatted_output() + def test_validate_prompt_returns_validation_results() + def test_build_prompt_creates_prompt_adapter() + def test_save_template_stores_in_database() +``` + +### **Integration Tests** + +#### **New Test Files** +``` +frontend/tests/integration/test_prompt_builder_workflow.py +frontend/tests/integration/test_sdk_integration.py +frontend/tests/integration/test_optimization_pipeline.py +``` + +**Integration Test Coverage:** +```python +# test_prompt_builder_workflow.py +class TestPromptBuilderWorkflow: + def test_complete_builder_to_optimization_flow() + def test_template_save_and_reload_workflow() + def test_builder_with_existing_dataset_integration() + def test_validation_feedback_loop() + +# test_sdk_integration.py +class TestSDKIntegration: + def test_prompt_adapter_creation_from_builder() + def test_nova_meta_prompter_optimization() + def test_miprov2_optimization_compatibility() + def test_evaluation_with_built_prompts() + +# test_optimization_pipeline.py +class TestOptimizationPipeline: + def test_builder_prompt_through_full_optimization() + def test_optimize_further_with_builder_prompts() + def test_baseline_vs_optimized_evaluation() + def test_prompt_candidate_extraction() +``` + +### **Test Data & Fixtures** + +#### **New Test Files** +``` +frontend/tests/fixtures/prompt_builder_fixtures.py +frontend/tests/data/sample_builder_templates.json +frontend/tests/data/validation_test_cases.json +``` + +**Test Fixtures:** +```python +# prompt_builder_fixtures.py +@pytest.fixture +def sample_builder_data(): + return { + "task": "Analyze customer feedback sentiment", + "context": ["Customer support context", "Product feedback analysis"], + "instructions": ["Use positive/negative/neutral classification", "Provide confidence scores"], + "response_format": ["JSON format with sentiment and confidence"], + "variables": ["customer_feedback"] + } + +@pytest.fixture +def invalid_builder_data(): + return {"task": "", "context": [], "instructions": []} + +@pytest.fixture +def mock_prompt_adapter(): + # Mock PromptAdapter for testing + pass + +@pytest.fixture +def test_database(): + # In-memory test database + pass +``` + +## ๐Ÿ“Š **Test Execution Strategy** + +### **Local Testing** +```bash +# Run unit tests +pytest frontend/tests/unit/test_prompt_builder*.py -v + +# Run integration tests +pytest frontend/tests/integration/test_prompt_builder*.py -v + +# Run all prompt builder tests +pytest frontend/tests/ -k "prompt_builder" -v + +# Generate coverage report +pytest --cov=frontend/services/prompt_builder --cov-report=html +``` + +### **Test Coverage Goals** +- **Unit Tests**: 95% coverage for core builder logic +- **Integration Tests**: 85% coverage for workflow paths +- **Route Tests**: 90% coverage for API endpoints +- **Database Tests**: 100% coverage for CRUD operations + +### **Performance Tests** +```python +# test_prompt_builder_performance.py +class TestPromptBuilderPerformance: + def test_preview_generation_under_100ms() + def test_validation_response_time() + def test_template_loading_performance() + def test_concurrent_builder_sessions() +``` + +## ๐Ÿ”ง **Modified Implementation Plan** + +### **Phase 1: Core Builder Service + Tests (Week 1)** + +#### **New Files** +``` +frontend/services/prompt_builder.py +frontend/tests/unit/test_prompt_builder.py +frontend/tests/unit/test_nova_prompt_template.py +frontend/tests/fixtures/prompt_builder_fixtures.py +``` + +### **Phase 2: Database Integration + Tests (Week 1)** + +#### **New Files** +``` +frontend/migrations/add_prompt_builder.py +frontend/tests/unit/test_prompt_builder_database.py +frontend/tests/integration/test_prompt_builder_workflow.py +``` + +### **Phase 3: UI Components + Route Tests (Week 2)** + +#### **New Files** +``` +frontend/components/prompt_builder.py +frontend/routes/prompt_builder.py +frontend/tests/unit/test_prompt_builder_routes.py +frontend/tests/data/sample_builder_templates.json +``` + +### **Phase 4: SDK Integration + Integration Tests (Week 3)** + +#### **New Files** +``` +frontend/tests/integration/test_sdk_integration.py +frontend/tests/integration/test_optimization_pipeline.py +frontend/tests/data/validation_test_cases.json +``` + +## ๐Ÿ“ **Updated File Summary** + +### **New Files (15)** +``` +# Core Implementation (6) +frontend/services/prompt_builder.py +frontend/components/prompt_builder.py +frontend/routes/prompt_builder.py +frontend/templates/prompt_builder/builder_form.py +frontend/templates/prompt_builder/preview_panel.py +frontend/migrations/add_prompt_builder.py + +# Unit Tests (4) +frontend/tests/unit/test_prompt_builder.py +frontend/tests/unit/test_nova_prompt_template.py +frontend/tests/unit/test_prompt_builder_database.py +frontend/tests/unit/test_prompt_builder_routes.py + +# Integration Tests (3) +frontend/tests/integration/test_prompt_builder_workflow.py +frontend/tests/integration/test_sdk_integration.py +frontend/tests/integration/test_optimization_pipeline.py + +# Test Data & Fixtures (2) +frontend/tests/fixtures/prompt_builder_fixtures.py +frontend/tests/data/sample_builder_templates.json +``` + +### **Modified Files (7)** +``` +frontend/app.py +frontend/database.py +frontend/components/layout.py +frontend/sdk_worker.py +frontend/simple_routes.py +frontend/config.py +frontend/requirements-test.txt # Add testing dependencies +``` + +**Total Implementation: 22 files (15 new, 7 modified), 3-week timeline with comprehensive testing** + +This design provides a comprehensive yet minimal approach to implementing the Optimized Prompt Builder, leveraging existing infrastructure while adding powerful new capabilities for prompt creation and optimization. diff --git a/.amazonq/docs/rate-limiting-analysis.md b/.amazonq/docs/rate-limiting-analysis.md new file mode 100644 index 0000000..4da6d85 --- /dev/null +++ b/.amazonq/docs/rate-limiting-analysis.md @@ -0,0 +1,304 @@ +# Rate Limiting Analysis for Nova Prompt Optimization + +## Overview + +This document explains the dual rate limiting system in the Nova Prompt Optimizer and why users experience rate limit errors despite setting appropriate limits. + +## Rate Limiting Architecture + +### Dual Rate Limiter System + +The Nova Prompt Optimizer implements **two independent rate limiters** that both use the same configuration value but operate on different API call streams: + +#### 1. Frontend Rate Limiter (BedrockInferenceAdapter) +- **File**: `/src/amzn_nova_prompt_optimizer/core/inference/adapter.py` +- **Class**: `BedrockInferenceAdapter` +- **Purpose**: Controls evaluation phase API calls +- **Scope**: + - Baseline prompt evaluation + - Optimized prompt evaluation + - Custom metric evaluation calls +- **Implementation**: Uses `RateLimiter` utility class +- **Configuration**: `rate_limit` parameter from optimization config + +#### 2. Backend Rate Limiter (RateLimitedLM) +- **File**: `/src/amzn_nova_prompt_optimizer/core/optimizers/miprov2/custom_lm/rate_limited_lm.py` +- **Class**: `RateLimitedLM` +- **Purpose**: Controls DSPy optimization phase API calls +- **Scope**: + - Meta-prompter calls + - Instruction generation + - Few-shot demo selection + - Prompt candidate testing +- **Implementation**: Wraps DSPy language models with rate limiting +- **Configuration**: Same `rate_limit` parameter from optimization config + +## Rate Limiter Implementation Details + +### RateLimiter Utility Class +**File**: `/src/amzn_nova_prompt_optimizer/util/rate_limiter.py` + +```python +class RateLimiter: + """ + Thread-safe rate limiter that controls the frequency of requests. + """ + def __init__(self, rate_limit: int = 2): + self.rate_limit = rate_limit # requests per minute + self.request_times = [] + self.lock = threading.Lock() + + def apply_rate_limiting(self): + # Implements sliding window rate limiting +``` + +**Features**: +- Thread-safe implementation +- Sliding window algorithm +- Configurable requests per minute +- Automatic request spacing + +### RateLimitedLM Wrapper +**File**: `/src/amzn_nova_prompt_optimizer/core/optimizers/miprov2/custom_lm/rate_limited_lm.py` + +```python +class RateLimitedLM(dspy.LM): + """ + A wrapper around DSPy language models that applies rate limiting. + """ + def __init__(self, model: dspy.LM, rate_limit: int = 2): + self.rate_limiter = RateLimiter(rate_limit=rate_limit) + self.wrapped_model = model + + def __call__(self, *args, **kwargs): + self.rate_limiter.apply_rate_limiting() + return self.wrapped_model(*args, **kwargs) +``` + +**Features**: +- Wraps any DSPy language model +- Applies rate limiting before each call +- Maintains original model interface +- Used for both task and prompt models + +## Rate Limit Configuration Flow + +### Configuration Source +```python +# In sdk_worker.py +rate_limit_value = config.get('rate_limit', 2) +``` + +### Frontend Application +```python +# BedrockInferenceAdapter initialization +inference_adapter = BedrockInferenceAdapter( + region_name="us-east-1", + rate_limit=rate_limit_value +) +``` + +### Backend Optimization +```python +# In miprov2_optimizer.py +task_lm = RateLimitedLM( + dspy.LM(f'bedrock/{task_model_id}'), + rate_limit=self.inference_adapter.rate_limit +) +prompt_lm = RateLimitedLM( + dspy.LM(f'bedrock/{prompter_model_id}'), + rate_limit=self.inference_adapter.rate_limit +) +``` + +## The Rate Limiting Problem + +### Why Users Experience Rate Limit Errors + +**Root Cause**: Both rate limiters operate **independently** and **simultaneously**, effectively doubling the actual API call rate. + +**Example Scenario**: +- User sets rate limit: **20 RPM** +- Frontend rate limiter: **20 RPM** for evaluation calls +- Backend rate limiter: **20 RPM** for optimization calls +- **Actual combined rate**: Up to **40 RPM** + +### Call Pattern Analysis + +During optimization, both systems make calls concurrently: + +``` +Time: 0-30 seconds +โ”œโ”€โ”€ Backend: Meta-prompting (2 calls at 20 RPM) +โ”œโ”€โ”€ Backend: Instruction generation (5 calls at 20 RPM) +โ””โ”€โ”€ Backend: Demo selection (3 calls at 20 RPM) + +Time: 30-60 seconds +โ”œโ”€โ”€ Backend: Candidate testing (15 calls at 20 RPM) +โ””โ”€โ”€ Frontend: Evaluation (2 calls at 20 RPM) + +Total: ~27 calls in 60 seconds = 27 RPM (exceeds AWS limits) +``` + +### AWS Bedrock Rate Limits + +**Nova Premier**: +- Default: 10-20 RPM (varies by region/account) +- Burst: Limited burst capacity + +**Nova Lite**: +- Default: 50-100 RPM (higher limits) +- Better for development/testing + +## Solutions and Recommendations + +### 1. Reduce Configured Rate Limit + +**Recommended Settings**: +```python +# For Nova Premier +rate_limit = 5 # Allows ~10 RPM combined + +# For Nova Lite +rate_limit = 15 # Allows ~30 RPM combined +``` + +### 2. Model Selection Strategy + +**Development/Testing**: +- Use **Nova Lite** for higher rate limits +- Faster iteration and testing + +**Production**: +- Use **Nova Premier** for best quality +- Set conservative rate limits (5-8 RPM) + +### 3. Dataset Size Optimization + +**Reduce API Calls**: +- Limit dataset to 3-5 records for testing +- Use batched evaluation (implemented) +- Consider record sampling for large datasets + +### 4. Optimization Timing + +**Sequential Processing**: +- Wait 2-3 minutes between optimization runs +- Allow rate limit windows to reset +- Monitor AWS CloudWatch for actual usage + +## Rate Limiting Best Practices + +### Configuration Guidelines + +1. **Start Conservative**: Begin with 5 RPM and increase gradually +2. **Monitor Usage**: Check AWS CloudWatch metrics +3. **Account for Both Systems**: Remember the dual limiter architecture +4. **Test with Small Datasets**: Validate rate limits before scaling + +### Debugging Rate Limit Issues + +**Check Configuration**: +```bash +# Verify rate limit setting in logs +grep "Rate limit" optimization_logs.txt + +# Look for rate limit debug messages +grep "DEBUG.*rate_limit" optimization_logs.txt +``` + +**Monitor API Calls**: +```bash +# Count Bedrock calls in logs +grep "bedrock" optimization_logs.txt | wc -l + +# Check for throttling errors +grep "ThrottlingException\|Too many requests" optimization_logs.txt +``` + +### Error Patterns + +**Common Rate Limit Errors**: +``` +litellm.RateLimitError: BedrockException - {"message":"Too many requests, please wait before trying again."} +``` + +**Throttling Indicators**: +``` +ClientError: ThrottlingException +HTTP 429 Too Many Requests +``` + +## Implementation Locations + +### Rate Limit Usage Points + +1. **SDK Worker** (`/frontend/sdk_worker.py`): + - Lines 241-243: BedrockInferenceAdapter initialization + - Lines 284-292: Rate limit configuration and logging + +2. **MiPro Optimizer** (`/src/amzn_nova_prompt_optimizer/core/optimizers/miprov2/miprov2_optimizer.py`): + - Lines 254, 256: Task and prompt model rate limiting + - Lines 375, 377: Duplicate rate limiting setup + +3. **Inference Adapter** (`/src/amzn_nova_prompt_optimizer/core/inference/adapter.py`): + - Line 78: Rate limiting application before model calls + - Line 56: Rate limiter initialization + +4. **Rate Limited LM** (`/src/amzn_nova_prompt_optimizer/core/optimizers/miprov2/custom_lm/rate_limited_lm.py`): + - Line 42: Rate limiting before wrapped model calls + +## Future Improvements + +### Potential Optimizations + +1. **Unified Rate Limiter**: Single rate limiter for both systems +2. **Dynamic Rate Adjustment**: Adjust based on AWS response times +3. **Queue-based Processing**: Serialize API calls across systems +4. **Regional Rate Limits**: Different limits per AWS region +5. **Account-aware Limiting**: Adjust based on account quotas + +### Monitoring Enhancements + +1. **Real-time Rate Tracking**: Dashboard showing current API usage +2. **Predictive Throttling**: Slow down before hitting limits +3. **Automatic Backoff**: Exponential backoff on rate limit errors +4. **Usage Analytics**: Historical rate limit usage patterns + +## Troubleshooting Guide + +### Quick Fixes + +1. **Immediate**: Reduce rate limit to 5 RPM +2. **Short-term**: Switch to Nova Lite model +3. **Long-term**: Implement batching optimizations + +### Diagnostic Steps + +1. Check current rate limit setting +2. Review recent optimization logs for throttling +3. Verify AWS account rate limits +4. Test with minimal dataset (2-3 records) +5. Monitor CloudWatch Bedrock metrics + +### Configuration Examples + +**Conservative (Recommended)**: +```json +{ + "model_mode": "lite", + "rate_limit": 5, + "record_limit": 5 +} +``` + +**Aggressive (Advanced Users)**: +```json +{ + "model_mode": "premier", + "rate_limit": 10, + "record_limit": 3 +} +``` + +This dual rate limiting architecture explains why users experience unexpected rate limit errors and provides clear guidance for optimal configuration. diff --git a/.amazonq/docs/refactoring-plan.md b/.amazonq/docs/refactoring-plan.md new file mode 100644 index 0000000..a544ae6 --- /dev/null +++ b/.amazonq/docs/refactoring-plan.md @@ -0,0 +1,489 @@ +# Nova Prompt Optimizer Frontend Refactoring Plan + +## Current State Analysis (Updated August 22, 2025) + +**Current Issues:** +- `app.py` is now 4,506 lines (increased from 4,496) with 53+ routes and 9+ functions +- Monolithic structure with mixed concerns (UI, business logic, routes) +- Difficult to maintain and test +- Poor separation of concerns +- **NEW**: Added dataset generation complexity with multiple approaches + +**Existing Structure (Updated):** +``` +frontend/ +โ”œโ”€โ”€ app.py (4,506 lines) - MONOLITHIC +โ”œโ”€โ”€ components/ (partially organized) +โ”‚ โ”œโ”€โ”€ layout.py +โ”‚ โ”œโ”€โ”€ navbar.py +โ”‚ โ”œโ”€โ”€ ui.py +โ”‚ โ””โ”€โ”€ metrics_page.py +โ”œโ”€โ”€ database.py +โ”œโ”€โ”€ sample_generator.py (NEW - 685 lines) +โ”œโ”€โ”€ dataset_conversation.py (NEW - 522 lines) +โ”œโ”€โ”€ simple_dataset_generator.py (NEW - 85 lines) +โ”œโ”€โ”€ simple_routes.py (NEW - 140 lines) +โ”œโ”€โ”€ sdk_worker.py +โ””โ”€โ”€ other utilities... +``` + +**Recent Changes Added:** +- **Dataset Generation System**: Two approaches (complex conversational + simple direct) +- **AI-Powered Sample Creation**: LLM-based dataset generation with prompt analysis +- **Multiple Generator Interfaces**: Both complex wizard and simple form approaches +- **Prompt Analysis Service**: Natural language processing for requirement extraction + +## Proposed Refactored Structure (Updated) + +``` +frontend/ +โ”œโ”€โ”€ app.py (main FastHTML app setup - ~100 lines) +โ”œโ”€โ”€ config.py (existing - configuration) +โ”œโ”€โ”€ database.py (existing - database layer) +โ”‚ +โ”œโ”€โ”€ routes/ (NEW - route handlers) +โ”‚ โ”œโ”€โ”€ __init__.py +โ”‚ โ”œโ”€โ”€ dashboard.py +โ”‚ โ”œโ”€โ”€ datasets.py +โ”‚ โ”œโ”€โ”€ prompts.py +โ”‚ โ”œโ”€โ”€ metrics.py +โ”‚ โ”œโ”€โ”€ optimization.py +โ”‚ โ”œโ”€โ”€ generator.py (complex dataset generation routes) +โ”‚ โ””โ”€โ”€ simple_generator.py (simple dataset generation routes) +โ”‚ +โ”œโ”€โ”€ services/ (NEW - business logic) +โ”‚ โ”œโ”€โ”€ __init__.py +โ”‚ โ”œโ”€โ”€ dataset_service.py +โ”‚ โ”œโ”€โ”€ prompt_service.py +โ”‚ โ”œโ”€โ”€ optimization_service.py +โ”‚ โ”œโ”€โ”€ sample_generator.py (move existing - complex generator) +โ”‚ โ”œโ”€โ”€ simple_dataset_generator.py (move existing - simple generator) +โ”‚ โ”œโ”€โ”€ dataset_conversation.py (move existing - conversational AI) +โ”‚ โ””โ”€โ”€ prompt_analysis_service.py (NEW - extract from conversation service) +โ”‚ โ””โ”€โ”€ metric_service.py (move existing) +โ”‚ +โ”œโ”€โ”€ components/ (existing - UI components) +โ”‚ โ”œโ”€โ”€ __init__.py +โ”‚ โ”œโ”€โ”€ layout.py (existing) +โ”‚ โ”œโ”€โ”€ navbar.py (existing) +โ”‚ โ”œโ”€โ”€ ui.py (existing) +โ”‚ โ”œโ”€โ”€ metrics_page.py (existing) +โ”‚ โ”œโ”€โ”€ dataset_components.py (NEW) +โ”‚ โ”œโ”€โ”€ prompt_components.py (NEW) +โ”‚ โ””โ”€โ”€ optimization_components.py (NEW) +โ”‚ +โ”œโ”€โ”€ utils/ (NEW - utilities) +โ”‚ โ”œโ”€โ”€ __init__.py +โ”‚ โ”œโ”€โ”€ simple_rate_limiter.py (move existing) +โ”‚ โ”œโ”€โ”€ file_utils.py (NEW) +โ”‚ โ””โ”€โ”€ validation.py (NEW) +โ”‚ +โ”œโ”€โ”€ static/ (NEW - static assets) +โ”‚ โ”œโ”€โ”€ css/ +โ”‚ โ”œโ”€โ”€ js/ +โ”‚ โ””โ”€โ”€ images/ +โ”‚ +โ””โ”€โ”€ templates/ (NEW - if needed for complex pages) + โ””โ”€โ”€ (optional Jinja2 templates) +``` + +## Detailed Refactoring Steps + +### Phase 1: Core Infrastructure (Priority: HIGH) + +#### 1.1 Create New App Structure +- **File:** `app.py` (refactored to ~100 lines) +- **Purpose:** Main FastHTML app setup, middleware, and route registration +- **Content:** + - App initialization + - Middleware setup + - Route imports and registration + - Static file serving + - Error handlers + +#### 1.2 Create Route Modules +- **Files:** `routes/*.py` +- **Purpose:** Separate route handlers by feature area +- **Breakdown:** + - `dashboard.py` - Home dashboard routes + - `datasets.py` - Dataset CRUD operations + - `prompts.py` - Prompt management routes + - `metrics.py` - Metric generation and management + - `optimization.py` - Optimization workflow routes + - `generator.py` - AI dataset generation routes + +### Phase 2: Service Layer (Priority: HIGH) + +#### 2.1 Extract Business Logic +- **Files:** `services/*.py` +- **Purpose:** Move business logic out of routes +- **Actions:** + - Move existing `sample_generator.py` to `services/` (685 lines) + - Move existing `simple_dataset_generator.py` to `services/` (85 lines) + - Move existing `dataset_conversation.py` to `services/` (522 lines) + - Move existing `metric_service.py` to `services/` + - Create new service classes for datasets, prompts, optimization + - **NEW**: Extract prompt analysis logic into separate service + +#### 2.2 Create Service Interfaces +```python +# services/dataset_service.py +class DatasetService: + def create_dataset(self, name, file_data) -> Dict + def get_dataset(self, dataset_id) -> Dict + def list_datasets(self) -> List[Dict] + def delete_dataset(self, dataset_id) -> bool + +# services/prompt_service.py +class PromptService: + def create_prompt(self, name, content) -> Dict + def get_prompt(self, prompt_id) -> Dict + def list_prompts(self) -> List[Dict] + def update_prompt(self, prompt_id, content) -> Dict + +# services/optimization_service.py +class OptimizationService: + def start_optimization(self, config) -> str + def get_optimization_status(self, opt_id) -> Dict + def get_optimization_results(self, opt_id) -> Dict + +# services/generator_service.py (NEW) +class GeneratorService: + def __init__(self): + self.complex_generator = SampleGeneratorService() + self.simple_generator = SimpleDatasetGenerator() + self.conversation_service = DatasetConversationService() + + def generate_complex(self, requirements) -> Dict + def generate_simple(self, prompt_content, num_samples) -> Dict + def analyze_prompt(self, prompt_text) -> Dict +``` + +### Phase 3: Component Organization (Priority: MEDIUM) + +#### 3.1 Expand UI Components +- **Files:** `components/*.py` +- **Purpose:** Create reusable UI components +- **New Components:** + - `dataset_components.py` - Dataset upload, display, management + - `prompt_components.py` - Prompt editor, selector, preview + - `optimization_components.py` - Optimization forms, progress, results + +#### 3.2 Standardize Component Interface +```python +# Standard component signature +def create_component_name(data: Dict, **kwargs) -> Div: + """Component description""" + return Div(...) +``` + +### Phase 4: Utilities and Static Assets (Priority: LOW) + +#### 4.1 Create Utility Modules +- **Files:** `utils/*.py` +- **Purpose:** Shared utility functions +- **Content:** + - File handling utilities + - Validation functions + - Common helpers + +#### 4.2 Organize Static Assets +- **Directory:** `static/` +- **Purpose:** Centralize CSS, JS, images +- **Structure:** + - `css/` - Custom stylesheets + - `js/` - Client-side JavaScript + - `images/` - Static images + +## Migration Strategy (Updated) - ZERO DOWNTIME APPROACH WITH AUTOMATED TESTING + +### Step 0: Test Infrastructure Setup (Pre-Phase 1) +1. Create comprehensive test suite for current functionality +2. Set up automated testing pipeline +3. Establish baseline test coverage +4. **โœ… AUTOMATED VALIDATION** - All tests pass before refactoring begins + +### Step 1: Preparation + Initial Testing +1. Create backup of current `app.py` (4,506 lines) +2. Create new directory structure +3. Set up empty module files with `__init__.py` +4. **NEW**: Plan for dual generator system (complex + simple) +5. **NEW**: Create phase-specific test suites +6. **โœ… APP REMAINS OPERATIONAL** - No changes to existing code yet +7. **๐Ÿงช RUN TESTS**: Baseline functionality validation + +### Step 2: Extract Routes (Week 1) - INCREMENTAL EXTRACTION + TESTING +1. Start with simplest routes (dashboard, static pages) +2. **NEW**: Extract simple generator routes first (`simple_routes.py` โ†’ `routes/simple_generator.py`) +3. **NEW**: Extract complex generator routes (`/datasets/generator/*` โ†’ `routes/generator.py`) +4. **CRITICAL**: Move one route at a time, test immediately +5. Keep old routes as fallback until new routes are verified +6. **โœ… APP FULLY FUNCTIONAL** after each route migration +7. **๐Ÿงช RUN TESTS**: Route extraction validation after each route moved + +### Step 3: Extract Services (Week 2) - PARALLEL DEVELOPMENT + TESTING +1. **NEW**: Move `simple_dataset_generator.py` to `services/` first (smallest, 85 lines) +2. **NEW**: Move `dataset_conversation.py` to `services/` (522 lines) +3. **NEW**: Move `sample_generator.py` to `services/` (685 lines) +4. Update imports gradually, maintain backward compatibility +5. **CRITICAL**: Services work alongside existing code +6. **โœ… APP FULLY FUNCTIONAL** - old and new code coexist +7. **๐Ÿงช RUN TESTS**: Service layer validation after each service moved + +### Step 4: Organize Components (Week 3) - COMPONENT MIGRATION + TESTING +1. Group related UI components +2. **NEW**: Create generator-specific components (`generator_components.py`) +3. Update routes to use new components one by one +4. **CRITICAL**: Maintain existing component imports during transition +5. **โœ… APP FULLY FUNCTIONAL** - gradual component replacement +6. **๐Ÿงช RUN TESTS**: Component integration validation after each component migrated + +### Step 5: Final Cleanup (Week 4) - SAFE REMOVAL + COMPREHENSIVE TESTING +1. Move utilities to `utils/` +2. **NEW**: Consolidate generator configurations +3. Remove old monolithic code **ONLY AFTER** new code is verified +4. Update documentation +5. **โœ… APP FULLY FUNCTIONAL** - cleaner, refactored architecture +6. **๐Ÿงช RUN TESTS**: Full regression testing suite + +## Automated Testing Strategy + +### Test Infrastructure (Created in Step 0) + +#### Unit Tests (`tests/unit/`) +```python +# tests/unit/test_simple_generator.py +def test_simple_generator_creation() +def test_sample_generation() +def test_error_handling() + +# tests/unit/test_dataset_conversation.py +def test_prompt_analysis() +def test_requirements_extraction() +def test_conversation_flow() + +# tests/unit/test_routes.py +def test_dashboard_route() +def test_dataset_routes() +def test_generator_routes() +``` + +#### Integration Tests (`tests/integration/`) +```python +# tests/integration/test_generator_workflow.py +def test_end_to_end_simple_generation() +def test_end_to_end_complex_generation() +def test_prompt_to_dataset_workflow() + +# tests/integration/test_database_operations.py +def test_dataset_crud_operations() +def test_prompt_crud_operations() +def test_optimization_workflow() +``` + +#### API Tests (`tests/api/`) +```python +# tests/api/test_endpoints.py +def test_all_routes_respond() +def test_generator_endpoints() +def test_dataset_upload_endpoints() +def test_optimization_endpoints() +``` + +### Testing Commands (Run After Each Phase) + +#### Phase 1 Testing: +```bash +# Baseline validation +python -m pytest tests/unit/ -v +python -m pytest tests/integration/ -v +python -m pytest tests/api/ -v +``` + +#### Phase 2 Testing (Route Extraction): +```bash +# Route-specific validation +python -m pytest tests/unit/test_routes.py -v +python -m pytest tests/api/test_endpoints.py -v +python -m pytest tests/integration/test_generator_workflow.py -v +``` + +#### Phase 3 Testing (Service Extraction): +```bash +# Service layer validation +python -m pytest tests/unit/test_simple_generator.py -v +python -m pytest tests/unit/test_dataset_conversation.py -v +python -m pytest tests/integration/test_database_operations.py -v +``` + +#### Phase 4 Testing (Component Organization): +```bash +# Component integration validation +python -m pytest tests/integration/ -v +python -m pytest tests/api/ -v +``` + +#### Phase 5 Testing (Final Cleanup): +```bash +# Full regression testing +python -m pytest tests/ -v --cov=. --cov-report=html +python -m pytest tests/performance/ -v # Performance regression tests +``` + +## Zero-Downtime Guarantees with Automated Validation + +### After Phase 1: โœ… **100% Operational + TESTED** +- All existing functionality intact +- New directory structure ready +- **๐Ÿงช All baseline tests pass** + +### After Phase 2: โœ… **100% Operational + TESTED** +- Routes extracted and working +- Old routes removed only after new routes tested +- **๐Ÿงช All route tests pass** + +### After Phase 3: โœ… **100% Operational + TESTED** +- Services extracted and integrated +- Business logic properly separated +- **๐Ÿงช All service tests pass** + +### After Phase 4: โœ… **100% Operational + TESTED** +- Components organized and reusable +- UI improvements implemented +- **๐Ÿงช All integration tests pass** + +### After Phase 5: โœ… **100% OPERATIONAL + IMPROVED + FULLY TESTED** +- Clean, maintainable architecture +- Better performance and organization +- **๐Ÿงช 100% test coverage achieved** + +## Test Automation Pipeline + +### Continuous Integration Setup: +```yaml +# .github/workflows/refactoring-tests.yml +name: Refactoring Validation +on: [push, pull_request] +jobs: + test: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v2 + - name: Setup Python + uses: actions/setup-python@v2 + - name: Install dependencies + run: pip install -r requirements.txt pytest pytest-cov + - name: Run unit tests + run: pytest tests/unit/ -v + - name: Run integration tests + run: pytest tests/integration/ -v + - name: Run API tests + run: pytest tests/api/ -v + - name: Generate coverage report + run: pytest --cov=. --cov-report=xml +``` + +### Pre-Commit Hooks: +```bash +# Run tests before each commit during refactoring +pre-commit install +# Automatically runs: pytest tests/unit/ tests/integration/ +``` + +## Benefits of Refactoring + +### Maintainability +- **Single Responsibility:** Each module has one clear purpose +- **Easier Testing:** Isolated components can be unit tested +- **Code Navigation:** Developers can quickly find relevant code + +### Scalability +- **Feature Addition:** New features can be added without touching core files +- **Team Development:** Multiple developers can work on different modules +- **Performance:** Smaller modules load faster + +### Code Quality +- **Separation of Concerns:** UI, business logic, and data access are separated +- **Reusability:** Components and services can be reused across features +- **Type Safety:** Better type hints and validation + +## Risk Mitigation + +### Testing Strategy +1. **Incremental Migration:** Move one module at a time +2. **Regression Testing:** Test each feature after migration +3. **Rollback Plan:** Keep backup of working version + +### Compatibility +1. **Import Compatibility:** Maintain backward compatibility during transition +2. **Database Schema:** No database changes required +3. **API Compatibility:** Maintain existing route URLs + +## Success Metrics (Updated) + +### Code Quality Metrics +- **Lines per File:** Target <500 lines per file (currently: app.py = 4,506 lines) +- **Cyclomatic Complexity:** Reduce complexity per function +- **Test Coverage:** Achieve >80% test coverage +- **NEW**: Generator Separation: Complex vs Simple generators properly isolated + +### Development Metrics +- **Feature Development Time:** Reduce time to add new features +- **Bug Fix Time:** Reduce time to locate and fix bugs +- **Onboarding Time:** Reduce time for new developers to understand codebase +- **NEW**: Generator Maintenance: Easier to maintain dual generator approaches + +### Architecture Metrics +- **Route Distribution:** No single file >500 lines +- **Service Isolation:** Clear separation between dataset generation approaches +- **Component Reusability:** UI components shared across generator types +- **NEW**: Generator Flexibility: Easy to add new generation approaches + +## Timeline (Updated) + +| Phase | Duration | Deliverables | New Considerations | +|-------|----------|-------------|-------------------| +| Phase 1 | Week 1 | Core infrastructure, route extraction | Extract both generator route systems | +| Phase 2 | Week 2 | Service layer, business logic extraction | Unify dual generator services | +| Phase 3 | Week 3 | Component organization, UI improvements | Create generator-specific components | +| Phase 4 | Week 4 | Utilities, static assets, final cleanup | Consolidate generator configurations | + +## Recent Changes Impact Assessment + +### Positive Impacts: +โœ… **Clean Simple Generator**: `simple_dataset_generator.py` (85 lines) is already well-structured +โœ… **Modular Routes**: `simple_routes.py` (140 lines) demonstrates good separation +โœ… **Service Pattern**: New generators follow service-oriented architecture + +### Challenges Added: +โš ๏ธ **Dual Complexity**: Now have both complex and simple generation approaches +โš ๏ธ **Route Duplication**: Similar functionality in different route handlers +โš ๏ธ **Service Overlap**: Both generators handle similar core functionality + +### Refactoring Priorities (Updated): +1. **HIGH**: Extract generator routes first (they're newest and cleanest) +2. **HIGH**: Create unified generator service interface +3. **MEDIUM**: Consolidate shared generator functionality +4. **LOW**: Maintain backward compatibility during transition + +## Approval Required + +Please review this plan and approve: + +- [ ] Overall architecture and structure +- [ ] Migration strategy and timeline +- [ ] Risk mitigation approach +- [ ] Success metrics and goals + +**Next Steps After Approval:** +1. Create backup and new directory structure +2. Begin Phase 1 implementation +3. Set up testing framework +4. Start incremental migration + +--- + +**Document Version:** 1.0 +**Created:** August 22, 2025 +**Author:** Amazon Q +**Status:** Pending Approval diff --git a/.amazonq/features/MetricBuilder.md b/.amazonq/features/MetricBuilder.md new file mode 100644 index 0000000..1b8e4bd --- /dev/null +++ b/.amazonq/features/MetricBuilder.md @@ -0,0 +1,354 @@ +# Metric Builder Feature - Implementation Plan + +## Overview + +The Metric Builder is a comprehensive feature that allows users to create custom evaluation metrics for their prompt optimization tasks through both visual and natural language interfaces. This addresses the core issue where MetricAdapter is an Abstract Base Class requiring concrete implementations for proper evaluation scoring. + +## Problem Statement + +Currently, optimization evaluations return 0 scores because: +- MetricAdapter is an ABC (Abstract Base Class) requiring concrete implementation +- Users cannot define custom metrics for their specific use cases +- No way to specify evaluation criteria for different output formats (JSON, text, categories, etc.) +- Optimization runs fail or produce meaningless scores without proper metrics + +## Solution Architecture + +### Core Components +1. **Metric Builder UI** - Visual interface for defining metrics +2. **Natural Language Processor** - Converts text descriptions to metric logic +3. **Code Generator** - Creates concrete MetricAdapter subclasses +4. **Database Storage** - Persists metric configurations +5. **Integration Layer** - Connects metrics to optimization runs + +## Database Schema + +### New Table: `metrics` +```sql +CREATE TABLE metrics ( + id TEXT PRIMARY KEY, + name TEXT NOT NULL, + description TEXT, + dataset_format TEXT NOT NULL, -- 'json', 'text', 'classification' + scoring_criteria TEXT NOT NULL, -- JSON string of criteria + generated_code TEXT NOT NULL, -- Python code for MetricAdapter + natural_language_input TEXT, -- Original NL description + created TIMESTAMP DEFAULT CURRENT_TIMESTAMP, + last_used TIMESTAMP, + usage_count INTEGER DEFAULT 0 +); +``` + +### Updated Table: `optimizations` +```sql +-- Add metric_id column to existing optimizations table +ALTER TABLE optimizations ADD COLUMN metric_id TEXT REFERENCES metrics(id); +``` + +## User Interface Design + +### Navigation Structure +``` +Home โ†’ Prompts โ†’ Datasets โ†’ **Metrics** โ†’ Optimizations โ†’ Results +``` + +### Metric Builder Page (`/metrics`) + +#### Header Section +- **Page Title**: "Metric Builder" +- **Subtitle**: "Define custom evaluation metrics for your optimization tasks" +- **Action Buttons**: + - "Create New Metric" (primary button) + - "Import from Template" (secondary button) + +#### Metrics List Section +- **Card-based layout** showing existing metrics +- **Each card displays**: + - Metric name and description + - Dataset format compatibility + - Usage count and last used date + - Actions: Edit, Duplicate, Delete, Test +- **Search and filter** by name, format, creation date + +#### Create/Edit Metric Modal +- **Two-tab interface**: + - Tab 1: "Natural Language" + - Tab 2: "Visual Builder" +- **Preview section** showing generated code +- **Test section** with sample data input/output + +## Feature Implementation Details + +### 1. Natural Language Interface + +#### Input Components +- **Large text area** for natural language description +- **Example prompts** to guide users: + - "Score based on correct sentiment and urgency classification" + - "Evaluate JSON output for category accuracy and completeness" + - "Check if response contains required fields and proper format" + +#### Processing Logic +- **Parse natural language** to identify: + - Output format (JSON, text, classification) + - Scoring criteria (accuracy, completeness, format validation) + - Weighting preferences +- **Generate scoring logic** based on parsed intent +- **Show preview** of generated MetricAdapter code + +#### Example NL Inputs and Outputs +``` +Input: "Score based on correct sentiment (positive/negative/neutral) and urgency (high/medium/low)" +Output: Generates metric checking sentiment and urgency fields with equal weighting + +Input: "Evaluate JSON with categories field containing boolean values, weight category accuracy 70% and format validation 30%" +Output: Generates metric with weighted scoring for categories and JSON validation +``` + +### 2. Visual Builder Interface + +#### Dataset Format Selection +- **Radio buttons** for format types: + - JSON with specific fields + - Text classification + - Multi-label classification + - Custom format + +#### Scoring Criteria Builder +- **Dynamic form** based on selected format +- **For JSON format**: + - Field name inputs + - Field type selection (boolean, string, number, array) + - Scoring method (exact match, similarity, contains) + - Weight slider (0-100%) +- **For text classification**: + - Expected classes/categories + - Matching method (exact, fuzzy, semantic) + - Case sensitivity toggle + +#### Visual Preview +- **Live code preview** showing generated MetricAdapter +- **Sample data testing** with input/output examples +- **Score calculation preview** showing how metrics are computed + +### 3. Code Generation Engine + +#### Template System +```python +# Base template for JSON metrics +class Generated{MetricName}Metric(MetricAdapter): + def parse_json(self, input_string: str): + # JSON parsing logic with fallbacks + + def _calculate_metrics(self, y_pred: Any, y_true: Any) -> Dict: + # Generated scoring logic based on criteria + + def apply(self, y_pred: Any, y_true: Any): + return self._calculate_metrics(y_pred, y_true) + + def batch_apply(self, y_preds: List[Any], y_trues: List[Any]): + # Batch processing logic +``` + +#### Generation Rules +- **Field validation** for JSON formats +- **Weighted scoring** based on user preferences +- **Error handling** for malformed inputs +- **Consistent return format** for optimization compatibility + +### 4. Integration with Optimization Flow + +#### Updated Optimization Page +- **New dropdown**: "Evaluation Metric" +- **Options**: + - "Select a metric..." (default) + - List of user-created metrics + - "Create new metric" (opens metric builder) + +#### Optimization Execution +- **Load selected metric** from database +- **Instantiate generated MetricAdapter** class +- **Use in NovaPromptOptimizer** for proper scoring +- **Store metric_id** in optimization record + +## Implementation Phases + +### Phase 1: Database and Backend (Week 1) +- [ ] Create metrics table schema +- [ ] Add metric_id to optimizations table +- [ ] Implement CRUD operations for metrics +- [ ] Create metric code generation engine +- [ ] Add metric loading to SDK worker + +### Phase 2: Basic UI (Week 2) +- [ ] Create metrics page with navigation +- [ ] Implement metrics list view +- [ ] Add create/edit metric modal +- [ ] Build natural language input interface +- [ ] Add code preview functionality + +### Phase 3: Visual Builder (Week 3) +- [ ] Implement visual builder interface +- [ ] Add dataset format detection +- [ ] Create dynamic form generation +- [ ] Build scoring criteria configurator +- [ ] Add live preview and testing + +### Phase 4: Integration (Week 4) +- [ ] Update optimization page with metric dropdown +- [ ] Integrate metric selection with optimization flow +- [ ] Add metric usage tracking +- [ ] Implement metric templates and examples +- [ ] Add comprehensive testing and validation + +## Technical Specifications + +### Frontend Components + +#### MetricBuilder.py +```python +def create_metrics_page(): + return Div( + create_navbar("metrics"), + Main( + create_metrics_header(), + create_metrics_list(), + create_metric_modal(), + cls="main-content" + ) + ) +``` + +#### MetricModal.py +```python +def create_metric_modal(): + return Div( + # Two-tab interface + create_tab_system([ + {"label": "Natural Language", "content": create_nl_interface()}, + {"label": "Visual Builder", "content": create_visual_builder()} + ]), + # Preview and test sections + create_code_preview(), + create_test_interface(), + cls="metric-modal hidden" + ) +``` + +### Backend Services + +#### MetricService.py +```python +class MetricService: + def generate_metric_code(self, criteria: Dict) -> str: + # Generate MetricAdapter subclass code + + def parse_natural_language(self, description: str) -> Dict: + # Parse NL input to scoring criteria + + def validate_metric(self, code: str) -> bool: + # Validate generated metric code + + def test_metric(self, code: str, sample_data: List) -> Dict: + # Test metric with sample data +``` + +### Database Operations + +#### metrics.py +```python +def create_metric(name: str, criteria: Dict, code: str) -> str: + # Create new metric record + +def get_metrics() -> List[Dict]: + # Get all user metrics + +def get_metric(metric_id: str) -> Dict: + # Get specific metric + +def update_metric_usage(metric_id: str): + # Update usage statistics +``` + +## User Experience Flow + +### Creating a New Metric +1. **Navigate to Metrics page** +2. **Click "Create New Metric"** +3. **Choose input method** (Natural Language or Visual Builder) +4. **Define scoring criteria**: + - NL: "Score based on sentiment accuracy and JSON format validation" + - Visual: Select fields, weights, and validation rules +5. **Preview generated code** +6. **Test with sample data** +7. **Save metric** with name and description + +### Using Metric in Optimization +1. **Navigate to Optimization page** +2. **Select prompt and dataset** +3. **Choose evaluation metric** from dropdown +4. **Configure optimization settings** +5. **Start optimization** with proper metric scoring + +## Error Handling and Validation + +### Metric Creation +- **Validate natural language** input for clarity +- **Check generated code** for syntax errors +- **Test metric** with sample data before saving +- **Prevent duplicate** metric names + +### Optimization Integration +- **Verify metric compatibility** with dataset format +- **Handle missing metrics** gracefully +- **Provide fallback** to default scoring if metric fails +- **Log metric errors** for debugging + +## Success Metrics + +### User Adoption +- **Number of custom metrics** created per user +- **Metric reuse rate** across optimizations +- **User preference** (NL vs Visual Builder) + +### Technical Performance +- **Metric generation success rate** +- **Optimization completion rate** with custom metrics +- **Score accuracy** compared to manual evaluation + +### User Experience +- **Time to create** a functional metric +- **User satisfaction** with generated metrics +- **Support ticket reduction** for evaluation issues + +## Future Enhancements + +### Advanced Features +- **Metric templates** for common use cases +- **Collaborative metrics** sharing between users +- **A/B testing** of different metric configurations +- **Automated metric suggestions** based on dataset analysis + +### Integration Improvements +- **Real-time metric validation** during creation +- **Metric performance analytics** and optimization +- **Export/import** metric configurations +- **API access** for programmatic metric creation + +## Risk Mitigation + +### Technical Risks +- **Code generation failures**: Implement robust templates and validation +- **Performance issues**: Cache generated metrics and optimize execution +- **Security concerns**: Validate and sandbox generated code + +### User Experience Risks +- **Complex interface**: Provide clear examples and guided tutorials +- **Learning curve**: Offer templates and common patterns +- **Integration confusion**: Clear documentation and error messages + +## Conclusion + +The Metric Builder feature addresses a critical gap in the Nova Prompt Optimizer by enabling users to create custom evaluation metrics tailored to their specific use cases. By providing both natural language and visual interfaces, we ensure accessibility for users with different technical backgrounds while maintaining the flexibility needed for complex evaluation scenarios. + +The phased implementation approach allows for iterative development and user feedback incorporation, ensuring the final product meets real-world needs and integrates seamlessly with the existing optimization workflow. diff --git a/.amazonq/features/ai-dataset-generation.md b/.amazonq/features/ai-dataset-generation.md new file mode 100644 index 0000000..0ce5303 --- /dev/null +++ b/.amazonq/features/ai-dataset-generation.md @@ -0,0 +1,246 @@ +# AI Dataset Generation Feature + +## Feature Overview +**Feature Name:** AI-Powered Dataset Generation +**Priority:** High +**Estimated Effort:** Medium + +## Problem Statement +- Users need evaluation datasets to optimize prompts but creating them manually is time-consuming +- Current workflow requires users to bring their own datasets +- Users may not have sufficient or diverse evaluation data +- Creating realistic test cases requires domain expertise + +## Solution Design +### User Experience - Detailed Workflow + +**Step 1: Entry Point** +- Navigate to Datasets page +- Two options: "Add Dataset" (existing) or "Generate Dataset" (new AI feature) +- Click "Generate Dataset" โ†’ Launch wizard + +**Step 2: Prompt Selection (Optional)** +- Option to select existing prompt as reference +- If selected: AI analyzes prompt to understand requirements +- If no prompt: Start from scratch + +**Step 3: Conversational Requirements Gathering** +- AI-powered conversational interface +- AI walks user through decision points using comprehensive checklist: + +**Core Requirements Checklist:** +- [ ] **Role and Persona Definition** + - What role should the LLM adopt? (e.g., "customer support agent", "medical expert") + - What persona characteristics are important? + - What domain expertise is required? + +- [ ] **Task Description** + - What is the specific goal of the dataset? + - What type of interactions/scenarios to generate? + - What is the intended use case for this data? + +- [ ] **Data Characteristics** + - [ ] **Diversity Requirements** + - Variations in phrasing and language style + - Different complexity levels (simple to complex) + - Tone variations (formal, casual, frustrated, etc.) + - Intent variations within the domain + - [ ] **Realism Requirements** + - Real-world scenario authenticity + - Natural user behavior patterns + - Contextually appropriate responses + - [ ] **Edge Cases** + - Challenging or unusual inputs + - Boundary conditions + - Error scenarios and exceptions + - Adversarial cases + - [ ] **Specific Constraints** + - Length limitations (character/word counts) + - Sentiment distribution requirements + - Language or terminology restrictions + - Compliance or safety requirements + +- [ ] **Input Data Format** + - What type of input data? (text, structured data, etc.) + - Input complexity and structure + - Required input fields/attributes + +- [ ] **Output Data Format** + - What should the model output? (classification, extraction, generation) + - Output structure and fields + - Expected response format + +- [ ] **Dataset Output Format** + - File format: JSONL or CSV (user selectable) + - **REQUIRED Column Structure:** + - `input` - The input data/prompt for the model + - `answer` - The expected output/response from the model + - No custom column names - standardized format for optimization pipeline + +- AI asks clarifying questions until all checklist items complete +- Validates understanding before proceeding to sample generation + +**Step 4: Initial Sample Generation** +- Generate 5 sample records based on gathered requirements +- Display each record separately for review + +**Step 5: Open Coding Annotation** +- User can annotate each record individually +- Annotations inform AI about: + - Quality issues + - Missing elements + - Desired improvements + - Format corrections +- AI iterates on records and generation prompt based on annotations + +**Step 6: Final Configuration** +- Specify number of records to generate (10-1000) +- Select dataset output format: JSONL or CSV +- **Standardized columns:** `input` and `answer` (hardcoded) +- Generate full dataset with consistent structure +- Save to database and uploads/ directory + +### Technical Design +- **AI Service Integration:** Use Nova models for dataset generation should be selectable +- **Preview System:** Show generated samples before saving +- **Edit Capability:** Allow users to modify individual samples +- **Format Validation:** Ensure generated data matches expected schema +- **Storage Integration:** Save to existing dataset management system + +## Feature Requirements +### Functional Requirements +- [ ] Natural language dataset description input +- [ ] AI-powered sample generation using Nova models +- [ ] Preview interface showing generated samples +- [ ] Individual sample editing capability +- [ ] Batch regeneration of samples +- [ ] Dataset size specification (10-1000 samples) +- [ ] Output format specification (JSON structure) +- [ ] Save generated dataset to database +- [ ] Integration with existing optimization pipeline + +### Non-Functional Requirements +- [ ] Generate 50 samples in under 30 seconds +- [ ] Support datasets up to 1000 samples +- [ ] Validate JSON structure of generated samples +- [ ] Handle AI generation failures gracefully + +## Implementation Checklist +### Backend Tasks +- [ ] Create conversational AI service for comprehensive requirements gathering +- [ ] Implement role/persona definition system +- [ ] Build task description clarification logic +- [ ] Add data characteristics specification (diversity, realism, edge cases, constraints) +- [ ] Create format specification system (input/output/dataset formats) +- [ ] Implement prompt analysis service (if prompt provided) +- [ ] Build comprehensive checklist validation system +- [ ] Create sample generation service (5 initial records) +- [ ] Implement annotation processing system +- [ ] Add iterative refinement logic based on annotations +- [ ] Create batch dataset generation service +- [ ] Add generator state management with checklist tracking +- [ ] Error handling for each step +- [ ] Standardized dataset format with `input` and `answer` columns +- [ ] JSONL and CSV output format support +- [ ] Integration with existing dataset storage (database + uploads/) + +### Frontend Tasks +- [ ] Add "Generate Dataset" button to Datasets page +- [ ] Create multi-step generator component +- [ ] Build prompt selection interface (optional) +- [ ] Implement conversational AI interface +- [ ] Create requirements checklist tracking +- [ ] Build sample record display (5 individual cards) +- [ ] Implement open coding annotation system +- [ ] Add iteration controls for sample refinement +- [ ] Create final configuration form +- [ ] Add progress tracking for full generation +- [ ] Integration with existing dataset management + +### Integration Tasks +- [ ] Connect to Nova AI models +- [ ] Integrate with existing dataset system +- [ ] End-to-end generation workflow +- [ ] Error handling and user feedback +- [ ] Performance optimization + +## Success Criteria +- [ ] Users can generate 50+ realistic samples in under 1 minute +- [ ] Generated samples are contextually relevant to user description +- [ ] 90%+ of generated samples are valid JSON format +- [ ] Users can successfully use generated datasets in optimization +- [ ] Positive user feedback on sample quality + +## Technical Architecture +### Generator State Management +``` +Step 1: Entry Point โ†’ Step 2: Prompt Selection โ†’ Step 3: Conversational Gathering โ†’ +Step 4: Sample Generation โ†’ Step 5: Annotation & Iteration โ†’ Step 6: Final Generation +``` + +### Components to Create +1. **Dataset Generation Interface** (`components/dataset_generator.py`) + - Multi-step generator interface + - State management between steps + - Progress tracking + +2. **Conversational AI Service** (`dataset_conversation.py`) + - Requirements gathering chatbot with comprehensive checklist + - Role/persona definition guidance + - Task description clarification + - Data characteristics specification (diversity, realism, edge cases, constraints) + - Format specification (input, output, dataset formats) + - Prompt analysis (if provided) + - Checklist completion validation + +3. **Sample Generation & Iteration** (`sample_generator.py`) + - Initial 5-sample generation + - Annotation processing + - Iterative refinement + +4. **Open Coding Annotation System** + - Individual record annotation interface + - Annotation-to-improvement mapping + - Feedback loop to generation prompts + +### API Endpoints +- `/datasets/generator/start` - Initialize generator session +- `/datasets/generator/analyze-prompt` - Analyze selected prompt +- `/datasets/generator/conversation` - Conversational requirements gathering +- `/datasets/generator/generate-samples` - Generate initial 5 samples +- `/datasets/generator/annotate` - Process annotations and iterate +- `/datasets/generator/finalize` - Generate full dataset + +### Data Flow +``` +Optional Prompt โ†’ AI Analysis โ†’ Conversational Gathering โ†’ Requirements Checklist โ†’ +Initial Samples โ†’ User Annotations โ†’ Iterative Refinement โ†’ Final Generation โ†’ +Standardized Format (input/answer columns) โ†’ Database + File Storage +``` + +## Risks & Considerations +- **AI Quality Risk:** Generated samples may not match user expectations + - *Mitigation:* Preview system with regeneration capability +- **Performance Risk:** Large dataset generation may be slow + - *Mitigation:* Progress tracking and batch processing +- **Cost Risk:** AI model calls for large datasets + - *Mitigation:* Reasonable size limits and user awareness + +## Dependencies +- Nova AI model access for generation +- Existing dataset storage system +- Current UI framework and styling + +## User Stories +1. **As a user**, I want to generate a dataset from an existing prompt so that I can create evaluation data for optimization +2. **As a user**, I want an AI to guide me through dataset requirements so that I don't miss important considerations +3. **As a user**, I want to see 5 sample records first so that I can verify the AI understands my needs +4. **As a user**, I want to annotate individual records so that I can provide specific feedback for improvement +5. **As a user**, I want the AI to iterate on samples based on my annotations so that the final dataset meets my standards +6. **As a user**, I want to specify the final dataset size and format so that it integrates with my workflow +7. **As a user**, I want the generated dataset saved automatically so that I can use it immediately for optimization + +--- +**Created:** 2025-08-20 +**Last Updated:** 2025-08-20 +**Status:** Planning diff --git a/.amazonq/features/new-feature-template.md b/.amazonq/features/new-feature-template.md new file mode 100644 index 0000000..b812e22 --- /dev/null +++ b/.amazonq/features/new-feature-template.md @@ -0,0 +1,70 @@ +# Feature Planning Template + +## Feature Overview +**Feature Name:** [Name of the feature] +**Priority:** [High/Medium/Low] +**Estimated Effort:** [Small/Medium/Large] + +## Problem Statement +- What problem does this solve? +- Who is the target user? +- What is the current pain point? + +## Solution Design +### User Experience +- How will users interact with this feature? +- What is the expected workflow? +- What are the key user actions? + +### Technical Design +- What components need to be modified/created? +- What are the data requirements? +- What are the integration points? + +## Feature Requirements +### Functional Requirements +- [ ] Requirement 1 +- [ ] Requirement 2 +- [ ] Requirement 3 + +### Non-Functional Requirements +- [ ] Performance requirements +- [ ] Security considerations +- [ ] Accessibility requirements + +## Implementation Checklist +### Backend Tasks +- [ ] Database schema changes +- [ ] API endpoints +- [ ] Business logic +- [ ] Error handling + +### Frontend Tasks +- [ ] UI components +- [ ] User interactions +- [ ] Form handling +- [ ] State management + +### Integration Tasks +- [ ] Component integration +- [ ] End-to-end testing +- [ ] Error scenarios +- [ ] Edge cases + +## Success Criteria +- [ ] Success metric 1 +- [ ] Success metric 2 +- [ ] User acceptance criteria + +## Risks & Considerations +- Risk 1: [Description and mitigation] +- Risk 2: [Description and mitigation] + +## Dependencies +- Dependency 1: [Description] +- Dependency 2: [Description] + +--- +**Created:** [Date] +**Last Updated:** [Date] +**Status:** [Planning/In Progress/Complete] diff --git a/.amazonq/rules/DEVELOPMENT_RULES.md b/.amazonq/rules/DEVELOPMENT_RULES.md new file mode 100644 index 0000000..a6670f8 --- /dev/null +++ b/.amazonq/rules/DEVELOPMENT_RULES.md @@ -0,0 +1,103 @@ +# Development Rules - Nova Prompt Optimizer Frontend + +## ๐Ÿšจ **CRITICAL RULE: Keep New Files Lightweight** + +**ALL NEW files MUST remain under 700 lines** + +### **Legacy Files (Grandfathered):** + +These existing files are exempt from the 700-line rule: +- `components/layout.py` (938 lines) โœ… EXEMPT +- `database.py` (1270 lines) โœ… EXEMPT +- `sample_generator.py` (777 lines) โœ… EXEMPT +- `sdk_worker.py` (966 lines) โœ… EXEMPT + +**Rule**: Do not make these files significantly larger, but they can remain as-is. + +### **Mandatory Structure for New Features:** + +1. **Route Handlers** โ†’ `routes/feature_name.py` +2. **UI Components** โ†’ `components/feature_name.py` +3. **Business Logic** โ†’ `services/feature_name_service.py` +4. **Database Operations** โ†’ Use existing `database.py` or extend with new methods + +### **New File Requirements:** + +- **Max 700 lines per NEW file** +- **Single responsibility** - one feature per file +- **Naming convention**: `folder/feature_name.py` +- **Setup function for routes**: `setup_feature_routes(app)` +- **Import in app.py**: Add to route setup section + +## **Before Adding ANY New Feature:** + +1. โœ… **Check file sizes**: `python3 check_structure.py` +2. โœ… **If any NEW file > 650 lines**: Extract code to additional files +3. โœ… **Plan file structure**: Which route/component/service files needed? +4. โœ… **Create files following naming conventions** +5. โœ… **Add route setup to app.py import section** + +## **Code Review Checklist:** + +- [ ] All NEW files under 700 lines +- [ ] Legacy files not significantly enlarged +- [ ] New routes in separate `routes/` files +- [ ] UI components in `components/` files +- [ ] Business logic in `services/` files +- [ ] Route setup function added to app.py imports +- [ ] No duplicate code across files +- [ ] Each file has single responsibility + +## **File Size Rules:** + +| File Status | Max Lines | Rule | +|-------------|-----------|------| +| **NEW FILES** | **700** | Strict limit for all new development | +| **LEGACY FILES** | **No Limit** | Grandfathered, avoid major growth | + +## **Current Architecture:** + +``` +โœ… COMPLIANT FILES: +app.py (255 lines) +routes/ (all under 700 lines) +components/ (most under 700 lines) +services/ (all under 700 lines) + +โœ… LEGACY EXEMPT FILES: +components/layout.py (938 lines) - EXEMPT +database.py (1270 lines) - EXEMPT +sample_generator.py (777 lines) - EXEMPT +sdk_worker.py (966 lines) - EXEMPT +``` + +## **Violation Consequences:** + +โŒ **If any NEW file exceeds 700 lines**: STOP development, extract code immediately +โš ๏ธ **If legacy file grows significantly**: Consider refactoring (but not required) +โŒ **If mixing concerns**: Refactor to separate route/component/service responsibilities + +## **Quick Commands:** + +```bash +# Run full compliance check (shows legacy exemptions) +python3 check_structure.py + +# Check specific file +wc -l filename.py + +# Monitor total lines +find . -name "*.py" -not -path "./.venv/*" | xargs wc -l +``` + +## **Current Status:** + +โœ… **All new development**: Must follow 700-line rule +โœ… **Legacy files**: Grandfathered and exempt +โœ… **Route architecture**: Fully compliant and lightweight + +**Focus**: Apply 700-line rule to NEW features only + +--- + +**โšก Remember: 700-line rule for NEW files = Better architecture for future development** diff --git a/.amazonq/rules/Development.md b/.amazonq/rules/Development.md new file mode 100644 index 0000000..9a9feb7 --- /dev/null +++ b/.amazonq/rules/Development.md @@ -0,0 +1,30 @@ +# Nova Prompt Optimizer - Development Rules + +## Code Modification Policy + +### Rule: No Fresh Implementations Without Explicit Approval + +**๐Ÿšซ PROHIBITED WITHOUT PERMISSION:** +- Creating "clean" versions of existing files +- Rewriting entire functions or classes for troubleshooting +- Major refactoring or restructuring of code +- Fresh implementations to replace existing working code + +**โœ… ALLOWED:** +- Minimal, targeted fixes to address specific errors +- Surgical changes to resolve immediate issues +- Small additions or modifications to existing code +- Bug fixes that preserve existing structure + +**๐Ÿ“‹ PROCESS:** +1. **Identify the specific issue** that needs fixing +2. **Make minimal changes** to address only that issue +3. **Preserve existing code structure** and patterns +4. **Ask for explicit permission** before any major changes or rewrites + +**๐ŸŽฏ PRINCIPLE:** +Fix the exact problem with the least amount of code change possible. When in doubt, ask for permission before making broader changes. + +--- + +*This rule ensures code stability and prevents unnecessary disruption to working systems.* diff --git a/.amazonq/rules/Styling.md b/.amazonq/rules/Styling.md new file mode 100644 index 0000000..260a24b --- /dev/null +++ b/.amazonq/rules/Styling.md @@ -0,0 +1,298 @@ +# Nova Prompt Optimizer - Frontend Styling Guide + +## Overview + +This document serves as the comprehensive styling reference for the Nova Prompt Optimizer frontend application. Our styling approach uses **ShadHead with Tailwind CSS** for consistent, modern UI components. + +## Design Philosophy + +### Core Principles +- **Modern & Clean**: Professional appearance with clean lines and modern aesthetics +- **Accessible**: WCAG compliant with proper ARIA labels and semantic HTML +- **Responsive**: Mobile-first design that works across all device sizes +- **Consistent**: Unified design language throughout the application +- **Performance**: Lightweight CSS with minimal dependencies + +## Technology Stack + +### CSS Framework +- **Primary**: ShadHead with Tailwind CSS (`ShadHead(tw_cdn=True, theme_handle=True)`) +- **Component Library**: Shad4FastHTML components +- **Why This Stack**: Theme-aware components, comprehensive utility classes, consistent design system + +### JavaScript Libraries +- **HTMX**: `https://unpkg.com/htmx.org@1.9.10` - Dynamic interactions +- **FastHTML JS**: Built-in FastHTML JavaScript utilities + +### Component Architecture +- **FastHTML Components**: Python-based component system +- **ShadHead**: Provides Tailwind CSS and theme handling +- **CSS Classes**: Tailwind utility classes with theme-aware custom properties + +## Layout System + +### Main Layout Structure +```python +# Location: components/layout.py +def create_main_layout(title, content, current_page="", user=None): + return Html( + Head( + Title(f"{title} - Nova Prompt Optimizer"), + ShadHead(tw_cdn=True, theme_handle=True), + # Additional scripts and styles + ), + Body( + # Navigation bar + create_navbar(current_page, user), + # Main content + Main(content, cls="main-content"), + cls=f"page-{current_page}" if current_page else "" + ) + ) +``` + +## Component Styling Standards + +### Buttons +```python +# Primary Button Classes +cls="inline-flex items-center justify-center whitespace-nowrap rounded-md text-sm font-medium ring-offset-background transition-colors focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-ring focus-visible:ring-offset-2 disabled:pointer-events-none disabled:opacity-50 bg-primary text-primary-foreground hover:bg-primary/90 h-10 px-4 py-2" + +# Secondary Button Classes +cls="inline-flex items-center justify-center whitespace-nowrap rounded-md text-sm font-medium ring-offset-background transition-colors focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-ring focus-visible:ring-offset-2 disabled:pointer-events-none disabled:opacity-50 border border-input bg-background hover:bg-accent hover:text-accent-foreground h-8 px-3 py-1 text-xs" +``` + +### Form Elements +```python +# Input Fields +cls="flex h-10 w-full rounded-md border border-input bg-background px-3 py-2 text-sm ring-offset-background file:border-0 file:bg-transparent file:text-sm file:font-medium placeholder:text-muted-foreground focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-ring focus-visible:ring-offset-2 disabled:cursor-not-allowed disabled:opacity-50" + +# Textarea Fields +cls="flex min-h-[80px] w-full rounded-md border border-input bg-background px-3 py-2 text-sm ring-offset-background placeholder:text-muted-foreground focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-ring focus-visible:ring-offset-2 disabled:cursor-not-allowed disabled:opacity-50" + +# Select Elements +cls="flex h-10 w-full rounded-md border border-input bg-background px-3 py-2 text-sm ring-offset-background file:border-0 file:bg-transparent file:text-sm file:font-medium placeholder:text-muted-foreground focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-ring focus-visible:ring-offset-2 disabled:cursor-not-allowed disabled:opacity-50" +``` + +### Cards +```python +# Card Container +cls="bg-card text-card-foreground shadow-sm border rounded-lg" + +# Card with Header and Body Structure +Div( + Div( + H4("Card Title", cls="card-title"), + cls="card-header" + ), + Div( + # Card content here + cls="card-body" + ), + cls="card" +) +``` + +### Typography +```python +# Headings +H1(cls="text-2xl font-bold mb-4") +H2(cls="text-xl font-semibold mb-4") +H3(cls="text-lg font-semibold mb-2") +H4(cls="font-semibold text-lg mb-2") + +# Text Colors +cls="text-muted-foreground" # Secondary text +cls="text-card-foreground" # Primary text on cards +cls="text-primary" # Brand color text +``` + +## Theme System + +### CSS Custom Properties +The ShadHead component provides theme-aware CSS custom properties: + +```css +/* Available through ShadHead theming */ +--background +--foreground +--card +--card-foreground +--primary +--primary-foreground +--muted +--muted-foreground +--border +--input +--ring +--accent +--accent-foreground +``` + +### Theme Usage +```python +# Use theme-aware classes instead of hardcoded colors +cls="bg-background text-foreground" # โœ… Good +cls="bg-white text-black" # โŒ Avoid + +cls="border-border" # โœ… Good +cls="border-gray-200" # โŒ Avoid + +cls="text-muted-foreground" # โœ… Good +cls="text-gray-600" # โŒ Avoid +``` + +## Layout Patterns + +### Container and Spacing +```python +# Page Container +cls="container mx-auto px-4 py-8" + +# Card Container +cls="max-w-2xl mx-auto" + +# Grid Layouts +cls="grid grid-cols-1 md:grid-cols-2 lg:grid-cols-3 gap-4" + +# Flexbox Layouts +cls="flex items-center justify-between" +cls="flex flex-col space-y-4" +``` + +### Responsive Design +```python +# Mobile-first responsive classes +cls="w-full md:w-1/2 lg:w-1/3" +cls="text-sm md:text-base lg:text-lg" +cls="p-4 md:p-6 lg:p-8" +``` + +## Component Integration + +### Avoiding create_main_layout Issues +When `create_main_layout` causes raw HTML display issues, use manual structure: + +```python +# โŒ Problematic (can cause raw HTML display) +return create_main_layout(title="Page", content=content) + +# โœ… Working alternative +return Html( + ShadHead(tw_cdn=True, theme_handle=True), + Body( + Main( + Div( + # Your content here + cls="container mx-auto px-4 py-8" + ) + ) + ) +) +``` + +### Form Structure +```python +Form( + Div( + Label("Field Label:", cls="block text-sm font-medium mb-2"), + Input(type="text", name="field", cls="[input-classes]"), + cls="mb-4" + ), + Button("Submit", type="submit", cls="[button-classes]"), + method="post", + action="/endpoint" +) +``` + +## Best Practices + +### 1. **Always Use ShadHead** +```python +# Required for proper styling +ShadHead(tw_cdn=True, theme_handle=True) +``` + +### 2. **Use Theme-Aware Classes** +- Prefer `text-muted-foreground` over `text-gray-600` +- Use `bg-background` instead of `bg-white` +- Use `border-border` instead of `border-gray-200` + +### 3. **Consistent Component Classes** +- Copy exact class strings from working components +- Don't modify or shorten the utility class chains +- Include all accessibility and focus classes + +### 4. **Avoid Layout Component Issues** +- Test `create_main_layout` first +- Fall back to manual HTML structure if needed +- Always include proper navigation and container structure + +### 5. **Form Styling** +- Use consistent label styling: `cls="block text-sm font-medium mb-2"` +- Include proper spacing: `cls="mb-4"` between form groups +- Use full utility class chains for inputs and buttons + +## Common Issues and Solutions + +### Raw HTML Display +**Problem**: Components showing as raw HTML text instead of rendering +**Solution**: +1. Ensure `ShadHead(tw_cdn=True, theme_handle=True)` is included +2. Avoid problematic layout components +3. Use manual HTML structure with proper CSS classes + +### Missing Styles +**Problem**: Components not styled properly +**Solution**: +1. Include complete utility class chains +2. Use theme-aware classes (`text-muted-foreground` not `text-gray-600`) +3. Ensure ShadHead is loading Tailwind CSS + +### Button Styling +**Problem**: Buttons don't match other pages +**Solution**: Use exact class strings from working components: +```python +# Copy this exact string for primary buttons +cls="inline-flex items-center justify-center whitespace-nowrap rounded-md text-sm font-medium ring-offset-background transition-colors focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-ring focus-visible:ring-offset-2 disabled:pointer-events-none disabled:opacity-50 bg-primary text-primary-foreground hover:bg-primary/90 h-10 px-4 py-2" +``` + +## Quick Reference + +### Essential Imports +```python +from fasthtml.common import * +from shad4fast import ShadHead, Button, Card +``` + +### Page Structure Template +```python +def create_page(): + return Html( + ShadHead(tw_cdn=True, theme_handle=True), + Body( + Main( + Div( + # Page content + cls="container mx-auto px-4 py-8" + ) + ) + ) + ) +``` + +### Form Template +```python +Form( + Div( + Label("Label:", cls="block text-sm font-medium mb-2"), + Input(type="text", name="field", cls="flex h-10 w-full rounded-md border border-input bg-background px-3 py-2 text-sm ring-offset-background file:border-0 file:bg-transparent file:text-sm file:font-medium placeholder:text-muted-foreground focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-ring focus-visible:ring-offset-2 disabled:cursor-not-allowed disabled:opacity-50"), + cls="mb-4" + ), + Button("Submit", type="submit", cls="inline-flex items-center justify-center whitespace-nowrap rounded-md text-sm font-medium ring-offset-background transition-colors focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-ring focus-visible:ring-offset-2 disabled:pointer-events-none disabled:opacity-50 bg-primary text-primary-foreground hover:bg-primary/90 h-10 px-4 py-2"), + method="post", + action="/endpoint" +) +``` + +This styling guide reflects the actual implementation used throughout the Nova Prompt Optimizer application. diff --git a/.gitignore b/.gitignore index fa90de1..8f31bb7 100644 --- a/.gitignore +++ b/.gitignore @@ -1,9 +1,15 @@ -# Python build artifacts +# Keep root directory clean - all application files should be in frontend/ +*.db +*.log +uploads/ +data/ +optimized_prompts/ +logs/ +.env __pycache__/ *.pyc -*.py[cod] -*$py.class -*.so +*.pyo +*.pyd .Python build/ develop-eggs/ @@ -22,16 +28,40 @@ wheels/ *.egg # Virtual environments -.venv/ venv/ +.venv/ env/ -ENV/ +.env/ -# IDE files -.idea/ +# IDE .vscode/ +.idea/ *.swp *.swo +*~ + +# OS +.DS_Store +Thumbs.db + +# Temporary files +*.tmp +*.temp +temp/ + +# Cleanup script +cleanup.sh +# Optimization config files +frontend/opt_conf/ + +# Amazon Q internal documentation and analysis +.amazonq/ + +# Kiro files +.kiro/ + +# GitHub workflow files (if not needed in public repo) +.github/ -# Misc -.DS_Store \ No newline at end of file +# Jupyter Notebook checkpoints (in any directory) +**/.ipynb_checkpoints/ diff --git a/.kiro/settings/mcp.json b/.kiro/settings/mcp.json new file mode 100644 index 0000000..53f188a --- /dev/null +++ b/.kiro/settings/mcp.json @@ -0,0 +1,4 @@ +{ + "mcpServers": { + } +} diff --git a/.kiro/specs/nova-prompt-optimizer-frontend/design.md b/.kiro/specs/nova-prompt-optimizer-frontend/design.md new file mode 100644 index 0000000..4011a43 --- /dev/null +++ b/.kiro/specs/nova-prompt-optimizer-frontend/design.md @@ -0,0 +1,1135 @@ +# Design Document + +## Overview + +The Nova Prompt Optimizer Frontend is a web-based application that provides an intuitive interface for the existing Nova Prompt Optimizer Python SDK. The system follows a clean separation between a FastAPI backend that wraps the existing adapters and a React frontend that provides the user interface. + +The architecture maintains the existing backend adapters unchanged while adding a web layer that enables dataset management, prompt editing, optimization execution, and human annotation workflows. The system supports the complete optimization pipeline from data upload through result analysis and quality assurance. + +## Architecture + +### Project Structure + +The frontend will be organized under a `/ui` directory in the project root, maintaining clean separation from the existing Python SDK: + +``` +nova-prompt-optimizer/ +โ”œโ”€โ”€ src/ # Existing Python SDK (unchanged) +โ”œโ”€โ”€ samples/ # Existing samples (unchanged) +โ”œโ”€โ”€ ui/ # New frontend application +โ”‚ โ”œโ”€โ”€ backend/ # FastAPI backend +โ”‚ โ”‚ โ”œโ”€โ”€ app/ +โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ __init__.py +โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ main.py # FastAPI application entry point +โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ config.py # Configuration settings +โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ dependencies.py # Dependency injection +โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ models/ # Pydantic models +โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ __init__.py +โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ dataset.py +โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ prompt.py +โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ optimization.py +โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ annotation.py +โ”‚ โ”‚ โ”‚ โ”‚ โ””โ”€โ”€ common.py +โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ routers/ # API route handlers +โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ __init__.py +โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ datasets.py +โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ prompts.py +โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ optimization.py +โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ evaluation.py +โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ annotations.py +โ”‚ โ”‚ โ”‚ โ”‚ โ””โ”€โ”€ websocket.py +โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ services/ # Business logic layer +โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ __init__.py +โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ dataset_service.py +โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ prompt_service.py +โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ optimization_service.py +โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ evaluation_service.py +โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ annotation_service.py +โ”‚ โ”‚ โ”‚ โ”‚ โ””โ”€โ”€ rubric_service.py +โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ adapters/ # Integration with existing SDK +โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ __init__.py +โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ dataset_adapter.py +โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ prompt_adapter.py +โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ optimization_adapter.py +โ”‚ โ”‚ โ”‚ โ”‚ โ””โ”€โ”€ evaluation_adapter.py +โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ core/ # Core utilities +โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ __init__.py +โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ exceptions.py +โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ logging.py +โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ security.py +โ”‚ โ”‚ โ”‚ โ”‚ โ””โ”€โ”€ tasks.py +โ”‚ โ”‚ โ”‚ โ””โ”€โ”€ db/ # Database layer +โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ __init__.py +โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ database.py +โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ models.py +โ”‚ โ”‚ โ”‚ โ””โ”€โ”€ migrations/ +โ”‚ โ”‚ โ”œโ”€โ”€ requirements.txt # Backend dependencies +โ”‚ โ”‚ โ”œโ”€โ”€ Dockerfile # Backend container +โ”‚ โ”‚ โ””โ”€โ”€ pytest.ini # Test configuration +โ”‚ โ”œโ”€โ”€ frontend/ # React frontend +โ”‚ โ”‚ โ”œโ”€โ”€ public/ +โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ index.html +โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ favicon.ico +โ”‚ โ”‚ โ”‚ โ””โ”€โ”€ manifest.json +โ”‚ โ”‚ โ”œโ”€โ”€ src/ +โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ components/ # React components +โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ common/ # Shared components +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ Layout/ +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ AppLayout.tsx +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ Navigation.tsx +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ””โ”€โ”€ index.ts +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ ui/ # Shadcn/UI components +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ button.tsx +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ input.tsx +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ dialog.tsx +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ table.tsx +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ card.tsx +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ badge.tsx +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ progress.tsx +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ tabs.tsx +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ form.tsx +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ select.tsx +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ textarea.tsx +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ toast.tsx +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ alert.tsx +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ dropdown-menu.tsx +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ sheet.tsx +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ separator.tsx +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ””โ”€โ”€ index.ts +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ Loading/ +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ LoadingSpinner.tsx +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ ProgressBar.tsx +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ””โ”€โ”€ index.ts +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ””โ”€โ”€ ErrorBoundary/ +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ ErrorBoundary.tsx +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ ErrorDisplay.tsx +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ””โ”€โ”€ index.ts +โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ dataset/ # Dataset management +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ DatasetUpload/ +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ DatasetUpload.tsx +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ FileDropzone.tsx +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ ColumnMapper.tsx +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ””โ”€โ”€ index.ts +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ DatasetList/ +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ DatasetList.tsx +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ DatasetCard.tsx +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ””โ”€โ”€ index.ts +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ DatasetPreview/ +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ DatasetPreview.tsx +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ DataTable.tsx +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ””โ”€โ”€ index.ts +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ””โ”€โ”€ index.ts +โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ prompt/ # Prompt management +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ PromptEditor/ +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ PromptEditor.tsx +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ CodeEditor.tsx +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ VariableDetector.tsx +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ””โ”€โ”€ index.ts +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ PromptPreview/ +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ PromptPreview.tsx +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ TemplateRenderer.tsx +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ””โ”€โ”€ index.ts +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ PromptLibrary/ +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ PromptLibrary.tsx +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ PromptCard.tsx +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ””โ”€โ”€ index.ts +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ””โ”€โ”€ index.ts +โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ optimization/ # Optimization workflow +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ OptimizationConfig/ +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ OptimizationConfig.tsx +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ OptimizerSelector.tsx +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ ModelSelector.tsx +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ””โ”€โ”€ index.ts +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ OptimizationProgress/ +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ OptimizationProgress.tsx +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ ProgressTracker.tsx +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ LogViewer.tsx +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ””โ”€โ”€ index.ts +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ OptimizationResults/ +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ OptimizationResults.tsx +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ ResultsComparison.tsx +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ MetricsVisualization.tsx +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ””โ”€โ”€ index.ts +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ””โ”€โ”€ index.ts +โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ annotation/ # Human annotation system +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ RubricGenerator/ +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ RubricGenerator.tsx +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ DimensionEditor.tsx +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ””โ”€โ”€ index.ts +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ AnnotationInterface/ +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ AnnotationInterface.tsx +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ AnnotationForm.tsx +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ ResultViewer.tsx +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ””โ”€โ”€ index.ts +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ AnnotationDashboard/ +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ AnnotationDashboard.tsx +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ AgreementMetrics.tsx +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ ConflictResolution.tsx +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ””โ”€โ”€ index.ts +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ””โ”€โ”€ index.ts +โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ evaluation/ # Evaluation and metrics +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ MetricBuilder/ +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ MetricBuilder.tsx +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ CodeEditor.tsx +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ MetricTester.tsx +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ””โ”€โ”€ index.ts +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ EvaluationResults/ +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ EvaluationResults.tsx +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ ScoreBreakdown.tsx +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ””โ”€โ”€ index.ts +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ””โ”€โ”€ index.ts +โ”‚ โ”‚ โ”‚ โ”‚ โ””โ”€โ”€ index.ts +โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ pages/ # Page components +โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ Dashboard/ +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ Dashboard.tsx +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ””โ”€โ”€ index.ts +โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ DatasetManagement/ +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ DatasetManagement.tsx +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ””โ”€โ”€ index.ts +โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ PromptWorkbench/ +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ PromptWorkbench.tsx +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ””โ”€โ”€ index.ts +โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ OptimizationWorkflow/ +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ OptimizationWorkflow.tsx +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ””โ”€โ”€ index.ts +โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ AnnotationWorkspace/ +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ AnnotationWorkspace.tsx +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ””โ”€โ”€ index.ts +โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ ResultsAnalysis/ +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ ResultsAnalysis.tsx +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ””โ”€โ”€ index.ts +โ”‚ โ”‚ โ”‚ โ”‚ โ””โ”€โ”€ index.ts +โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ hooks/ # Custom React hooks +โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ useDataset.ts +โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ usePrompt.ts +โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ useOptimization.ts +โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ useAnnotation.ts +โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ useWebSocket.ts +โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ useLocalStorage.ts +โ”‚ โ”‚ โ”‚ โ”‚ โ””โ”€โ”€ index.ts +โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ services/ # API client services +โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ api/ +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ client.ts # Base API client +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ datasets.ts +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ prompts.ts +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ optimization.ts +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ annotations.ts +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ””โ”€โ”€ index.ts +โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ websocket/ +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ client.ts +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ handlers.ts +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ””โ”€โ”€ index.ts +โ”‚ โ”‚ โ”‚ โ”‚ โ””โ”€โ”€ index.ts +โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ store/ # State management +โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ context/ +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ AppContext.tsx +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ DatasetContext.tsx +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ OptimizationContext.tsx +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ””โ”€โ”€ index.ts +โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ reducers/ +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ datasetReducer.ts +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ optimizationReducer.ts +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ””โ”€โ”€ index.ts +โ”‚ โ”‚ โ”‚ โ”‚ โ””โ”€โ”€ index.ts +โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ types/ # TypeScript type definitions +โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ api.ts +โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ dataset.ts +โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ prompt.ts +โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ optimization.ts +โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ annotation.ts +โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ common.ts +โ”‚ โ”‚ โ”‚ โ”‚ โ””โ”€โ”€ index.ts +โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ utils/ # Utility functions +โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ formatting.ts +โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ validation.ts +โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ file-handling.ts +โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ date-time.ts +โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ constants.ts +โ”‚ โ”‚ โ”‚ โ”‚ โ””โ”€โ”€ index.ts +โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ styles/ # Styling +โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ globals.css # Global styles + Shadcn/UI base +โ”‚ โ”‚ โ”‚ โ”‚ โ””โ”€โ”€ components.css # Custom component styles +โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ lib/ # Utility libraries +โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ utils.ts # Shadcn/UI utility functions (cn, etc.) +โ”‚ โ”‚ โ”‚ โ”‚ โ””โ”€โ”€ validations.ts # Form validation schemas +โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ assets/ # Static assets +โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ images/ +โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ icons/ +โ”‚ โ”‚ โ”‚ โ”‚ โ””โ”€โ”€ fonts/ +โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ App.tsx # Main App component +โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ App.css +โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ index.tsx # React entry point +โ”‚ โ”‚ โ”‚ โ””โ”€โ”€ index.css +โ”‚ โ”‚ โ”œโ”€โ”€ package.json # Frontend dependencies +โ”‚ โ”‚ โ”œโ”€โ”€ tsconfig.json # TypeScript configuration +โ”‚ โ”‚ โ”œโ”€โ”€ tailwind.config.js # Tailwind CSS configuration +โ”‚ โ”‚ โ”œโ”€โ”€ components.json # Shadcn/UI configuration +โ”‚ โ”‚ โ”œโ”€โ”€ vite.config.ts # Vite build configuration +โ”‚ โ”‚ โ””โ”€โ”€ Dockerfile # Frontend container +โ”‚ โ”œโ”€โ”€ shared/ # Shared utilities and types +โ”‚ โ”‚ โ”œโ”€โ”€ types/ +โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ api-schema.ts # Shared API types +โ”‚ โ”‚ โ”‚ โ””โ”€โ”€ index.ts +โ”‚ โ”‚ โ”œโ”€โ”€ utils/ +โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ validation.py # Shared validation logic +โ”‚ โ”‚ โ”‚ โ””โ”€โ”€ constants.py +โ”‚ โ”‚ โ””โ”€โ”€ scripts/ +โ”‚ โ”‚ โ”œโ”€โ”€ generate-types.py # Generate TS types from Pydantic +โ”‚ โ”‚ โ””โ”€โ”€ setup-dev.sh +โ”‚ โ”œโ”€โ”€ docker-compose.yml # Development environment +โ”‚ โ”œโ”€โ”€ docker-compose.prod.yml # Production environment +โ”‚ โ”œโ”€โ”€ .env.example # Environment variables template +โ”‚ โ”œโ”€โ”€ .gitignore # Git ignore rules for UI +โ”‚ โ””โ”€โ”€ README.md # UI-specific documentation +โ”œโ”€โ”€ requirements.txt # Existing Python requirements +โ”œโ”€โ”€ pyproject.toml # Existing project config +โ””โ”€โ”€ README.md # Main project README +``` + +### High-Level Architecture + +``` +โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” +โ”‚ React SPA โ”‚ โ”‚ FastAPI โ”‚ โ”‚ Nova Prompt โ”‚ +โ”‚ Frontend โ”‚โ—„โ”€โ”€โ–บโ”‚ Backend โ”‚โ—„โ”€โ”€โ–บโ”‚ Optimizer SDK โ”‚ +โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ (Unchanged) โ”‚ +โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ + โ”‚ โ”‚ โ”‚ + โ”‚ โ”‚ โ”‚ + โ–ผ โ–ผ โ–ผ +โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” +โ”‚ Browser โ”‚ โ”‚ Task Queue โ”‚ โ”‚ AWS Bedrock โ”‚ +โ”‚ Storage โ”‚ โ”‚ (Background โ”‚ โ”‚ Integration โ”‚ +โ”‚ โ”‚ โ”‚ Jobs) โ”‚ โ”‚ โ”‚ +โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ +``` + +### Component Architecture + +The system is organized into distinct layers: + +**Frontend Layer (React + TypeScript + Shadcn/UI)** + +- Component-based UI using Shadcn/UI design system with Tailwind CSS +- Consistent design language with accessible, customizable components +- State management using React Context and custom hooks +- Real-time updates through WebSocket connections +- Responsive design supporting desktop and tablet interfaces +- Dark/light theme support built into Shadcn/UI components + +**API Layer (FastAPI)** + +- RESTful endpoints wrapping existing Python adapters +- Background task management for long-running operations +- WebSocket support for real-time progress updates +- Authentication and session management + +**Integration Layer** + +- Direct imports of existing Nova Prompt Optimizer adapters +- No modifications to existing backend code +- Proper error handling and logging +- Resource management and cleanup + +**Data Layer** + +- File storage for uploaded datasets and generated artifacts +- Database for metadata, user sessions, and annotation data +- Caching layer for optimization results and intermediate data + +## Components and Interfaces + +### Shadcn/UI Integration + +The frontend will use Shadcn/UI as the primary component library, providing: + +**Design System Benefits:** + +- Consistent, accessible components built on Radix UI primitives +- Tailwind CSS integration with CSS variables for theming +- Copy-paste component architecture for easy customization +- Built-in dark/light mode support +- TypeScript-first with excellent type safety + +**Key Shadcn/UI Components Used:** + +- `Card`, `CardHeader`, `CardContent`, `CardFooter` for layout containers +- `Button`, `Input`, `Textarea`, `Select` for form controls +- `Table`, `TableHeader`, `TableBody`, `TableRow`, `TableCell` for data display +- `Dialog`, `Sheet`, `DropdownMenu` for overlays and navigation +- `Progress`, `Badge`, `Alert` for status and feedback +- `Tabs`, `Separator`, `Form` for organization and structure +- `Toast` for notifications and user feedback + +**Setup Configuration:** + +```json +// components.json +{ + "$schema": "https://ui.shadcn.com/schema.json", + "style": "default", + "rsc": false, + "tsx": true, + "tailwind": { + "config": "tailwind.config.js", + "css": "src/styles/globals.css", + "baseColor": "slate", + "cssVariables": true + }, + "aliases": { + "components": "@/components", + "utils": "@/lib/utils" + } +} +``` + +**Custom Theme Configuration:** + +```css +/* globals.css */ +@tailwind base; +@tailwind components; +@tailwind utilities; + +@layer base { + :root { + --background: 0 0% 100%; + --foreground: 222.2 84% 4.9%; + --primary: 221.2 83.2% 53.3%; + --primary-foreground: 210 40% 98%; + /* Additional custom variables for Nova branding */ + --nova-blue: 214 100% 59%; + --nova-purple: 262 83% 58%; + } + + .dark { + --background: 222.2 84% 4.9%; + --foreground: 210 40% 98%; + /* Dark mode variables */ + } +} +``` + +### Frontend Components + +**Core Layout Components** + +```typescript +interface AppLayoutProps { + children: React.ReactNode; + currentStep: WorkflowStep; + onStepChange: (step: WorkflowStep) => void; +} + +interface NavigationProps { + steps: WorkflowStep[]; + currentStep: WorkflowStep; + completedSteps: WorkflowStep[]; +} +``` + +**Dataset Management Components (Shadcn/UI Implementation)** + +```typescript +interface DatasetUploadProps { + onUploadComplete: (dataset: Dataset) => void; + acceptedFormats: string[]; + maxFileSize: number; +} + +// Example implementation with Shadcn/UI +const DatasetUpload: React.FC = ({ onUploadComplete }) => { + return ( + + + Upload Dataset + + Upload a CSV or JSON file to get started with prompt optimization + + + +
+ +

+ Drag & drop your file here, or click to browse +

+
+
+
+ + +
+
+ + +
+
+
+ + + +
+ ); +}; + +interface DatasetPreviewProps { + dataset: Dataset; + maxRows: number; + onColumnMapping: (mapping: ColumnMapping) => void; +} + +interface DatasetListProps { + datasets: Dataset[]; + selectedDataset?: Dataset; + onSelect: (dataset: Dataset) => void; + onDelete: (datasetId: string) => void; +} +``` + +**Prompt Editor Components (Shadcn/UI Implementation)** + +```typescript +interface PromptEditorProps { + prompt?: Prompt; + variables: string[]; + onSave: (prompt: Prompt) => void; + onPreview: (prompt: Prompt, sampleData: any) => void; +} + +// Example implementation with Shadcn/UI +const PromptEditor: React.FC = ({ prompt, onSave }) => { + return ( +
+ + + Prompt Configuration + + +
+ + +
+ + + + System Prompt + User Prompt + + + + +