Merge pull request #82 from halcyon-past/feature/dev/sandbox-preview-mode

feat: Build a sandbox preview mode for uploads

Author: halcyon-past

ARCHITECTURE.md

@@ -136,10 +136,11 @@ sequenceDiagram
 - **Map:** Each chunk is mapped over Gemini concurrently via Pub/Sub. If a chunk fails extraction, a LangGraph state machine catches the error and loops back for up to 3 self-correction retries.
 - **Reduce:** A Firestore transactional counter tracks chunk completions and eventually merges them into a unified `.xlsx` file.
 
-### 3. Strict Schema Enforcement & Auto-Clean
-**Problem:** LLMs are prone to hallucinating formats or omitting columns.
+### 3. Strict Schema Enforcement, Auto-Clean, & Preview Mode
+**Problem:** LLMs are prone to hallucinating formats or omitting columns. Running a full pipeline on an invalid schema is costly.
 **Solution:** 
-- If a target schema is provided, Gemini is forced to map the data directly to a JSON Schema object (`response_schema`).
+- **Sandbox Preview Mode:** The system can be triggered in a preview mode where the split phase instantly cuts the input to exactly 10 rows. This allows rapid validation of the extraction quality without wasting excessive compute tokens on dead runs.
+- **Schema Mapping:** If a target schema is provided, Gemini is forced to map the data directly to a JSON Schema object (`response_schema`).
 - **Auto-Clean Mode:** If no target schema is provided, Structurify dynamically infers the schema, cleans up the mess (capitalization, whitespaces, date formats), and returns the entire spreadsheet as a valid JSON array.
 
 ### 4. Asynchronous Email Notifications

CHANGELOG.md

@@ -5,6 +5,15 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [2.4.0] - 2026-08-18
+### Added
+- **Sandbox Preview Mode**: Added a preview-only mode that processes just the first 10 rows of a file so users can validate schema fit and transformation quality before committing to a full long-running job. This significantly reduces wasted tokens and improves trust for first-time users.
+  - The UI now features a "Run Preview" action.
+  - Preview jobs are flagged in the pipeline UI, skipping heavy processing.
+  - The completion screen for preview mode prompts users to launch a full job if the results are satisfactory.
+- **Custom Toast Notifications**: Replaced all native browser `alert()` and `confirm()` dialogues across the application with non-blocking, stylish `react-hot-toast` notifications.
+- **Admin Dashboard UI Update**: Moved the Admin button out of the main header and into the profile dropdown menu to streamline navigation and keep administrative functions discreetly accessible to verified admins and owners.
+
 ## [2.3.0] - 2026-08-17
 ### Added
 - **Enterprise SSO Support**: Introduced support for Enterprise SSO via SAML and OIDC through Firebase Identity Platform.

README.md

@@ -11,12 +11,14 @@ Built on a completely decoupled **Serverless Fan-Out Architecture** on Google Cl
 ## <img src="./docs/icons/architecture.svg" width="28" align="absbottom" alt="architecture" /> Core Features
 
 - **Strict Schema Enforcement**: Define exactly the JSON/Excel schema you need, and Structurify will enforce strict type-casting and structure.
+- **Sandbox Preview Mode**: Process just the first 10 rows of a dataset to validate schema fit and AI transformation quality before committing to a full long-running job.
 - **Auto-Clean Mode**: Don't know the schema? Structurify will automatically infer the schema from the file headers and repair capitalization, trim whitespace, and standardize date formats across the board.
 - **Email Notifications**: Upload a massive dataset (over 1MB), and Structurify will immediately email you a tracking link to watch the live progress, followed by a final success email with your secure download URL.
 - **Enterprise SSO & Multi-Tenant Authentication**: Provides robust authentication via Firebase Identity Platform, supporting Google OAuth, SAML, and OIDC enterprise SSO. Features automatic account linking for identity conflict resolution and maps users to isolated multi-tenant workspaces based on their provider `tenantId`.
 - **Massive Scalability**: The backend acts as a lightweight router while heavy data processing is handled by scalable workers via Cloud Pub/Sub, preventing Gateway Timeouts on long jobs.
 - **Dynamic Configuration & Prompt Management**: An integrated Admin UI backed by a real-time Firestore synchronization engine allows operators to hot-swap Gemini LLM models, tune chunk sizes, and edit system AI prompts entirely on the fly without ever redeploying code.
 - **Graceful Job Cancellation**: Safely halt massive in-flight jobs via a UI cancel button. In-memory TTL caching on workers ensures instant cancellation without generating "ghost jobs" or burning Firestore read quotas.
+- **Custom Toast Notifications**: Uses non-blocking `react-hot-toast` popups instead of native browser alerts to provide users with a clean, modern experience when editing settings or executing administrative actions.
 - **Enterprise Observability & Billing**: Logs rich telemetry into Firestore (`job_audits`), tracking LLM Token Usage via atomic transactions, File Sizes, IP Addresses, and exact Job Runtimes to power strict rate limits and future billing models.
 
 ---

backend/src/api/routers/jobs.py

@@ -30,7 +30,7 @@ async def create_job(
     try:
         firestore_svc.create_job(
             job_id, request.file_path, request.file_name, request.target_schema, now, 
-            request.email, request.role, request.plan, user_id, ip_address
+            request.email, request.role, request.plan, user_id, ip_address, request.is_preview
         )
     except Exception as e:
         raise HTTPException(status_code=500, detail=f"Failed to create job document: {str(e)}")
@@ -39,7 +39,7 @@ async def create_job(
     try:
         pubsub_svc.publish_job(
             job_id, request.file_path, request.target_schema, 
-            request.email, request.role, request.plan, user_id, ip_address
+            request.email, request.role, request.plan, user_id, ip_address, request.is_preview
         )
     except Exception as e:
         # Mark as failed if publish fails
@@ -54,7 +54,8 @@ async def create_job(
     return {
         "job_id": job_id,
         "status": "queued",
-        "message": "Job accepted and queued for processing."
+        "message": "Job accepted and queued for processing.",
+        "is_preview": request.is_preview
     }
 
 @router.get("/{job_id}", response_model=JobStatusResponse)
@@ -77,7 +78,8 @@ async def get_job_status(
         processed_rows=data.get("processed_rows"),
         error_message=data.get("error_message"),
         total_chunks=data.get("total_chunks"),
-        completed_chunks=data.get("completed_chunks")
+        completed_chunks=data.get("completed_chunks"),
+        is_preview=data.get("is_preview", False)
     )
 
 @router.post("/{job_id}/cancel")

backend/src/models/schemas.py

@@ -17,11 +17,13 @@ class JobRequest(BaseModel):
     role: Optional[str] = "guest"
     plan: Optional[str] = "free"
     user_id: Optional[str] = None
+    is_preview: bool = False
 
 class JobResponse(BaseModel):
     job_id: str
     status: str
     message: str
+    is_preview: bool = False
 
 class JobStatusResponse(BaseModel):
     job_id: str
@@ -32,3 +34,4 @@ class JobStatusResponse(BaseModel):
     error_message: Optional[str] = None
     total_chunks: Optional[int] = None
     completed_chunks: Optional[int] = None
+    is_preview: Optional[bool] = False

backend/src/services/firestore.py

@@ -8,7 +8,7 @@ class FirestoreService:
     def __init__(self, client: firestore.Client = None):
         self.db = client or get_firestore_client()
 
-    def create_job(self, job_id: str, file_path: str, file_name: str, target_schema: dict, created_at: str, email: str = None, role: str = "guest", plan: str = "free", user_id: str = None, ip_address: str = None):
+    def create_job(self, job_id: str, file_path: str, file_name: str, target_schema: dict, created_at: str, email: str = None, role: str = "guest", plan: str = "free", user_id: str = None, ip_address: str = None, is_preview: bool = False):
         job_ref = self.db.collection("jobs").document(job_id)
         job_data = {
             "job_id": job_id,
@@ -21,7 +21,8 @@ def create_job(self, job_id: str, file_path: str, file_name: str, target_schema:
             "role": role,
             "plan": plan,
             "user_id": user_id,
-            "ip_address": ip_address
+            "ip_address": ip_address,
+            "is_preview": is_preview
         }
         if email:
             job_data["email"] = email

backend/src/services/pubsub.py

@@ -10,7 +10,7 @@ def __init__(self, publisher: pubsub_v1.PublisherClient = None):
         self.publisher = publisher or get_pubsub_publisher()
         self.topic_path = self.publisher.topic_path(settings.GOOGLE_CLOUD_PROJECT, settings.PUBSUB_TOPIC_ID)
 
-    def publish_job(self, job_id: str, file_path: str, target_schema: dict, email: str = None, role: str = "guest", plan: str = "free", user_id: str = None, ip_address: str = None):
+    def publish_job(self, job_id: str, file_path: str, target_schema: dict, email: str = None, role: str = "guest", plan: str = "free", user_id: str = None, ip_address: str = None, is_preview: bool = False):
         message_data = json.dumps({
             "job_id": job_id,
             "file_path": file_path,
@@ -19,7 +19,8 @@ def publish_job(self, job_id: str, file_path: str, target_schema: dict, email: s
             "role": role,
             "plan": plan,
             "user_id": user_id,
-            "ip_address": ip_address
+            "ip_address": ip_address,
+            "is_preview": is_preview
         }).encode("utf-8")
 
         future = self.publisher.publish(self.topic_path, data=message_data)

backend/test_firestore.py

@@ -0,0 +1,16 @@
+import sys
+import json
+from dotenv import load_dotenv
+load_dotenv('.env')
+
+from google.cloud import firestore
+
+try:
+    db = firestore.Client(project="structurify-504821")
+    doc = db.collection('settings').document('system').get()
+    if doc.exists:
+        print(json.dumps(doc.to_dict(), indent=2))
+    else:
+        print("Document does not exist.")
+except Exception as e:
+    print(f"Error: {e}")

backend/tests/conftest.py

@@ -11,14 +11,14 @@ def generate_upload_url(self, file_path: str, content_type: str, expiration_minu
         return f"https://mock-storage.com/{file_path}"
 
 class MockPubSubService:
-    def publish_job(self, job_id: str, file_path: str, target_schema: dict, email: str = None, role: str = "guest", plan: str = "free", user_id: str = None, ip_address: str = None):
+    def publish_job(self, job_id: str, file_path: str, target_schema: dict, email: str = None, role: str = "guest", plan: str = "free", user_id: str = None, ip_address: str = None, is_preview: bool = False):
         pass # mock success
 
 class MockFirestoreService:
     def __init__(self):
         self.jobs = {}
 
-    def create_job(self, job_id: str, file_path: str, file_name: str, target_schema: dict, created_at: str, email: str = None, role: str = "guest", plan: str = "free", user_id: str = None, ip_address: str = None):
+    def create_job(self, job_id: str, file_path: str, file_name: str, target_schema: dict, created_at: str, email: str = None, role: str = "guest", plan: str = "free", user_id: str = None, ip_address: str = None, is_preview: bool = False):
         self.jobs[job_id] = {
             "job_id": job_id,
             "status": "queued",
@@ -27,7 +27,7 @@ def create_job(self, job_id: str, file_path: str, file_name: str, target_schema:
             "role": role,
             "plan": plan,
             "user_id": user_id,
-            "ip_address": ip_address
+            "ip_address": ip_address, "is_preview": is_preview
         }
 
     def update_job_status(self, job_id: str, status: str, error_message: str = None, updated_at: str = None):

backend/tests/test_jobs.py

@@ -9,7 +9,7 @@ def test_create_and_get_job(client, mock_firestore):
         "plan": "pro"
     }
     response = client.post("/api/v1/jobs/", json=payload)
-    assert response.status_code == 202
+    assert response.status_code == 202, response.text
     data = response.json()
     assert "job_id" in data
     assert data["status"] == "queued"
@@ -22,6 +22,35 @@ def test_create_and_get_job(client, mock_firestore):
     job_data = response.json()
     assert job_data["job_id"] == job_id
     assert job_data["status"] == "queued"
+    assert job_data["is_preview"] is False
+
+def test_create_and_get_job_preview(client, mock_firestore):
+    # Test Create Preview Job
+    payload = {
+        "file_path": "uploads/test.csv",
+        "file_name": "test.csv",
+        "target_schema": {"name": "String"},
+        "email": "test@example.com",
+        "role": "admin",
+        "plan": "pro",
+        "is_preview": True
+    }
+    response = client.post("/api/v1/jobs/", json=payload)
+    assert response.status_code == 202, response.text
+    data = response.json()
+    assert "job_id" in data
+    assert data["status"] == "queued"
+    assert data["is_preview"] is True
+    
+    job_id = data["job_id"]
+    
+    # Test Get Job
+    response = client.get(f"/api/v1/jobs/{job_id}")
+    assert response.status_code == 200
+    job_data = response.json()
+    assert job_data["job_id"] == job_id
+    assert job_data["status"] == "queued"
+    assert job_data["is_preview"] is True
 
 def test_get_nonexistent_job(client):
     response = client.get("/api/v1/jobs/invalid-id")