Mission: Principal Data Scientist - Multilingual Semantic Theme Discovery & Retrieval Quality Study

[tourajshah]

Mission/open goal Description

Problem Statement

Zava's Product and Support organizations are operating on conflicting customer sentiment metrics derived from cherry-picked reviews. Unstructured user feedback is highly fragmented across public reviews and private support chats, spans three languages (English, Spanish, French), and lacks baseline ground-truth labels.

Operational Constraints & Systemic Noise

1. Corpus Limitations: The evaluation dataset is restricted to 83 total records, meaning raw vector search tools are mathematically forced to pull distant contextual neighbors once true signals are exhausted.
2. Multilingual Disruption: Standard keyword indexing or string matching fails across regional boundaries, making semantic embedding similarity the only viable vector for cross-lingual discovery.
3. Domain Cross-Pollination: Merging conversational support transcripts and structured product reviews into a single document vector space introduces high levels of operational boilerplate noise, signature blocks, and nomenclature traps.

Executive Evaluation Framework

To defensibly audit the system's precomputed 1536-dimensional embedding spaces, this study establishes a dual-tier evaluation matrix over cross-lingual vector neighbor footprints retrieved via FindSimilarDocsByDocId ($K=5$). Rather than relying on unverified black-box clustering algorithms, we implement manual human-labeling to calculate and contrast two distinct metrics:

• Strict Root-Cause Precision ($P_{\text{strict}}@5$): Measures the system's capacity to isolate identical engineering failure modes (e.g., dead temperature sensor hardware).
• Broad Domain Precision ($P_{\text{broad}}@5$): Measures the system's capacity to route generalized topics (e.g., any technical setup issue), exposing structural vulnerabilities and informing a production-ready, scalable hybrid reranking pipeline.

Harness and model

GitHub Copilot Chat in Agent Mode running on the MAI-Code-1-Flash model.

Turn-by-turn journey

Turn 1

• Prompt: "I am taking on the Principal Data Scientist mission for Zava. Before we do any analysis, let's discover our environment. Please use your SQL MCP tools to list the available tools and describe the database entities so we can see what tables we have access to."
• Agent response, abridged: The agent acknowledged the environment discovery goal and executed describe_entities on the server. The database returned the core schemas (dbo.Products, dbo.Customers, dbo.Employees, dbo.SupportTickets, dbo.SupportChats) along with the unified dbo.Docs table containing precomputed 1536-dimensional embeddings and the custom stored procedure FindSimilarDocsByDocId. This confirmed that both reviews and support chats occupy a single searchable document space.

Turn 2

• Prompt: "Let's look at the customer text data to identify our themes. Please use the read records tool on the Doc entity to pull the first 15 records. Show me the DocId, SourceType, Title, Body, and the TagsJson fields for each so we can analyze what customers are saying across different languages."
• Agent response, abridged: The agent initially targeted a singular Doc entity, which triggered an EntityNotFound error from the MCP server. It automatically initiated a recovery loop by parsing dab-config.json to find the correct system identifier (Docs), then successfully pulled the first 15 records. The output exposed a complex multilingual dataset, highlighting an English apparel quality issue (Docs 2, 7) and a critical Spanish 5-star translation anomaly containing explicit defect complaints (DocId 9).

Turn 3

• Prompt: "Now, let's perform our first vector similarity search. Please use the custom tool FindSimilarDocsByDocId with @docid = 2 and @TopN = 5. Display the matching DocIds, their titles, languages, and their cosine distance scores so we can audit what the system thinks is similar."
• Agent response, abridged: The agent executed the custom vector tool FindSimilarDocsByDocId using our English defect seed. It returned the top 5 closest mathematical neighbors: DocId 2 (0.0000), DocId 7 (0.2377), DocId 18 (0.3197), DocId 21 (0.3268), and DocId 3 (0.3392). The initial telemetry revealed a severe regional language bias: despite verified Spanish and French product defects existing in the broader dataset, the neighborhood was 100% isolated to English text.

Turn 4

• Prompt: "Let's inspect the text of our vector neighbors to calculate our true precision. Please use the read records tool on the Docs entity to pull the Body and TagsJson fields specifically for DocId 18 and DocId 21."
• Agent response, abridged: The agent ran a read_records call to inspect the complete content of the neighboring documents. The text revealed a major semantic trap in DocId 18: the model paired a highly satisfied 5-star review with our 1-star defect seed simply because they shared long, identical product noun descriptions ("Elite Short Sleeve Men's Top in teal/orange, size M"). This over-indexing of nouns results in a strict root-cause retrieval precision score of only 40.0%.

Turn 5

• Prompt: "Let's find our next seed document for Probe 2. Please query the Docs entity to find 5 sample records where the SourceType is 'SupportChat' and the TagsJson contains either 'language:es' or 'language:fr'. Display their DocId, Title, Body, and TagsJson clearly so we can pick a non-English support conversation as our next seed."
• Agent response, abridged: The agent attempted a server-side OData string filter (contains) on the JSON metadata column, which failed with a BadRequest error due to service syntax limits. The agent adapted by simplifying the server call to just SourceType eq 'SupportChat', pulling a broader pool of 20 rows, and filtering the language tags locally in its memory buffer. From this pool, it isolated DocId 62 (a Spanish support transcript detailing an app crash and sensor hardware failure) to serve as Probe 2.

Turn 6

• Prompt: "Let's execute Probe 2 using our non-English support chat seed. Please run the custom tool FindSimilarDocsByDocId with @docid = 62 and @TopN = 5. Display the matching DocIds, titles, languages, source types, and cosine distances so we can audit if the system can bridge languages and cross from support chats to reviews."
• Agent response, abridged: The agent executed the vector search stored procedure for the Spanish support seed (DocId = 62). The model returned an exclusively Spanish, 100% SupportChat neighborhood consisting of Docs 64 (0.1103), 79 (0.1988), 78 (0.2207), and 55 (0.2240). This confirmed a double-isolation flaw: the vector space failed to bridge the language gap to English, and it remained completely locked inside the conversational chat domain, ignoring relevant public product reviews.

Turn 7

• Prompt: "To complete our audit for Probe 2, we need to read the actual text. Please use the read records tool on the Docs entity to pull the Body and TagsJson fields specifically for DocId 64, DocId 79, DocId 78, and DocId 55. Let's see what these customers are actually experiencing."
• Agent response, abridged: The agent completed the data pull using a multi-ID document filter. Human audit of the text segments exposed a major corporate boilerplate trap in DocId 78: the system misrouted a routine clothing size exchange directly into the critical technical hardware cluster because both files shared the identical automated signature string: "producto textil inteligente de Zava". While broad topic routing precision sat at 80.0%, strict engineering root-cause precision remained highly vulnerable at 40.0%.

Completion

• Yes, the agent completed the mission or goal.
• No, the agent did not complete the mission or goal.

Bonus work

1. Rejection of Algorithmic Shortcuts (The Data Integrity Pivot)

During the study, the agent attempted to write automated post-retrieval filters checking for localized string lists (e.g., if 'defect' in text). As a Principal Data Scientist, I intercepted this behavior and forced the agent to remove it. Because keyword grouping fails across multiple languages and is completely vulnerable to sarcasm or word-reuse drift (like the 5-star defect misclick in DocId 9), applying a hardcoded keyword patch defeats the entire purpose of semantic vector spaces.

2. Implementation of a Headless Processing Artifact (zava_scientist_audit.py)

Due to webview rendering constraints within the container environment, I bypassed the broken visual Jupyter extensions entirely and executed a dedicated Python processing script directly against the terminal engine. The script utilizes the container's pre-installed core data science stack (pandas and matplotlib) to programmatically handle our human-labeled data frames and output a high-resolution precision variance chart directly to disk.

import pandas as pd
import matplotlib.pyplot as plt

print("====================================================")
print("  ZAVA SEMANTIC RETRIEVAL QUALITY STUDY (AUDIT)     ")
print("====================================================\n")

# 1. GROUND TRUTH DATABASE FOOTPRINT (Defensible Human Labels)
audit_data = [
    # PROBE 1: English Review Seed (Theme: Structural Apparel Defects / Weak Stitching)
    {"Seed": 2, "Match": 2, "Lang": "en", "Strict": True,  "Broad": True,  "Error_Type": "None (Seed)"},
    {"Seed": 2, "Match": 7, "Lang": "en", "Strict": True,  "Broad": True,  "Error_Type": "None (True Positive)"},
    {"Seed": 2, "Match": 18, "Lang": "en", "Strict": False, "Broad": False, "Error_Type": "Product Noun Trap (Happy Buyer)"},
    {"Seed": 2, "Match": 21, "Lang": "en", "Strict": False, "Broad": True,  "Error_Type": "Sizing Context Drift"},
    {"Seed": 2, "Match": 3, "Lang": "en", "Strict": False, "Broad": True,  "Error_Type": "Sizing Context Drift"},

    # PROBE 2: Spanish Chat Seed (Theme: Smart Fabric Sensor Module / App Crashing Failures)
    {"Seed": 62, "Match": 62, "Lang": "es", "Strict": True,  "Broad": True,  "Error_Type": "None (Seed)"},
    {"Seed": 62, "Match": 64, "Lang": "es", "Strict": True,  "Broad": True,  "Error_Type": "None (True Positive)"},
    {"Seed": 62, "Match": 79, "Lang": "es", "Strict": False, "Broad": True,  "Error_Type": "Boilerplate Glitch (Soft Tech)"},
    {"Seed": 62, "Match": 55, "Lang": "es", "Strict": False, "Broad": True,  "Error_Type": "Boilerplate Glitch (Soft Tech)"},
    {"Seed": 62, "Match": 78, "Lang": "es", "Strict": False, "Broad": False, "Error_Type": "Corporate Signature Noun Trap"}
]

df = pd.DataFrame(audit_data)

# 2. CALCULATE METRICS RIGOROUSLY USING PANDAS
def compute_p_at_k(df, seed_id, column, k=5):
    subset = df[df['Seed'] == seed_id]
    return subset[column].sum() / k

p1_strict = compute_p_at_k(df, 2, 'Strict')
p1_broad  = compute_p_at_k(df, 2, 'Broad')
p2_strict = compute_p_at_k(df, 62, 'Strict')
p2_broad  = compute_p_at_k(df, 62, 'Broad')

print("----------------------------------------------------")
print("          EVALUATION METRICS LEDGER                 ")
print("----------------------------------------------------")
print(f"Probe 1 (EN Review) | Strict P@5: {p1_strict*100:.1f}% | Broad P@5: {p1_broad*100:.1f}%")
print(f"Probe 2 (ES Chat)   | Strict P@5: {p2_strict*100:.1f}% | Broad P@5: {p2_broad*100:.1f}%")
print("----------------------------------------------------")

# 3. QUANTIFY ARCHITECTURAL FLAWS FOR WRITTEN CRITIQUE
print("\n----------------------------------------------------")
print("          VECTOR SEARCH VULNERABILITY ANALYSIS      ")
print("----------------------------------------------------")
failures = df[df['Strict'] == False]
failure_counts = failures['Error_Type'].value_counts()
for flaw, occurrences in failure_counts.items():
    print(f"-> {flaw}: Found {occurrences} time(s) in sample footprint.")

# 4. EXECUTIVELY PLOT THE TRUE VARIANCE
fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(14, 6), sharey=True)

strict_counts = df.groupby(['Seed', 'Strict']).size().unstack(fill_value=0)
strict_counts.index = ['Probe 1 (EN Review)', 'Probe 2 (ES Chat)']
strict_counts.plot(kind='bar', stacked=True, color=['#e74c3c', '#2ecc71'], ax=ax1)
ax1.set_title('Strict Root-Cause Matching\n(True Quality/Hardware Signals Only)')
ax1.set_ylabel('Document Footprint Count (K=5)')
ax1.legend(['False Positives', 'True Positives'])

broad_counts = df.groupby(['Seed', 'Broad']).size().unstack(fill_value=0)
broad_counts.index = ['Probe 1 (EN Review)', 'Probe 2 (ES Chat)']
broad_counts.plot(kind='bar', stacked=True, color=['#e74c3c', '#3498db'], ax=ax2)
ax2.set_title('Broad Topic Domain Matching\n(Includes Soft Context/Sizing Drift)')
ax2.legend(['Noise/Blunders', 'Broad Domain Matches'])

plt.suptitle('Zava Vector Retrieval Quality Audit: Signal vs. Noise Variance', fontsize=14, fontweight='bold')
plt.tight_layout()

output_path = 'retrieval_audit_plot.png'
plt.savefig(output_path, dpi=300)
print(f"\n[SYSTEM] Rigorous comparison visualization saved to disk as: '{output_path}'")

Audited Signal vs. Noise Distribution Plot

3. Core Decisions & Metadata Traceability Ledger

• The Decision to Reject Single-Tier Relevance Evaluation: A shallow data demonstration would blend generic semantic neighbors together, presenting a deceptively high 80% broad domain precision score to management. By creating an explicit separation between operational support triage (broad topic matching) and deep manufacturing analysis (strict root-cause matching), we proved that the raw embedding vector space drops to a 40% signal retention accuracy rate under precise diagnostic contexts.
• The Tiny Corpus Limitation: With only 83 total records, the vector tool is mathematically forced to satisfy the requested $K=5$ parameter. When the volume of true root-cause anomalies is exhausted, the engine pulls distant data neighbors that share generic sentence syntax, leading to structural false positives.

4. System Model Card Note

• Operational Strengths: Excellent for high-speed, coarse topic grouping within a single language segment. Capable of mapping broad thematic concepts (e.g., linking a defect on a short sleeve shirt to a long sleeve shirt based purely on syntax structure).

• Operational Blind Spots: Highly vulnerable to language isolation (fails to bridge English, Spanish, and French documents natively within the top search tiers). Highly vulnerable to noun-weight drift (matching completely happy customers with angry complaints due to identical product style strings) and conversational boilerplate noise (routing standard sizing returns into highly technical engineering clusters due to legal signature fragments).

5. Where the Agent Struggled and Autonomously Pivoted

Rather than requiring manual human overrides for minor syntax stumbles, the agent demonstrated typical agentic loop recovery behaviors alongside clear semantic boundaries that required human validation:

1. Autonomous Recovery from Entity Name Instability: In Turn 2, the agent blindly targeted the singular entity name Doc. When the database layer rejected the call, the agent automatically initiated an internal reflection step, reading the repository's configuration files (dab-config.json) to re-map the correct system entity identifier (Docs) without human intervention.

2. Dynamic Adaptation to OData Constraints: In Turn 5, the agent hit a hard limitation regarding how the MCP server interprets complex string logical filters on JSON columns, resulting in a BadRequest code. The agent successfully pivoted from a server-side filtering strategy to a localized client-side ingestion workflow, reading rows sequentially to process the multilingual metadata filters internally.

3. The Logical Semantic Trap (Human Reining Required): Where the system truly required human-in-the-loop guidance was navigating semantic alignment. The agent initially drifted toward implementing shallow keyword filtering scripts (e.g., checking for specific hardcoded strings like 'defect' or 'fallo') to clean the Pandas data pool. I intercepted this approach on the grounds of data integrity, directing the pipeline to evaluate pure semantic vector distribution profiles instead of falling back on fragile keyword rules that fail across English, Spanish, and French context boundaries.

6. What to Test or Improve Next (Production Scale Architecture)

To scale this system beyond the 83-document footprint without relying on fragile keyword scripts, Zava must replace the raw, un-sanitized single-vector lookup pipeline with a robust Two-Stage Hybrid Semantic Ingestion and Reranking Architecture:

Stage 1: Ingestion-Tier Vector Space Alignment & Domain Sanitization
To structurally bridge the cross-language and cross-source isolation gaps before data ever reaches the stored procedure, the ingestion pipeline must implement two critical modifications:

1. Transition to a Contrastive Multilingual Bi-Encoder: Replace the baseline embedding model with a true translation-invariant multilingual dense vector model (e.g., multilingual-e5-large or Cohere embed-multilingual-v3). These models are trained explicitly using contrastive loss functions on cross-lingual parallel sentence pairs, mathematically forcing English, Spanish, and French concepts into identical geometric coordinates.

2. Boilerplate Strip-Pre-processing & Content Sanitization: Support transcripts must pass through an automated regex or layout-parsing cleaner prior to vector generation. System logs, employee greeting blocks ("producto textil inteligente de Zava"), and static product description noun metadata must be stripped, leaving only raw user-generated problem descriptions. This collapses the structural style vector, preventing the embedding model from segregating data by source type (SupportChat vs. Review).

Stage 2: Dual-Input Neural Cross-Encoder Reranking
At runtime, the top candidate results retrieved by the aligned multi-lingual vector space are passed into a localized, instruction-tuned LLM Cross-Encoder.
Unlike Bi-Encoders which compare precomputed vectors independently, the Cross-Encoder performs joint deep attention over both text inputs side-by-side. The model evaluates the pair using a strict root-cause validation prompt:

"Analyze these two customer communications across any language or source differences. Disregard identical product metadata nouns, item colors, sizes, or matching introductory phrases. Determine if both users are experiencing the exact same underlying technical failure mode or manufacturing physical defect. Output a binary classification index (Yes/No) along with a localized confidence interval metric."

This hybrid execution layer entirely neutralizes residual token noun traps, handles underlying sarcasm, and filters soft context sizing drifts dynamically before metrics are surface-routed to executive leadership.

7. System Architecture Diagram

graph TD
    classDef database fill:#2c3e50,stroke:#34495e,stroke-width:2px,color:#fff;
    classDef engine fill:#3498db,stroke:#2980b9,stroke-width:2px,color:#fff;
    classDef audit fill:#f1c40f,stroke:#d68910,stroke-width:2px,color:#2c3e50;
    classDef trap fill:#e74c3c,stroke:#c0392b,stroke-width:2px,color:#fff;
    classDef production fill:#2ecc71,stroke:#27ae60,stroke-width:2px,color:#fff;

    subgraph Layer_1 [Data Context Domain]
        DB_Corpus[(dbo.Docs Corpus<br>83 Multilingual Rows)]:::database
        DB_Metadata[TagsJson Metadata<br>Language, Scenario, Rating]:::database
    end

    subgraph Layer_2 [Stage 1: Raw Vector Retrieval Engine]
        Proc_Lookup[EXEC dbo.FindSimilarDocsByDocId]:::engine
        Embedding_Space[1536-Dim Embedding Matrix<br>Cosine Distance Calculation]:::engine
        Proc_Lookup --> Embedding_Space
    end
    DB_Corpus --> Proc_Lookup

    subgraph Layer_3 [The Forensic Audit & Metrics Arena]
        Audit_Script[zava_scientist_audit.py<br>Pandas Execution Engine]:::audit
        subgraph Probes [Audited Evaluation Seeds]
            P1[Probe 1: EN Review Seed<br>DocId 2: Structural Defects]:::audit
            P2[Probe 2: ES Chat Seed<br>DocId 62: Hardware/App Faults]:::audit
        end
        subgraph Metrics [Dual-Tier Framework Decisions]
            M_Strict[Strict Precision Score<br>P_strict@5 = 40%]:::audit
            M_Broad[Broad Precision Score<br>P_broad@5 = 80%]:::audit
        end
        Output_Plot([retrieval_audit_plot.png<br>Visual Noise Distribution]):::audit
    end
    Embedding_Space --> Probes
    Probes --> Audit_Script
    Audit_Script --> Metrics
    Metrics --> Output_Plot

    subgraph Layer_4 [Documented Systemic Failure Traps]
        T_Language[Language Silo Effect<br>Model groups tokens<br>strictly by source tongue<br>Cross-lingual bridges fail]:::trap
        T_Noun[Product Noun Trap<br>Model over-indexes nouns<br>Happy 5-Star reviews mask<br>1-Star anger complaints]:::trap
        T_Boilerplate[Corporate Boilerplate Trap<br>Legal signatures & UI logs<br>dominate transcripts<br>Sizing returns pollute faults]:::trap
    end
    Embedding_Space -.-> T_Language
    Embedding_Space -.-> T_Noun
    Embedding_Space -.-> T_Boilerplate

    subgraph Layer_5 [Production Architecture Recommendations]
        R_Ingestion[1. Ingestion Alignment<br>& Domain Sanitization<br>Contrastive Multilingual<br>Bi-Encoder Pipeline<br>+ Boilerplate Strip]:::production
        R_Reranker[2. Dual-Input Neural<br>Cross-Encoder Reranking<br>Joint Attention Review<br>over Top 20 Candidates]:::production
        R_Output[3. Optimized Output Feed<br>Eliminates Noun Traps,<br>Silos & Context Drift<br>Dynamically]:::production
        R_Ingestion --> R_Reranker --> R_Output
    end
    Metrics --> R_Ingestion
    DB_Metadata --> R_Reranker

Loading