Mission: Principal Data Scientist — Semantic Voice of Customer System with Multilingual Theme Discovery and Retrieval Audit

[Pranitha-Seemalamudi]

Mission/open goal Description

Build a semantic voice-of-customer theme discovery and retrieval-quality study over Zava's 89-document multilingual corpus (English, Spanish, French). Discover customer themes using precomputed vector embeddings and the find_similar_docs_by_doc_id MCP tool, evaluate retrieval quality with hand-labeled precision audits at k=5 and k=10, and connect findings to real business impact via SQL aggregate queries.

Harness and model

GitHub Copilot Chat in Agent mode — MAI-Code-1-Flash

Turn-by-turn journey

Turn 1: Used describe_entities to discover all 9 MCP entities. Confirmed Docs entity with 1536-dim embeddings.

Turn 2: Retrieved all 89 documents via read_records on Docs. Found 44 Reviews and 45 SupportChats across English (59), Spanish (19), French (11).

Turn 3: Read body text of selected documents to identify 5 seed candidates for theme discovery.

Turn 4: Called find_similar_docs_by_doc_id for 5 seeds (DocIds 2, 44, 55, 89, 87) with TopN=8. French seed DocId 44 retrieved Spanish and English neighbors on the same smart fabric failure theme — cross-language signal confirmed without translation.

Turn 5: Fetched Body text for 15 neighbor documents via read_records to enable honest hand-labeling.

Turn 6: Hand-labeled 30 neighbor pairs. Computed Precision@5 (0.80, 1.00, 0.80) and Precision@10 (0.90, 0.70, 0.80). Found 3 false positives — vocabulary collisions and one fuzzy boundary case.

Turn 7: Discovered 2 additional themes — Tech Support Troubleshooting (DocId 89) and Order Status Inquiries (DocId 87). Total: 5 distinct themes.

Turn 8: Used aggregate_records on SalesOrderLines and SupportTickets to trace Theme 2 SKUs to business data. Found $22,035 revenue tied to products with avg satisfaction score of 1/5.

Turn 9: Built and executed voice-of-customer.ipynb with all sections.

Dead end: Copilot used "Doc" instead of "Docs" entity name twice — causing EntityNotFound errors. Fixed by checking dab-config.json. Agent also merged precision scores across themes until I pushed back and required separate per-theme evaluation.

Completion

• Yes, the agent completed the mission or goal.
• No, the agent did not complete the mission or goal.

Bonus work

1. Extended precision audit to Precision@10 for all 3 core themes, revealing precision degradation from 1.00 to 0.70 for the Smart Fabric theme at deeper retrieval depth.

2. Discovered 2 additional themes beyond the 3 required — Tech Support / Connectivity Troubleshooting and Order Status / Fulfillment Inquiries — bringing total themes to 5.

3. Cross-language validation heatmap showing semantic similarity successfully crosses language boundaries — French seed retrieved Spanish and English neighbors on the same theme without translation.

4. Theme overlap Venn diagram identifying DocId 42 as a genuine semantic boundary case appearing in two themes.

5. Cosine distance distribution analysis with threshold identification — soft cutoff at 0.43 separates true positives from false positives in the Smart Fabric theme.

6. Business impact analysis via SQL — traced Smart Fabric failure theme to $22,035 revenue at risk (SKU ZCPTM-SS-M-BW: 228 units, 9 tickets, avg satisfaction score 1/5).

7. Full executed notebook with 7 analytical sections, all charts generated and verified — voice-of-customer-executed.ipynb pushed to GitHub.

Notebook: https://github.com/Pranitha-Seemalamudi/sql-ai-promptathon/blob/main/voice-of-customer-executed.ipynb