Mission: Cross-Language Semantic Retrieval Audit for Zava Voice of Customer Application

[capgemhon]

Mission/open goal Description

Evaluate whether Zava's semantic retrieval can reliably identify cross-language customer feedback themes across reviews and support conversations, and measure quality using hand-labeled Precision@5.

Harness and model

GitHub Copilot coding agent in VS Code - Model: GPT-5.3-Codex

Turn-by-turn journey

1) Vector similarity tool

• Tool: mcp_sql_mcp_serve_find_similar_docs_by_doc_id
• Purpose: nearest-neighbor retrieval from precomputed vectors by DocId
• Evidence of calls:
  • DocId=1, TopN=5 -> returned top doc ids [1, 24, 11, 15, 9]
  • DocId=2, TopN=5 -> returned top doc ids [2, 7, 18, 21, 3]
  • DocId=3, TopN=5 -> returned top doc ids [3, 7, 2, 21, 6]

2) SQL MCP execute path

• Tool: mcp_sql_mcp_serve_execute_entity
• Entity executed: FindSimilarDocsByDocId
• Purpose: verify equivalent retrieval through SQL MCP execute pipeline
• Evidence of call:
  • entity=FindSimilarDocsByDocId, DocId=2, TopN=5 -> returned top doc ids [2, 7, 18, 21, 3]

3) SQL MCP entity discovery

• Tool: mcp_sql_mcp_serve_describe_entities
• Purpose: identify available entities and callable operations
• Evidence:
  • Returned entities including Doc and FindSimilarDocsByDocId

Tool-path correction noted

• Attempted SQL MCP table reads through mcp_sql_mcp_serve_read_records.
• Encountered EntityNotFound for configured runtime path.
• Correction: used execute + vector retrieval calls and included explicit evidence in notebook metrics.

Prompt 1

"Create an artifact voc_semantic_retrieval_audit.ipynb ... evaluate cross-language themes and hand-labeled Precision@5."

• Outcome: notebook created with retrieval, label template, P@5 evaluation, and error analysis.

Prompt 2

"run the notebook artifact"

• Outcome: execution attempted; kernel selection friction occurred.
• Correction: proceeded with deterministic file-based completion where needed.

Prompt 3

"It says no completed label file found yet ... can this be done"

• Outcome: generated precision_at_5_labels.csv baseline from template + retrieval output.

Prompt 4

"can you add the above working script into the artifact"

• Outcome: added optional auto-fill baseline section into notebook.

Prompt 5

"at the end can you add ... model-card style note ... overclaimed and reined it in"

• Outcome: added final model-card section with strengths, limitations, and overclaim control statement.

Prompt 6

"did it use sql mcp in the artifact" and "incorporate find_similar_docs_by_doc_id and one sql mcp tool"

• Outcome: incorporated MCP evidence and calculations section.
• Added metrics:
  • mean non-self distance@5 = 0.2987
  • tool agreement for DocId 2: Jaccard@5 = 1.0, exact order match True

Prompt 7

"can you provide ... prompt.md, tools.md, journey.md, architecture.png, improvements.md"

• Outcome: created full documentation package and workflow diagram.

Prompt 8

"place those files in the github repo in capgemhon"

• Outcome: force-added ignored artifact paths, committed, and pushed to origin/main.

Completion

• Yes, the agent completed the mission or goal.
• No, the agent did not complete the mission or goal.

Bonus work

• Added optional baseline auto-label workflow to accelerate annotation.
• Added MCP cross-tool consistency calculations.
• Added model-card style risk framing and overclaim mitigation language.
• Added architecture diagram and improvement roadmap.

Files are on here
https://github.com/capgemhon/sql-ai-promptathon/tree/main/artifacts/semantic_retrieval_audit

[capgemhon]

Files are on here
https://github.com/capgemhon/sql-ai-promptathon/tree/main/artifacts/semantic_retrieval_audit