E2e rag robustness fixes#2625
Open
mkankana wants to merge 15 commits into
Open
Conversation
mkankana
commented
Jul 9, 2026
mkankana
commented
Contributor
- Fixed a bug causing 0 retrievals due to json parsing error.
- Updated the folder name of model to match the mlc storage download patterns.
This commit improves the reliability of the multi-shot retrieval system:
1. **Robust JSON Parsing** (multi_shot_retrieval.py, accuracy_eval.py):
- Use regex-based extraction to handle JSON in markdown code blocks
- Extract JSON objects from mixed text/markdown LLM responses
- Add fallback behavior when JSON parsing fails instead of crashing
- Applied to: evaluate_document_relevance(), check_sufficiency(),
generate_search_queries(), query_rewriter(), call_judge()
2. **URL Extraction Fixes** (multi_shot_retrieval.py):
- Support both 'original_url' and 'source' metadata fields
- Convert Wikipedia filenames (en.wikipedia.org_wiki_*.html) to URLs
- Debug logging for URL extraction issues
3. **Metrics Compatibility** (multi_shot_retrieval.py):
- Add backward-compatible aliases (precision, recall, f1)
- SUT expects different key names than evaluation module produces
- Added 'retrieved_urls' to metrics output
4. **Config Updates** (user.conf):
- Added e2e-datasetup.Offline configuration
These changes make the system more resilient to LLM output variations
without changing the underlying retrieval logic or model assignments.
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Download additional 470 URLs files.
…lc storage download
Contributor
|
MLCommons CLA bot: |
Align naming end-to-end across the workload and submission tooling: - checker constants.py: e2e -> e2e-rag-qna, e2e_vectorDB -> e2e-rag-db - user.conf loadgen keys and FromConfig namespaces - reference-script output dirs (nested accuracy/performance to avoid the run_output/output collision between QnA accuracy and perf runs) - datasetup scripts: patch e2e-rag-db.Offline.* (was dead e2e-datasetup.*) - compliance/TEST09/e2e-rag -> e2e-rag-qna (+ README, run_compliance paths) - fix stale "E2E DocGrader" header comments Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The RAG-DB accuracy test only did local self-consistency checks and
never verified the built vector DB against the reference manifest, so a
divergent corpus/embedding could still pass.
- db_manifest.py: extract cmd_verify body into reusable verify_manifest()
that returns {passed, failures, metrics} instead of sys.exit; add a
--retriever_model override (manifest's stored path is a system-specific
absolute path that breaks verify on other systems) with fallback to the
manifest value.
- datasetup_accuracy_eval.py: run verify_manifest() when --manifest is
given; record results in accuracy_results["manifest"], gate overall pass
on it, and write a manifest section into accuracy.txt. Skipped cleanly
when no manifest is provided.
- reference_mlperf_datasetup_accuracy.sh: pass the bundled reference
manifest (scripts/db_manifest_intel_xpu.json.gz) by default.
- scripts/verify_db_manifest.sh: pass local retriever model to verify.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The RAG-QnA accuracy run produced only accuracy_results.json; the
submission checker needs an accuracy.txt with an "Accuracy:" line.
- accuracy_eval.py: write accuracy.txt into the loadgen log dir with the
LLM judge answer accuracy as a percentage (answer_accuracy * 100),
matching the checker's E2E_ACCURACY pattern. hash= line and truncation
are added later by truncate_accuracy_log.py during submission prep.
- reference_mlperf_{accuracy,perf}.sh: fix stale user.conf sed patch that
still targeted the old e2e.Offline key (now e2e-rag-qna.Offline).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The submission checker gates compliance validation on the models_TESTxx lists; e2e-rag-qna was in none, so TEST09 results were silently unchecked despite the workload shipping run_compliance_test09.sh and producing verify_output_len.txt. Add e2e-rag-qna to models_TEST09 (v6.1) so the checker requires TEST09/verify_output_len.txt and validates it contains "TEST PASS". e2e-rag-db has no compliance test and is left out. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Workload-side fixes validated end-to-end via a 10-doc/10-query smoke run
through OpenRouter in a container:
- run_compliance_test09.sh: pass --audit_conf to reference_mlperf.py so
loadgen actually applies the TEST09 audit.config (previously the default
audit.conf was used, leaving accuracy_log_sampling_target=0 and an empty
mlperf_log_accuracy.json -> verification found no entries). Make PERF_COUNT
overridable and sync the audit.config min_query_count to it. Source the
TEST09 config from the main repo's compliance tree (copied in / overridable
via COMPLIANCE_DIR).
- reference_mlperf_{perf,accuracy}.sh: capture and propagate the python exit
code (was masked by a trailing echo, so failures looked like success).
- datasetup_accuracy_eval.py: write a checker-compliant "Accuracy:" line
(indexing success rate) and always emit the DB Manifest Verification
section (PASS/FAIL/SKIPPED) in accuracy.txt.
- reference_mlperf_datasetup_accuracy.sh: honor an explicit empty/"none"
MANIFEST to skip the manifest check (use ${VAR-default} so empty is
respected).
- user.conf: e2e-rag-db counts 2503->2515; drop dead e2e-datasetup.* keys.
- .gitignore: ignore run_output_*/output/submission smoke artifacts.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The e2e-rag rename originally also edited the submission checker's constants.py, but all checker changes now live in the dedicated e2e-rag-submission-checker-fixes branch/PR. Revert constants.py to upstream here so the workload PR touches no checker files and does not conflict with the checker PR. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…s-fixes # Conflicts: # e2e-rag/evaluation.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.