Update endpoints structure#2630
Conversation
|
MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅ |
This reverts commit 7d8ae1a.
…tale results.json (#2631) The endpoints accuracy artifact was renamed from results.json (with a responses field) to accuracy/accuracy_results.json in mlcommons/endpoints#400, which also dropped the responses field entirely (per-sample text now lives in events.jsonl, not part of the submission). truncate_accuracy_log.py still looked for the old results.json/responses shape and would only log "missing" for every endpoints submission. Detect the endpoints layout via the scenario-root config.yaml and skip truncation, and remove the now-dead PERFORMANCE_ENDPOINTS_DIR/ACCURACY_ENDPOINTS_DIR constants left over from the loader.py refactor to ENDPOINTS_SCENARIO_DIR. Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
|
Tested this submission checker with a couple of internal valid log. Find a bug: currently since both accuracy and performance logs are generated together, submission checker will enforce n_samples_issued and n_samples_completed in performance/result_summary.json to be exactly the same size of the dataset. However, this enforcement should be in accuracy log instead of performance log. Performance log should be valid as long as min duration is > 600000 ms, and number of samples > performance dataset size. This is behavior is correct in legacy loadgen's result parsing path, but not correct in endpoint_parser.py |
| │ │ │ │ │ │ │ │ ├── config.yaml | ||
| │ │ │ │ │ │ │ │ ├── results.json | ||
| │ │ │ │ │ │ │ │ └── results_summary.json | ||
| │ │ │ │ │ │ │ └── run_1 |
There was a problem hiding this comment.
@pgmpablo157321 , could you review PR #2637 which adresses this and below comment by zihaok.
Thanks @zihaok for pointing it out
|
Hi @pgmpablo157321 , could you see #2636 and merge that to this branch. CC: @nvzhihanj |
* Fix submission directory structure for endpoints * Fix filename in endpoints submission structure Corrected the filename from 'results_summary.json' to 'result_summary.json' in the endpoints submission structure documentation. --------- Co-authored-by: Pablo Gonzalez <pgmpablo97@gmail.com>
5 participants
Partially address #2628