Fix missing rollouts in dashboard - #531
Conversation
|
Documentation preview: https://modal-labs-training-gym--training-gym-previews-preview-r-78da35.modal.run/531/docs |
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
zhouhelena1
changed the title
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
| continue | ||
| return sorted(summaries, key=lambda summary: summary.rollout_id) | ||
| key = f"{training_run_id}__{summary.rollout_id:08d}" | ||
| if key not in summaries and key not in keys: |
There was a problem hiding this comment.
🟡 Malformed canonical record hides rollout
When a canonical record is malformed, list_summaries_for_run discards its valid legacy summary. Failed canonical validation then removes that rollout from the dashboard.
Learn more
A legacy summary is usable independently of its canonical rollout payload. This condition suppresses that summary solely because the canonical filename exists. The later recovery loop catches malformed canonical data and continues, but it cannot restore the discarded summary. The method therefore omits a rollout that it could still list.
Example: run__00000012.json exists but contains an invalid rollout payload, while the legacy index has a valid summary for rollout 12. The legacy summary is skipped, canonical validation fails, and rollout 12 is absent instead of retaining its summary.
Recommended fix: Keep valid legacy summaries as fallbacks for canonical keys. Replace them only after the corresponding canonical payload validates and produces a rebuilt summary.
Was this helpful? React with 👍 or 👎 to provide feedback.
1 participant
The dashboard reads
training-rollouts-summary/summary.jsonto build the rollout list.