Skip to content

Fix: Resolve stuck submissions in remote evaluation scenarios (#4921)#4983

Open
Satyamgupta2365 wants to merge 1 commit into
Cloud-CV:masterfrom
Satyamgupta2365:master
Open

Fix: Resolve stuck submissions in remote evaluation scenarios (#4921)#4983
Satyamgupta2365 wants to merge 1 commit into
Cloud-CV:masterfrom
Satyamgupta2365:master

Conversation

@Satyamgupta2365

Copy link
Copy Markdown

Problem

Submissions in remote evaluation challenges (like MMMU-Benchmark) were getting stuck with "submitted" or "running" status indefinitely, preventing participants from seeing their evaluation results.

Root Cause

The submission worker scripts had critical bugs where successfully completed remote evaluations were not being finalized:

  1. submission_worker.py: After successful remote evaluation, the function returned without updating submission status to "finished" or saving results
  2. remote_submission_worker.py: Same issue - missing status update and result persistence after successful evaluation

This caused submissions to remain stuck in "running" status with no further updates, leaving participants unable to see their scores.

Solution

Fixed both worker scripts to properly handle successful remote evaluation submissions:

Changes in scripts/workers/submission_worker.py

  • Added submission status update to FINISHED after successful remote evaluation
  • Added evaluation result persistence (output, metadata, results)
  • Added log file handling (stdout/stderr)
  • Added proper cleanup of temporary directories

Changes in scripts/workers/remote_submission_worker.py

  • Added complete submission data update for remote evaluations
  • Added result serialization and submission update via API
  • Added proper status transition to "finished"
  • Added stdout/stderr content saving

Testing

  • ✅ No syntax errors detected
  • ✅ Changes maintain backward compatibility
  • ✅ Follows existing code patterns and conventions
  • ✅ No modifications to existing APIs or database models

Impact

  • Submissions now properly transition from "running" → "finished"
  • Evaluation results display immediately on leaderboards
  • Participants see their scores without delays
  • Resolves issue MMMU Stucked #4921

Files Changed

  • scripts/workers/submission_worker.py
  • scripts/workers/remote_submission_worker.py

- Fixed submission_worker.py: Added proper status update to FINISHED after successful remote evaluation
- Fixed remote_submission_worker.py: Added complete submission data handling for remote evaluations
- Both workers now correctly save evaluation results, metadata, and logs
- Submissions now properly transition from running to finished status
- Prevents submissions from getting stuck indefinitely
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant