Skip to content

Hotfix/task routing fix#77

Closed
laurejt wants to merge 6 commits into
mainfrom
hotfix/task-routing-fix
Closed

Hotfix/task routing fix#77
laurejt wants to merge 6 commits into
mainfrom
hotfix/task-routing-fix

Conversation

@laurejt

@laurejt laurejt commented May 8, 2026

Copy link
Copy Markdown
Contributor

Associated Issue(s): resolves #76

Changes in this PR

  • Fixes concept-eval recipe by adding a custom session factory and disabling auto_exclude_current as suggested by Prodigy support
  • Updates changelog

Notes

  • To reproduce the task routing error, it's important to restart the server after one language's session completes the majority of examples. On the restart, the other language session should only be able to access the small number of examples that have not be annotated.
  • I recommend using the tiny test dataset I created debug-task1.jsonl which contains 5 examples per language for reproducing the error and testing the fix
  • To test the fix, prodigy_recipe_url needs to be updated to https://raw.githubusercontent.com/Princeton-CDH/muse/refs/heads/hotfix/task-routing-fix/src/muse/annotation/annotation_recipes.py
  • I also recommend updating the dataset name prodigy_dataset to something not in the test database to avoid encountering any preexisting annotation data. You can see what datasets are in the database by running prodigy stats -l

Reviewer Checklist

  • Confirm that annotators are served all examples irrespective of any other annotator's progress
  • Confirm that annotators are not re-served examples they've already annotated

@laurejt laurejt requested review from rlskoeser and tanhaow and removed request for tanhaow May 8, 2026 16:09
@laurejt laurejt marked this pull request as ready for review May 8, 2026 16:14
@laurejt laurejt requested a review from tanhaow May 8, 2026 16:20

@tanhaow tanhaow left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tested on the staging server and it worked well. I didn't change the recipe back just in case you want to do more testing on the branch. Just remember to revert it before you merge. :)

@laurejt laurejt closed this May 8, 2026
@laurejt laurejt deleted the hotfix/task-routing-fix branch May 8, 2026 21:39
@laurejt

laurejt commented May 8, 2026

Copy link
Copy Markdown
Contributor Author

Renamed branch to match conventions (should have been version number), which is now under #78

@laurejt laurejt mentioned this pull request May 8, 2026
2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Resolve annotation routing and task sharing in prodigy

2 participants