Skip to content

Report MONAI Model Zoo bundles awaiting a triage decision - #508

Open
garybrowne wants to merge 2 commits into
mainfrom
feat/monai-model-triage
Open

garybrowne wants to merge 2 commits into
mainfrom
feat/monai-model-triage

Conversation

@garybrowne

Copy link
Copy Markdown
Contributor

Closes #485.

The gap

filter_whitelist intersects the whitelist with the Zoo listing, so a newly published bundle is invisible — nobody learns about it without going and looking. That was the reviewer's question on #477: "How will we know when potential models are added to MONAI without checking them ourselves?"

Approach

Splits what was an implicit on/off list into three states, only two of which are written down:

State Where recorded Meaning
approved models: generate and publish
declined declined: reviewed, deliberately not published
candidate derived not yet triaged

Candidates are computed as available − models − declined, so monai_whitelist.yaml records human decisions only and needs no edit when the Zoo changes.

declined:
  mednist_gan:
    reason: synthetic image GAN, not a clinical pipeline
    at_version: 0.4.4

at_version scopes a decline to what was actually reviewed — a model declined as immature resurfaces once the Zoo moves past it, where a plain blacklist buries it permanently. Omit it to decline every version.

What gets reported

Three things, as an S4 issue per sync run:

  1. New bundles to triage — emitted paste-ready with modality/species/region blank, so approving is copy, fill three fields, commit
  2. Declined models with a newer version — bump at_version to re-decline
  3. Approved models missing from the Zoo — previously skipped silently by filter_whitelist, leaving a spec building against a model that is no longer published

Nothing to report means no issue is opened.

Why not auto-generate a PR per new model

The reviewer's original suggestion. Two reasons it was not taken, both discussed on #485:

  • The Zoo has 35 bundles against 1 approved, so it would open ~34 PRs — each vendoring a bundle — for humans to reject. It also inverts the default to "publish unless stopped".
  • modality/species/region are human anatomical classification absent from bundle metadata, and they determine where the spec lands in the tree. An auto-generated PR would have to guess them.

Verification

  • 33 tests pass (25 existing + 8 new), no regressions
  • Run against the live Model Zoo: reports 34 candidates, matching 35 bundles minus the one approved
  • All three detectors exercised with a fixture covering every state, including a decline with no at_version correctly never going stale

Also in here

  • permissions: block added to monai-sync.yml — it had none, so it ran on default token permissions. issues: write is needed for the new step; contents/pull-requests made explicit for create-pull-request. Slightly beyond the issue scope, so worth a look.
  • Schedule moved to 0 16 * * 0 — 02:00 Monday Sydney (AEST). Drifts to 03:00 over AEDT since cron has no timezone; not worth two schedules for a weekly background job.

Not covered

  • The gh issue create step is untested in CI — everything else was run locally, but issue creation only exercises on a real dispatch.
  • Issue per run, as agreed, rather than one long-lived issue that gets updated. The latter avoids weekly noise but needs find-or-create logic — worth adding only if the noise turns out to be real.

🤖 Generated with Claude Code

garybrowne and others added 2 commits September 18, 2026 15:59
`filter_whitelist` intersects the whitelist with the Zoo listing, so a newly
published bundle was invisible -- nobody learned about it without going and
looking. This adds the reporting the #477 reviewer asked for.

Splits what was an implicit on/off list into three states, only two of which
are written down:

  approved   `models:`    generate and publish
  declined   `declined:`  reviewed, deliberately not published
  candidate  derived      not yet triaged

Candidates are computed as (available - models - declined), so the whitelist
records human decisions only and needs no edit when the Zoo changes. Declines
carry `at_version`, scoping the decision to what was actually reviewed: a model
declined as immature resurfaces once the Zoo moves past it, where a plain
blacklist would bury it permanently. Omitting `at_version` declines every
version, for reasons that will never change.

Also surfaces approved models that have disappeared from the Zoo, which
`filter_whitelist` skips silently -- leaving a spec in the tree building
against a model that is no longer published.

Not auto-generating a PR per new bundle: the Zoo has 35 bundles against 1
approved, so that would open ~34 PRs for humans to reject, and it inverts the
default to "publish unless stopped". The anatomy fields are also human
classification absent from bundle metadata, so an auto-generated spec would
have to guess where in the tree it belongs. The report emits them paste-ready
with the fields blank instead.

Issue per run for now, as agreed -- a single updated issue avoids weekly noise
but needs find-or-create logic; worth adding only if the noise is real.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
16:00 UTC Sunday, which is 02:00 Monday in AEST. Drifts to 03:00 over AEDT
since cron has no timezone -- immaterial for a weekly background job.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

Update model auto-discovery and management process

1 participant