Skip to content

Improve run_project feedback and large-cohort scaling - #77

Merged
michaelhallquist merged 1 commit into
mainfrom
codex/improve-run-project-feedback
Sep 1, 2026
Merged

Improve run_project feedback and large-cohort scaling#77
michaelhallquist merged 1 commit into
mainfrom
codex/improve-run-project-feedback

Conversation

@michaelhallquist

Copy link
Copy Markdown
Contributor

Summary

  • add plain-language status messages before Slurm submission, including a clear notice when BrainGnomes must read a large container file in full
  • keep exact container identification in the existing project-local cache, without introducing a user-wide cache
  • apply requested subject filters before recursive session discovery to reduce unnecessary filesystem work
  • bound subject-selection previews and submission progress output for cohorts with hundreds or thousands of subjects
  • update release documentation and tests for the 0.9-2 behavior

This is a follow-up to #76 based on performance review of run_project with the fMRIPrep 25.2.5 pipeline test data.

Validation

  • devtools::document()
  • full devtools::test(): passed; one opt-in Slurm integration test skipped by its guard
  • clean-source devtools::check(): 0 errors, 1 environment warning for unavailable qpdf, and 2 environment/compiler notes
  • focused provenance, discovery, progress, dry-run, and subject-ID tests: passed
  • synthetic 1,000-subject discovery benchmark: 0.395 seconds for full discovery and 0.073 seconds when selecting one subject

Packaging note

Large local runtime directories and check artifacts were already covered by .Rbuildignore. The final check used a clean source snapshot because R package build enumeration can still traverse a very large live development tree before applying those exclusions.

Explain pre-submission work in applied-user language, report bounded subject progress, and avoid traversing unrequested subject session trees. Keep exact project-local artifact checksums while making slow first reads visible.
@michaelhallquist
michaelhallquist merged commit 247fe72 into main Sep 1, 2026
6 checks passed
@michaelhallquist
michaelhallquist deleted the codex/improve-run-project-feedback branch September 1, 2026 17:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant