Fix moodle:sync stacking by cutting per-user Moodle HTTP call volume

[Celeo]

Summary

moodle:sync runs on a 3-hour schedule in the current namespace api-worker pod. A single run was taking 17+ hours, so new runs kept stacking on top of unfinished ones — 6 concurrent instances observed live on the pod, each ~55-63MB RSS, driving CPU to the pod's limit and climbing day over day. Left alone this was on track to eventually OOM the pod.

Why #111 didn't fix it

#111 (merged/deployed 2026-07-19) memoized VATUSAMoodle::getCategories() and added a 30s per-call HTTP timeout, on the theory that a category-fetch N+1 was what pushed a run past the 120-minute withoutOverlapping() lock. Two days after that deploy, the pod still had 6 stacked moodle:sync processes with elapsed times 17h47m → 2h47m — one per missed 3-hour tick, none ever completing. So the N+1 fix was real but not the dominant cost.

Actual root cause: MoodleSync::handle() walks the entire VATUSA user table and fires one core_user_get_users HTTP call per user just to check "is this CID in Moodle?" The large majority of users aren't, so this alone is tens of thousands of sequential round-trips per run. Every user who is in Moodle then triggers several more one-item-at-a-time write calls (cohorts, roles, enrolments) — staff users trigger dozens via a nested category/course enrolment loop. Runtime > the 120-minute lock window, so the lock auto-expires mid-run and the next scheduler tick starts a fresh overlapping instance.

Per api/CLAUDE.md, that 120-minute window on moodle:sync was deliberately shortened (commit a324474) so an orphaned lock (left behind by a pod restart/OOM) self-heals in ~1-2h instead of Laravel's 24h default — so widening it isn't an option. This PR instead cuts real runtime back under the existing window.

What's changing — using Moodle's bulk endpoints

Moodle's Web Service functions are bulk-capable almost everywhere (they accept arrays), but our wrappers only ever called them with single-element arrays, paying one HTTP round-trip per item. This PR:

1. Kills the whole-table existence-check HTTP storm. Adds VATUSAMoodle::getAllUserIdMap(), which builds a full CID -> Moodle user id map with one query against the Moodle DB connection this class already uses elsewhere (clearUserCohorts, getContext, etc. all use DB::connection('moodle')). MoodleSync::handle() now does an in-memory lookup against that map instead of an HTTP call per VATUSA user, and passes the known id into sync() so its own redundant second getUserId() call is skipped too. This removes ~all of the dominant cost.
2. Batches the per-user write calls. Adds assignCohortsBulk(), assignRolesBulk(), enrolUsersBulk() to VATUSAMoodle (the existing single-item methods are untouched — other callers depend on them: ULSHelper, AcademyController, MoodleCompetency). sync() now collects each user's cohort/role/enrolment items and flushes each as one bulk call, turning ~5-30 calls/user into ~3-4.

Explicitly unchanged: Kernel.php scheduling, the withoutOverlapping() values, and every existing VATUSAMoodle method signature. The single-user moodle:sync {cid} CLI path still does its original create-or-update getUserId() lookup — only the whole-table bulk path changed.

Behavior preservation

The bulk sync path only ever iterated users that passed if ($this->moodle->getUserId($user->cid)) — i.e. users already confirmed to exist in Moodle; it never created new users from that path. Iterating $moodleIds->has($user->cid) against the new map is equivalent, so no user that was previously synced is now skipped, and no user is newly synced that wasn't before. Per-user batching doesn't change ordering relative to other users, and each user's own clear-then-assign sequence (clearUserCohorts/clearUserRoles happen before the corresponding bulk assign, same as before) is preserved.

Testing

• Local: ./vendor/bin/phpunit — 5/5 passing, including a new tests/Unit/MoodleUserIdMapTest.php covering the one non-obvious assumption the swap relies on (an int CID lookup matching a map keyed from the Moodle DB's string idnumber column values, via PHP's numeric-string array key coercion).
• The existing suite is smoke-only (in-memory SQLite, no real Moodle), so the bulk Moodle calls themselves can't be integration-tested locally.

Recommended before merging to prod traffic

• Deploy this branch to a dev/staging overlay first.
• Run php artisan moodle:sync manually and time it — expect minutes, not hours.
• Spot-check a few users' cohorts/roles/enrolments in Moodle match what the old per-item path produced (no behavioral drift from batching).
• After deploying to prod, confirm via kubectl -n current exec <api-worker-pod> -- ps -o pid,etime,rss,args that at most one moodle:sync process is ever live, and that it completes well within the 3-hour interval. Cross-check logs for matching Starting/Finished scheduled task: moodle:sync pairs (the after hook never fired while the job was stuck) and watch pod CPU/memory flatten instead of climbing.

Risks

• Bulk-call atomicity is unverified. Moodle's bulk Web Service functions accept an array of items in one call; if one item in the array is invalid (e.g. a category lookup returned null for a context id), it's not confirmed whether Moodle rejects the whole batch or processes valid items and reports per-item errors. Worth watching for on the staging run and after prod rollout — if it does hard-fail whole batches, a bad category mapping for one facility could now block cohort/role assignment for other facilities in the same call, where before it would only have failed that one item.
• getAllUserIdMap() reads idnumber/deleted directly from the Moodle DB, same access pattern as existing methods in this class, but it's a new query shape — if Moodle's schema differs from what's assumed (idnumber as the CID, deleted = 0 for active users), the map could be built incorrectly. Worth a spot-check against real data in staging.
• Scope is intentionally narrow: no changes to MoodleCompetency.php, the scheduler config, or the timeout PR Fix moodle:sync piling up overlapping instances via category-fetch N+1 #111 already added.

🤖 Generated with Claude Code