Skip to content

feat(scim): add migrate_and_sync_users orchestrator command - #3837

Draft
shaidar wants to merge 1 commit into
sar/fix-scim-name-mappingfrom
sar/migrate-and-sync-users-orchestrator
Draft

feat(scim): add migrate_and_sync_users orchestrator command#3837
shaidar wants to merge 1 commit into
sar/fix-scim-name-mappingfrom
sar/migrate-and-sync-users-orchestrator

Conversation

@shaidar

@shaidar shaidar commented Aug 10, 2026

Copy link
Copy Markdown
Contributor

What are the relevant tickets?

N/A

Description (What does it do?)

Replaces the old two-step manual migration process — running migrate_edx_data, then an ad hoc SCIM sync script with no field-level visibility or validation — with one auditable command, migrate_and_sync_users:

  • Stage 1 (backfill): call_command("migrate_edx_data", type="users") unless --skip-edx-migration.
  • Stage 2 (classify, not mutate): every sync candidate (is_active=True, not yet SCIM-synced) is run through LearnUserAdapter._resolve_name()'s three tiers, purely for classification and reporting — this stage never writes to legal_address.
    • legal_address complete → sync normally.
    • Only User.name available (split derived) → sync normally by default, flagged as lower-confidence in the report — not --force-gated, since for this population the split is the best data that exists.
    • Neither available → blocked by default; --force syncs anyway with a blank name, explicitly flagged rather than silently dropped.
  • Stage 3 (sync): mitol.scim.api.sync_users_to_scim_remote(...) in batches, with per-user structured logging instead of only aggregate 409-conflict counts (the old ad hoc script's only error visibility).
  • Stage 4 (verify): diffs the name.givenName/name.familyName in each returned UserState.response_body against what was sent — no extra Keycloak API calls, since the Bulk response already echoes the created resource.
  • Stage 5 (report): writes a JSON report with three buckets — synced-and-verified, blocked-or-failed (with reason), verified-but-mismatched (would catch a transport/plugin-level bug, not just a source-data gap) — to --report-path or stdout.

How can this be tested?

users/management/tests/migrate_and_sync_users_test.py mocks scim_api.sync_users_to_scim_remote's return value (a list of duck-typed fake UserState objects, decoupled from whatever mitol-django-scim version happens to be installed) and covers: dry-run classification without syncing, tier-3 (no name data) users being blocked by default and synced with --force, matching/mismatched response-body verification, and failed syncs being reported distinctly rather than silently swallowed.

Ran locally with a fresh test DB (uv run pytest users/management/tests/migrate_and_sync_users_test.py --create-db): all passing. ruff check clean.

Additional Context

Draft, blocked on a dependency: this command calls scim_api.sync_users_to_scim_remote(...) and reads .success/.response_body/.error off the returned list — but the currently published mitol-django-scim still returns None from that function, not a list. The companion PR mitodl/ol-django#544 adds that return value; until it's released and this repo's mitol-django-scim pin (pyproject.toml) is bumped, running this command for real would raise a TypeError iterating None. Do not merge before that dependency is updated.

Based on top of #3836 (the LearnUserAdapter fix this command's Stage 2 relies on via _resolve_name()) — that should merge first.

@github-actions

Copy link
Copy Markdown

OpenAPI Changes

Show/hide changes
## Changes for v0.yaml:
No changes detected

## Changes for v1.yaml:
No changes detected

## Changes for v2.yaml:
No changes detected

Unexpected changes? Ensure your branch is up-to-date with main (consider rebasing).

@shaidar
shaidar force-pushed the sar/migrate-and-sync-users-orchestrator branch 2 times, most recently from 574ee71 to 005b207 Compare August 10, 2026 19:57
Replaces the old two-step manual migration process (running
migrate_edx_data, then an ad hoc SCIM sync script with no field-level
visibility or validation) with one auditable command:

- Stage 1: backfill edX user data (call_command migrate_edx_data).
- Stage 2: classify every sync candidate via LearnUserAdapter's
  _resolve_name() tiers - report only, never mutates legal_address.
  Only candidates with no name data anywhere are blocked by default
  (override with --force); a split-from-User.name candidate syncs
  normally, flagged as lower-confidence in the report.
- Stage 3: sync via mitol.scim.api.sync_users_to_scim_remote, with
  per-user structured logging instead of only aggregate error counts.
- Stage 4: verify what Keycloak actually stored by diffing the
  response body sync_users_to_scim_remote now returns, against what
  was sent - no extra API calls needed.
- Stage 5: write a JSON report (synced-and-verified /
  blocked-or-failed / verified-but-mismatched).

Depends on the response-capture change in
mitodl/ol-django#544 - sync_users_to_scim_remote
needs to return UserState.response_body for Stage 4 to work. Opening
as a draft until that PR releases and the mitol-django-scim pin here
is bumped.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@shaidar
shaidar force-pushed the sar/migrate-and-sync-users-orchestrator branch from 3e806b6 to b60c56c Compare August 11, 2026 13:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant