Skip to content

feat(courses): add --dry-run support to migrate_edx_data's users type - #3842

Open
shaidar wants to merge 4 commits into
mainfrom
sar/migrate-edx-data-users-dry-run
Open

feat(courses): add --dry-run support to migrate_edx_data's users type#3842
shaidar wants to merge 4 commits into
mainfrom
sar/migrate-edx-data-users-dry-run

Conversation

@shaidar

@shaidar shaidar commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

What are the relevant tickets?

N/A

Description (What does it do?)

_migrate_users() never checked options.get("dry_run") at all, unlike the course_certificates/entitlements migration types in this same file, which already support it. So migrate_edx_data --type users --dry-run silently wrote real User/LegalAddress/UserProfile rows despite the flag — no error, no warning, just a false sense of safety.

This mirrors the existing dry-run pattern already used elsewhere in this file (e.g. _migrate_course_certificates): inside the batch loop, count what would be created — net of the existing existing_emails de-dup check, so the count matches what a real run would actually produce — and log a [DRY RUN] Would create N users summary instead of calling _bulk_create_users/_bulk_create_legal_addresses/_bulk_create_user_profiles.

How can this be tested?

Added courses/management/commands/test_migrate_edx_data.py (this command had no test coverage at all before). Uses a minimal fake Trino cursor/connection so _migrate_users() can be exercised directly without a real Trino connection. Covers:

  • --dry-run creates zero User rows and correctly excludes already-existing emails from the count.
  • The dry-run count accumulates correctly across multiple fetchmany batches, not just within one.
  • A real (non-dry-run) call still creates User rows as before.

Reverted the fix locally and confirmed both dry-run tests fail (real writes happened despite dry_run=True) before restoring it. Ran ruff check on both changed files — clean, aside from three pre-existing, unrelated missing-docstring findings on lines this PR doesn't touch.

Additional Context

Written and tested against a fresh Django-and-Postgres test environment built directly off main (which now includes the recent ECOMMERCE_DEFAULT_PAYMENT_GATEWAY fix from #3832), so no CI-blocking workarounds were needed.

@github-actions

Copy link
Copy Markdown

OpenAPI Changes

Show/hide changes
## Changes for v0.yaml:
No changes detected

## Changes for v1.yaml:
No changes detected

## Changes for v2.yaml:
No changes detected

Unexpected changes? Ensure your branch is up-to-date with main (consider rebasing).

shaidar and others added 3 commits August 12, 2026 11:13
_migrate_users() never checked options.get("dry_run") at all, unlike
the course_certificates/entitlements migration types which already
support it - so `migrate_edx_data --type users --dry-run` silently
wrote real User/LegalAddress/UserProfile rows despite the flag.
Mirrors the existing dry-run pattern used elsewhere in this file:
count what would be created (net of existing_emails dedup) and log a
[DRY RUN] summary instead of calling the bulk_create methods.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Django's command auto-discovery scans every .py file directly under
management/commands/ as a candidate command module, so
test_migrate_edx_data.py showed up (and would fail) in ./manage.py -h.
Move it to management/tests/, matching the existing convention used by
every other management command test in this app (and in users/).
@shaidar
shaidar force-pushed the sar/migrate-edx-data-users-dry-run branch from 82f7416 to c2fb70b Compare August 12, 2026 16:14
Comment on lines 391 to +393
)

if dry_run:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug: The user migration dry-run can report an inflated count of new users when an incoming email matches an existing user's username but not their email.
Severity: LOW

Suggested Fix

To fix the inaccurate count, the query for existing users should collect both usernames and emails. Instead of just values_list("email", flat=True), fetch both username and email fields. Then, create a set of all existing usernames and emails to check against, ensuring that an incoming email that matches either an existing username or email is correctly excluded from the new user count in the dry-run.

Prompt for AI Agent
Review the code at the location below. A potential bug has been identified by an AI
agent. Verify if this is a real issue. If it is, propose a fix; if not, explain why it's
not valid.

Location: courses/management/commands/migrate_edx_data.py#L391-L393

Potential issue: In the `migrate_edx_data` management command, the dry-run logic for
user migration can produce an inaccurate count of new users. The process identifies
existing users by checking for matches in both the `username` and `email` fields but
only collects the `email` values from the matched records. If an incoming user email
from the data source matches an existing user's `username` but not their `email`, the
dry-run will incorrectly count this as a new user to be created. However, during a real
run, the `bulk_create` operation with `ignore_conflicts=True` will silently fail due to
the unique constraint on the `username`, resulting in zero users being created. This
discrepancy leads to an inflated count in the dry-run report.

Did we get this right? 👍 / 👎 to inform future reviews.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant