Skip to content

Validate live responses against models in CI (drift detection) #997

Description

@sigma67

Part of #307. Base branch: ytmusicapi-2. Blocked until the Phase 2 issues have landed.

The original motivation for #307: notice sooner when YouTube Music changes something, instead of finding out from a user's bug report.

Scope

Add validation of live responses against the models in CI. Today's tests mostly assert on result counts, which catches a section disappearing but not a field changing type, being renamed or quietly vanishing.

Approach to settle in this issue: the models ship with extra="allow" for consumer safety (see #980), so the drift check needs to be stricter than normal runtime behaviour — for example validating with extra="forbid" in tests so a newly appearing YTM field surfaces as a failure rather than being silently absorbed.

Watch out for

  • This must fail loudly but not flakily. YTM A/B tests mean some variation is normal and expected — see the a/b label and fix: skip unexpected renderer types in parse_content_list #970.
  • Consider running it on a schedule rather than per PR, so drift is caught even when nobody is pushing.
  • It needs the shared test account, so mind the existing rate-limiting constraints.

Definition of done

  • A schema change on YTM's side produces a clear failure naming the model and the field.
  • Normal A/B variation does not produce failures.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestv2-modelspydantic response models effort (#307)

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions