Skip to content

test: characterization suite + score-service parity report - #5

Open
beta-devin-ai-integration[bot] wants to merge 3 commits into
uk-credit-legacyfrom
devin/1792675183-characterization-suite
Open

beta-devin-ai-integration[bot] wants to merge 3 commits into
uk-credit-legacyfrom
devin/1792675183-characterization-suite

Conversation

@beta-devin-ai-integration

@beta-devin-ai-integration beta-devin-ai-integration Bot commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Adds a black-box HTTP characterization suite (characterization-tests, JUnit 4 / Java 8) that pins what the legacy estate does today, so score-service-boot can be checked for parity before cutover. The tests are skipped in the normal build. Run them with -Dcharacterization=true against running services. -Dscore.url switches the score-service under test, and -Dgroups=com.ukcredit.characterization.ScoreApi limits the run to the checks that go through score-service.

  • Golden master: after scripts/db-reset.sh (seed 20241231), GET /score/{id}?asOf=2024-12-31 for all 256 seeded consumers is recorded in golden/score-uks3.json, and GET /creditfile/{id} for 100001–100016 in golden/creditfile-edge-cases.json. Regenerate with -Dgolden.record=true. Each run writes field-level diffs to target/characterization/<target>/*-golden-diff.json.
  • One named test per quirk: StatusXCutoffTest, DisputeSlaTest, UtilisationRoundingTest, NameMatchingTest, AddressHistoryMergeTest, plus DisputeScoreFlowTest, which checks for DISPUTE_OPEN and DISPUTE_CLOSE rows in SCORE_HISTORY and restores its own state.
  • Behaviour pinned as it is today, beyond what the docs say:
    • the SLA cutoff ignores seconds (17:30:59 still counts as same day);
    • an unknown region (NI) gets no bank holidays at all;
    • the maiden-name alias is not used for matching;
    • dispute-triggered rescoring uses the score service's current date, not 2024-12-31.
  • .github/workflows/characterization.yml (plain GitHub Actions, pull_request → uk-credit-legacy): set up Java 8 → build → db-reset → run-all → suite. The matrix is legacy, plus boot when score-service-boot/ exists on the branch. The boot leg uses Java 21 and starts Boot on :8092 against the same Postgres (SPRING_PROFILES_ACTIVE=postgres, PORT, DB_*). It points dispute-service at Boot and runs the ScoreApi checks. Surefire reports and the parity report are uploaded as artifacts.
  • characterization-tests/parity/parity_report.py builds docs/cutover/score-service-parity-report.md from two runs. Root causes, approved intentional changes and out-of-suite notes are recorded in docs/cutover/parity-findings.json.

Parity summary: legacy score-service vs score-service-boot

Candidate: #7 (devin/1792677011-score-service-boot3 @ 8e8b546). It was checked out locally without committing to it and run on JDK 21 against the same PostgreSQL 15 database. Each implementation got a fresh db-reset first.

Result Checks Consumer scores (golden master)
MATCHED 5 256
DIFFERED 0 0
INTENTIONAL CHANGE 0 0

Recommendation: GO, subject to sign-off from Engineering, QA, Risk/Model Governance and Platform. Changes that #7 declares intentional but this suite doesn't exercise are listed in the report for explicit acceptance: JSON error bodies instead of Jetty HTML, default port 9082, and the shared SCR_HIST_SEQ.

The Actions run on this PR covers only the legacy target, because score-service-boot/ isn't on this branch. The boot matrix leg runs automatically on PRs where both are present, for example #7 once this lands.

Link to Devin session: https://cognition.beta.devinenterprise.com/sessions/b69735ce58a04288bfdaa7bb3d35c858
Open in Devin Desktop: https://cognition.beta.devinenterprise.com/desktop/session/b69735ce58a04288bfdaa7bb3d35c858?variant=devin
Requested by: @sofiasepielli1


Devin Review

Status Commit
⚪ Not started —

Run Devin Review

Devin Review (Staging)

Co-Authored-By: Beta-Devin AI <248786709+beta-devin-ai-integration[bot]@users.noreply.github.com>
@beta-devin-ai-integration

Copy link
Copy Markdown
Contributor Author

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

  • Disable automatic comment, CI, and merge conflict monitoring

sofiasepielli1 and others added 2 commits October 22, 2026 13:41
Co-Authored-By: Beta-Devin AI <248786709+beta-devin-ai-integration[bot]@users.noreply.github.com>
…up in workflow

Co-Authored-By: Beta-Devin AI <248786709+beta-devin-ai-integration[bot]@users.noreply.github.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant