Fix commitment discount eligibility fetch (Retail Prices API pagination)

[RolandKrummenacher]

Summary

The weekly Update Commitment Discount Eligibility workflow has been failing on every run. This fixes two independent root causes and regenerates the open-data file.

1. The fetch was broken and non-reproducible

Update-CommitmentDiscountEligibility.ps1 paged the Azure Retail Prices API with an undocumented multi-field $orderby and an explicit $top:

• The API now rejects multi-field $orderby with HTTP 400 (fails on page 1 — the recent ~30s failures).
• Following NextPageLink while sending $top is also broken: the API's NextPageLink decrements $top per page into negatives (1000 → 0 → -1000), returning HTTP 400.
• Even setting that aside, the documented NextPageLink/$skip pagination has no stable total order across page requests — a single long traversal silently drops a scattered ~2–24% of rows, so two identical runs disagreed (~76% Jaccard).

Fix: follow NextPageLink verbatim (no $orderby, no $top), shard each query by serviceFamily so traversals stay short, and repeat each shard unioning by meterId until the set is stable. A full run is now reproducible (two runs byte-identical) and more complete (captures meters the old walk dropped). Added a completeness guard (-MaxShrinkFraction, default 0.15) that aborts before writing if a run is materially incomplete, and switched to writing the freshly-fetched set so retired meters age out instead of accumulating forever.

2. The workflow tried to open a PR, which Actions can't do here

gh pr create failed every run (Actions aren't permitted to open PRs in this org — confirmed via run history: the Create PR step failed while the fetch succeeded). The workflow now pushes a dated branch and surfaces a one-click "create PR" link in the job summary for a maintainer to open (which then triggers Open Data CI and normal review).

Data change

Regenerated CommitmentDiscountEligibility.csv: 121,301 → 92,163 rows

• +6,630 meters the old broken pagination silently missed
• −35,768 retired meters no longer returned by the API (sampled 15/15 confirmed gone)
• 166 corrected eligibility flags

Test plan

• Two full runs produce byte-identical output (reproducibility)
• Completeness-guard arithmetic unit-tested (healthy → write, outage → abort, migration → abort unless overridden)
• Output verified: LF, no BOM, fully quoted, sorted by MeterId, correct columns
• Reviewer: scan the diff for unexpected changes; spot-check a few meter IDs against the Azure Pricing Calculator
• Confirm the first scheduled/dispatched run pushes a branch and posts the PR link (the Actions bot's branch-push permission can only be confirmed by a real run)

🤖 Generated with Claude Code

[Copilot]

Pull request overview

This PR aims to make the weekly “Update Commitment Discount Eligibility” automation reliable again by fixing Azure Retail Prices API pagination handling in Update-CommitmentDiscountEligibility.ps1 and updating the GitHub Actions workflow to push an update branch + provide a manual PR creation link (since Actions cannot open PRs in this org).

Changes:

• Reworks the Retail Prices API fetch to follow NextPageLink verbatim, shard queries by serviceFamily, and repeat/union results until stable; adds a completeness guard to prevent overwriting open data with incomplete results.
• Updates the workflow to push a uniquely named branch and write a “Create pull request” compare-link to the job summary instead of calling gh pr create.
• Regenerates the open-data dataset (per PR description).

Reviewed changes

Copilot reviewed 2 out of 3 changed files in this pull request and generated 3 comments.

File	Description
src/scripts/Update-CommitmentDiscountEligibility.ps1	Implements sharded/repeated API traversal with convergence + completeness guards and writes a fresh (non-cached) dataset.
.github/workflows/opendata-commitment-eligibility.yml	Removes PR-creation permissions/step; pushes a branch and surfaces a PR link in the job summary.
src/open-data/CommitmentDiscountEligibility.csv	Regenerated eligibility data output (not shown in diff snippet, but described in PR).

[RolandKrummenacher]

Thanks @copilot-pull-request-reviewer. Addressed:

Retry/backoff null .Response (line 117) — fixed in 2573af3. On network/DNS/timeout errors the exception has no .Response, so …Response.Headers['Retry-After'] indexed $null and threw "Cannot index into a null array" inside the catch, bypassing the retry. Now guarded — a missing .Response is treated as a retryable "Network error".

$riResult.Keys / $spResult.Keys (lines 247, 260) — not a bug. For a PowerShell [hashtable], key access takes precedence over the .NET .Keys property, so $riResult.Keys returns the stored meter hashtable, not the member-name collection. Verified:

$r = @{ Keys = @{ a=1; b=2; c=3 }; NonConverged = @('x') }
$r.Keys.GetType().Name      # Hashtable
$r.Keys.Count               # 3   (not 2)
$r.Keys.ContainsKey('a')    # True

This also matches the end-to-end runs: RI-eligible meters: 69063 / SP-eligible meters: 82122 and a correct, reproducible CSV — if .Keys had returned the 2-member collection, every count would have been 2. Left as-is.