Release: Copilot usage reports, Copilot audit field capture, Teams add-on deprecation, CI enforcement

[sambetts]

Release candidate: dev → main.

Base main @ 5ff6327 → head dev @ a936caf (merge-base 2ada302). main's extra commits are prior release merges and contribute no tree difference, so two-dot and three-dot diffs are identical.

31 squash-merged PRs, 210 files, +37,032 / −3,531, 44 commits (the remaining commits are older-style branch merges and a few direct pushes). This body is kept current as PRs land; it describes the release as it now stands rather than as it was first opened. Last updated for #314.

Closes #270, closes #271, closes #272, closes #276, closes #278, closes #286, closes #290, closes #295, closes #297, closes #302, closes #285, closes #287, closes #294, closes #296, closes #312.

(The first ten were closed by hand on 2026-08-21 once their fixes reached dev; the keywords are kept so the association is recorded on merge to main. main is the default branch, so these fire here — they would have been inert in the dev-targeted PRs.)

---

1. What is in this release

PR	Change	Schema	Config schema
#262	Copilot audit import persists parsed-but-dropped fields	5 new tables + columns on 4 existing	—
#260	Imports the three Graph Microsoft 365 Copilot usage reports	3 new tables	1.9.0 → 1.10.0 (additive)
#279	Deprecates Teams add-on / app-install tracking, removes the import	Conditionally drops 3 tables	1.10.0 → 2.0.0 (BREAKING)
#291	Copilot AI interaction history import (opt-in, statistics only)	10 new tables	2.0.0 → 2.1.0 (additive)
#292	Copilot licence adoption tool (find unused seats / unlicensed heavy users)	—	—
#263	Stops mailbox-less users flooding App Insights with 404 exceptions	—	—
#264 #265 #266 #267 #268 #269	Telemetry service: EasyAuth fix, App Insights instrumentation, dashboard rework, tests + CI/CD, Node 20 removal	—	—
#288	Telemetry service integration tests + an unenforced authorisation scope, fixed	—	—
#289	Azure configuration drift detection (maintainer CI only)	—	—
#280 #282	CI: required checks now actually run on PRs; EF migration snapshot guard	—	—
#284	Release review fixes (manual-script prerequisite, O(N²) row filter)	—	—
#293	Multi-model review fixes: adoption period fan-out, drift-summary redaction, import cadence gate, back-off, dedup re-key and 7 more	—	—
#298	Copilot interaction history: UserGroupsFilter becomes an optional narrowing rather than a precondition; unnarrowed user selection moved into SQL	—	—
#299	Tests.FakeDataGen: generates Copilot prompt history and the new audit tables; stronger upgrade-from-stable rehearsal	—	—
#301	Reverts the profiling weekly-boundary change out of this release (the underlying bug is tracked in #300, with an [Ignore]d repro test kept in place)	—	—
#303	Both Copilot adoption trend series computed in a single pass instead of two identical scans of copilot_chats → audit_events (#295)	—	—
#304	Retention bound added to the three Copilot usage-report tables, which were growing forever, plus per-user purge for the activity log (#286)	—	—
#305	GitHub API rate limiting no longer breaks CI or real installs; a 403 is explained instead of looking like a permissions error (#302)	—	—
#306	Copilot interaction history: wildcard scope and group-discovery caps no longer weaken the Graph call brake; truncated scope reported as a failure (#297)	—	—
#307	RC fixes: Graph paging failures no longer recorded as successful empty imports (#285); accessed-resource de-dup keeps distinct actions instead of collapsing them (#287); interaction-history de-dup bounded to a time window (#294); keyword_id index for the orphan-keyword cleanup (#296)	3 index migrations	—
#314	Fixes fractional chart axis labels (a 0–1 sentiment axis rendered as "1, 1, 0, 0, 0, 0") and corrects the sentiment scale in the chart description — it is a 0.0–1.0 positive-sentiment confidence, not −1 to +1	—	—
#313	Reports: Copilot prompt insights. Adds a word cloud of the most common extracted prompt phrases, a weekly prompt-sentiment line chart and a prompt-language breakdown to the Copilot tab — shown only when cognitive services are configured (#312)	—	—
#310	Makes the activity-report phase fail per report rather than all-or-nothing, and stops #309's strict paging aborting install verification. Also closes two persisted paths that still wrote Graph request URLs into the import-log error column, and hardens the de-dup fail-open sentinel against a SqlDateTime overflow	—	—
#309	Release-critique follow-ups: the legacy daily usage reports (SharePoint/Teams/OneDrive/Yammer/Exchange) had the same #285 hole and now fail loudly instead of recording an empty day; a fail-closed de-dup guard made fail-open; Graph URLs kept out of the persisted error column; three documentation errors	—	—
#308	Engineering notes and PR template: corrects two pieces of guidance disproven by this release's own measurements, reconciles a contradiction in the schema-performance gate, and adds migration/config checklists	—	—
—	Combined O365 + Copilot profiling data generator (Tests.FakeDataGen), pushed directly to dev	—	—

Verified against the diff, not the PR text

Claim	Verification
CONFIG_VERSION ends at 2.1.0	BaseSolutionInstallConfig.cs on dev reads "2.1.0"
7 migrations, each with a .manual.sql	7 .cs + 7 matching .manual.sql in the main...dev diff (1:1, verified by name)
Create DB.sql needs no change	Unchanged in the diff — and it contains no Copilot tables at all, not even copilot_chats from 2024. It is a legacy base schema; every Copilot table has always come from a migration. This is verified, not an omission.
Config-schema delta	Added GraphCopilotUsageReports, CopilotInteractionHistory; removed GraphUserApps

The break is the removal of ImportTaskSettings.GraphUserApps. Older configs still load — the property is simply ignored.

---

2. Migrations

Seven new migrations — four additive (bar the conditional drop in DeprecateTeamsAddons), plus three index migrations from #307. Each has its own .manual.sql.

Note the diff also shows an eighth migration file, 202607231700001_CoverCopilotAccessedResourceDedup.cs, as modified. That migration already shipped in the previous stable release; #309 changed only its XML doc comment (to record that #307 supersedes its de-dup tuple, and that upgrading from before it builds the index twice). Its Up_Sql / Down_Sql are byte-identical, so no customer database is affected and no new manual script is required. Verified: the diff for that file contains no non-comment lines.

Migration	Effect	Cost
202608190622001_CopilotDroppedAuditFields	5 Copilot tables + 9 nullable columns, and two FK indexes on copilot_event_accessed_resources	Tables/columns are metadata-only and instant. The index pair is one of the two heavy steps in the release (the other is the dedup rebuild below, on the same table).
202608190725064_AddCopilotUsageReports	3 usage-report tables	Metadata-only; only lock on existing data is a brief schema lock on dbo.users for the FK
202608191533567_DeprecateTeamsAddons	Drops the 3 Teams add-on tables only when all three are empty	EXISTS test, not COUNT(*), so O(1)-ish even on a billion-row table
202608200600001_AddCopilotInteractionHistory	10 empty interaction-history tables + indexes	Purely additive: no table altered, no rewrite, no blocking lock. Seconds at any size
202608210700001_WidenCopilotAccessedResourceDedupIndex	Rebuilds IX_copilot_event_accessed_resources_dedup with action_id + list_item_unique_id_id as key columns (#287)	Drop + rebuild on an existing, potentially large table. 5.9 s / 83 MB at 2M rows; ~30 s at 10M, ~5 min at 100M. Same table and same window as the CopilotDroppedAuditFields index pair
202608210700002_IndexCopilotInteractionKeywordsByKeyword	(keyword_id, interaction_id) on the Copilot keyword link table (#296)	Table is new in this release → empty on upgrade → instant
202608210700003_IndexCopilotInteractionsDedupWindow	(session_id, created_utc) INCLUDE (graph_interaction_id) (#294)	Table is new in this release → empty on upgrade → instant

Gate classification (schema-performance rule)

Stating this explicitly, because an unclassified migration reads as an unmeasured one:

Migration	Class	Measurement
AddCopilotUsageReports	Additive; both indexes are functional upsert keys	Not required — supplied anyway (§3)
AddCopilotInteractionHistory	Additive; all 19 indexes on new empty tables	Not required — retro-measured in round 2
DeprecateTeamsAddons	Removal only, conditional on emptiness	Not required
CopilotDroppedAuditFields	Additive tables/columns + 2 FK indexes on an existing large table	Judgement call. Both columns are NULL on 100% of existing rows (no backfill), so there is no "before" query. Build time + storage supplied, plus a no-regression check (insert_junction 328 → 314 ms). Classified not performance-motivated; needs conscious sign-off
WidenCopilotAccessedResourceDedupIndex	Correctness-motivated, with a measured performance defence	✅ 500-row batch 1,623 → 1,622 reads / 2 ms; 20,000-row batch 63,889 → 63,883 reads, 85 → 94 ms; seek retained at both. Rejected INCLUDE variant documented (fewer reads, 5.5x slower)
IndexCopilotInteractionKeywordsByKeyword	Performance-motivated	✅ per-user purge 11,256 → 137 reads, 155 → 16 ms, Scan → Seek; bulk sweep 797 → 437 ms (reads unchanged, reported honestly)
IndexCopilotInteractionsDedupWindow	Performance-motivated	✅ 288,000 → 12,467 rows loaded, 5,484 → 3,050 reads, 45 → 6 ms, seek retained. Bounding without the index measured worse than not bounding (337 ms) — code and migration must ship together

Non-migration SQL in gate scope: common_upsert_copilot_agents.sql carries a performance-motivated de-dup re-key (1.8x fewer reads / 5.7x faster, §8) and, from #307, the full-tuple de-dup correctness fix.

Maintenance window — the only part that can cause an outage

CopilotDroppedAuditFields builds IX_..._action_id and IX_..._list_item_unique_id_id on copilot_event_accessed_resources, one of the larger tables on a Copilot-heavy tenant. Measured at synthetic scale (offline build, buffer pool dropped, medians of 3) on a 3,000,000-row junction table:

Index	Build time	Size
IX_..._action_id	2.5 s	40.7 MB
IX_..._list_item_unique_id_id	3.1 s	40.7 MB
pair	5.6 s	81 MB

The build attempts ONLINE on Enterprise (EngineEdition 3), Azure SQL DB (5) and Managed Instance (8), falling back to offline elsewhere. On Standard/Express/Web the build is offline and locks the table, so those tenants should upgrade in a maintenance window with the importer stopped.

copilot_event_accessed_resources is now touched twice — the FK index pair above, and the dedup index rebuild in 202608210700001_WidenCopilotAccessedResourceDedupIndex. The other five migrations create new/empty tables and need no window.

Rows in copilot_event_accessed_resources	FK pair	Dedup rebuild	Combined (linear)	Plan for
1M	~2 s	~3 s	~5 s	~5 s
10M	~19 s	~30 s	~50 s	1–2 min
100M	~3 min	~5 min	~8 min	15–25 min

The "plan for" column is deliberately more pessimistic than the arithmetic. The per-million figures are linear extrapolations from a 2M-row measurement, and index builds are O(n log n) with a sort that will spill to tempdb well before 100M rows — so budget 2–3x the linear estimate and make sure tempdb has room.

Storage. The FK pair adds ~27 MB per million rows. The widened dedup index is 83 MB per 2M rows gross, but it replaces the existing 6-key index (~69 MB), so the net addition is only ~7 MB per million. At 100M rows: ~3.4 GB net. Two things that figure does not cover, and which an admin must have free anyway:

• Peak space during the rebuild — old index, new index and sort space coexist. Budget roughly +4 GB transient at 100M rows.
• Transaction log growth — in FULL recovery a large index build generates multi-GB of log. Check log space and backup cadence before starting.

Apply the database migrations before or with the binaries — never run the new importer against a pre-migration schema. The merge SQL is an embedded resource in the importer and the bounded de-dup read is C#, but both of their supporting indexes arrive via migration. Because the manual-SQL path deliberately allows schema and binaries to be applied separately, "new binaries, old schema" is a reachable state — and by this release's own measurements it is slower than the version being upgraded from: the full-tuple merge against the un-widened index measured 530 ms vs 85 ms (5.6x slower, seek lost), and the bounded de-dup read without its index measured 337 ms vs 45 ms (7.5x slower, seek lost). Both recover completely once the migrations are applied.

2.1 Three snapshot defects found and fixed before merge

#260, #279 and #291 were each authored before the migrations that ended up preceding them, so their .resx model snapshots predated the branch they merged into. That is issue #271 exactly, and it is why #282 exists.

Migration	Target before	after	Entity sets
AddCopilotUsageReports	53,548	55,764	128 → 133
DeprecateTeamsAddons	52,488	54,332	127 → 130
AddCopilotInteractionHistory	stale (pre-#260/#279 model)	regenerated	130 → 140

Reproduced before fixing, on a scratch merge with the LocalDB test database dropped and the solution rebuilt:

Tree	Migration tests
dev alone	27 passed
dev + #260 as authored	18 failed, 8 passed — There is already an object named 'copilot_event_accessed_resource_actions' in the database.
dev + #260 repaired	27 passed

#291's stale snapshot failed differently and more loudly — AutomaticMigrationDataLossException at context construction. Its regenerated snapshot was verified by decompressing both and diffing entity sets: the delta is exactly the 10 interaction-history entities, with nothing removed (no Teams add-on entity resurrected, no usage-report entity dropped).

#291's migration was also re-timestamped from 202608191347242 to 202608200600001, because the original sorted before the already-merged 202608191533567_DeprecateTeamsAddons — i.e. it would have inserted itself into the middle of applied history.

2.2 The manual scripts were stale too — reviewer hotspot

The least obvious part of the release.

.manual.sql scripts stamp dbo.__MigrationHistory with an embedded copy of the model snapshot. Re-scaffolding the .resx left those embedded blobs pointing at the old model, so a DBA upgrading by hand would have stamped an out-of-date model and EF would then have generated an automatic migration at runtime — reintroducing the same failure on hand-upgraded databases only. The installer path would never have shown it.

Script	Embedded model	Expected	Status
CopilotDroppedAuditFields	80,778 hex	80,778	already correct
AddCopilotUsageReports	53,548 base64	55,764	regenerated
DeprecateTeamsAddons	78,730 hex	81,496	regenerated
AddCopilotInteractionHistory	stale	44,289 bytes	regenerated, verified to round-trip to identical base64

The regenerated blobs were executed against SQL Server and confirmed to assemble to the correct DATALENGTH (41,823 and 40,748 bytes). Two prerequisites were also corrected: AddCopilotUsageReports' (#284) and AddCopilotInteractionHistory' (from IndexReportDateQueries to DeprecateTeamsAddons).

---

3. Performance KPIs

Required by the repo's "prove every schema change improves performance before it is approved for stable" rule.

Benchmarked the hot path of the new import: the per-batch existence lookup in CopilotUsageUserDetailLoader.SaveBatchAsync, which reads back already-stored rows so unchanged rows are not rewritten. It runs once per 1,000-user batch — 200 times per import at the 200,000-user baseline.

Synthetic scale: 200,000 users × 15 daily report refreshes × 2 periods (D7 + D28) = 6,000,000 rows. Scattered user ids, since report order and users.id order are uncorrelated on a real tenant. Medians of 6 runs, first discarded, OPTION (RECOMPILE). SELECT * because the loader materialises full entities. All data synthetic — no customer database was involved.

Read path — IX_date_user_id_report_period_days

Window	Before (no index)	After (index)
realistic (1 date, 1 period)	120,447 reads / 628 ms / Clustered Index Scan	6,142 reads / 15 ms / Index Seek	20× fewer reads, 42× faster
narrow (1 date, 2 periods)	120,447 / 2,119 ms / scan	20,618 / 165 ms / seek	5.8× fewer reads
wide (15 dates, 2 periods)	120,447 / 2,790 ms / scan	252,397 / 5,643 ms / seek	⚠️ 2× slower

The realistic row is the one that matters: one Graph report call covers a single period and returns a single reportRefreshDate, so reportDates and periods each hold exactly one value per batch.

Projected per import at 200,000 users (200 batches): ~126 s → ~3 s, and ~24.1M → ~1.2M logical reads.

The wide-window regression is real and deliberately reported. At 30,000 matched rows the key lookups (the index does not cover the ~23 columns EF materialises) cost more than a scan. This loader never issues that shape, but anything that later widens the date range — a backfill importing many refresh dates at once — would hit it and would need a covering index. Recorded here so that is a conscious decision rather than a surprise.

Write path

Variant	20,000 rows	rows/sec	Projected 200,000
With index	328 ms	60,975	3.3 s
Without index	250 ms	80,000	2.5 s

The index costs ~0.8 s of write time per full import and saves ~123 s of read time. It is also the natural-key uniqueness constraint, so it has to exist regardless.

Index build: 6.3 s on 6M rows, 151.2 MB. The table starts empty on every existing customer, so the build at upgrade time is instant.

These are SQL-side figures; they exclude EF and network round-trip overhead. The delta attributable to the index is what is measured, and that is what the decision rests on.

---

4. Upgrade and combined verification

Tests.UnitTests/UpgradeFromStableTests.cs rehearses the upgrade an existing customer actually performs, rather than starting from an empty database:

1. Build a database at main's newest migration (202608131055001_IndexReportDateQueries).
2. Seed it — including a Greek URL (Καλημέρα κόσμε) to catch Unicode truncation.
3. Upgrade to the newest migration on dev.
4. Assert row counts survive, the Greek URL is byte-intact, and the new Copilot tables exist.

Both branches of DeprecateTeamsAddons' conditional drop are covered, which an empty-database test can never reach:

Case	Result
Teams add-on tables hold data	Tables retained, row counts unchanged ✅
Teams add-on tables empty	Tables dropped ✅

#291 and #292 had never been tested against each other, so the merged tree was verified directly at e93630b: full solution builds (Debug + Release), and 94/94 pass across the migration-pipeline, migration-cleanup, interaction-history and adoption suites against a dropped-and-replayed LocalDB — so the whole chain reaches 202608200600001 with no snapshot mismatch. Their only file overlap was Entities.csproj and Tests.UnitTests.csproj; git merged both cleanly and the build confirms it. All dev CI is green at that SHA.

---

5. CI changes

#280 fixes two independent defects that meant required checks frequently never reported, leaving PRs at BLOCKED and mergeable only by admin bypass:

• pull_request listened only for ready_for_review, which a PR opened non-draft never emits.
• A workflow-level paths: filter meant a PR touching only src/TelemetryService/** never started the workflow at all — and a workflow that never starts reports nothing.

The gate is applied per step, not as a job-level if:, because GitHub evaluates a job-level if: before expanding the matrix: a skipped test_dotnet reports as plain test_dotnet and the required test_dotnet (Release) context never appears. Measured on #280 itself.

test_aitracker was also gated on src/AnalyticsEngine/** rather than the tracker's own src/SPO/AITracker/**, so its tests never ran when the tracker changed.

#282 adds the migration snapshot guard that would have caught all three defects in §2.1 on the pull request. #289 adds a weekly read-only az deployment group what-if drift check (maintainer-side only; deploys nothing).

---

6. New user-visible and privacy surfaces

Two things in this release are not bug fixes and need their own treatment in the customer-facing release notes:

#291 — Copilot AI interaction history (opt-in). This is the first feature that sends customer Copilot prompt text out of the tenant, to Azure AI Language, and only when cognitive services are configured. No prompt or response text is persisted — bodies are reduced to counts and discarded, no column can hold one, response bodies are kept out of logs, and Copilot responses are never scored. But "nothing is stored" is not "nothing leaves the tenant", and the notes must say so plainly.

It is off by default and is capped per cycle — because the endpoint is one HTTP call per user with no tenant-wide or delta form, so an unscoped run at the 200k-user baseline would be 200k Graph calls per cycle. It also needs AiEnterpriseInteraction.Read.All (application, no delegated form), which the installer does not grant: missing admin consent is the most likely reason for this import to silently do nothing.

#292 — Copilot licence adoption tool. New admin API (CopilotAdoptionAPIController) and admin-app pages for finding unused seats and unlicensed heavy users. Read-only and additive: no EF model change, no migration, no config-schema change; it queries existing tables via a context factory.

#288 also fixed a live authorisation defect on the maintainer telemetry service: the Telemetry.Read scope was never enforced, because [RequiredScope] is inert metadata whose handler was never registered, and [Authorize(Roles = …)] replaces the default policy rather than extending it. A token with the correct role but the wrong scope returned 200 before, 403 after. Exposure was limited — the app role is assignment-gated and EasyAuth validates audience and allowedApplications — so this was a missing defence-in-depth layer, not an open door. It affects the maintainer dashboard only, not customer deployments.

---

7. Risks and reviewer hotspots

1. The manual-script model blobs (§2.2). Highest-value thing to check. Verify by reconstructing an embedded blob and comparing it to the .resx Target.
2. The offline index builds on non-Enterprise editions (§2). The only part of the release that can take a visible outage on a large tenant. copilot_event_accessed_resources is rebuilt twice (FK index pair, then the dedup index), and the other five migrations are instant — so it is easy to under-budget the window. See the sizing table and the storage/log notes in §2.
3. Upgrade ordering (§2). New binaries against a pre-migration schema is measurably slower than the release being upgraded from, because the merge SQL and the bounded de-dup read both assume indexes that arrive by migration. This only bites sites that apply schema by hand, which is exactly the audience the manual scripts serve.
4. A failing daily usage report now aborts that cycle's activity-report phase (Release critique follow-ups: the same #285 hole in the legacy daily reports, a fail-closed dedup guard, and three doc errors #309). The SharePoint/Teams/OneDrive/Yammer/Exchange daily loaders previously swallowed a 403 or 5xx and recorded an empty day, after which the phase stamped itself complete for 24 hours — a broken import looked healthy and idle, exactly the Graph paging failures are recorded as successful empty imports #285 symptom on an older code path. They now use strict paging, so the failure propagates and the phase is retried next cycle instead of going quiet. This restores the contract the phase is already written around (it deletes its "last imported" timestamp before loading and only re-saves it once every loader completes), but it is a genuine behaviour change on a long-standing path: one failing report type now costs that cycle's other reports, which are picked up on the next run. Admin-visible effect: a misconfigured tenant will now surface real errors in Application Insights where it previously showed a clean, empty import. That is the point, but it should be flagged so a burst of new errors after upgrading is read correctly. Two related admin-visible consequences:
   • While any one usage report is failing, the once-a-day throttle stays disarmed, so the phase re-runs every cycle and re-downloads the full window for all reports — roughly 10x the normal Graph call volume until the underlying error is fixed. No data is lost and no other import is affected; it is efficiency only. Tracked as Activity reports: a single failing usage report disables the finalized-date skip list, causing a full re-download every cycle #311.
   • A tenant with a permanently broken report will stop emitting the "usage reports finished" telemetry event, so any "hasn't finished in X hours" monitoring will start firing. That is the fix working as intended, not a new fault.
5. The accessed-resource de-dup identity changed (Copilot RC fixes: Graph paging failures, accessed-resource dedup correctness, and two bounded reads (#285, #287, #294, #296) #307). Rows are now keyed on the full seven-column tuple instead of five. Two consequences worth watching: a resource accessed with more than one action now produces one junction row per action rather than one in total, and on re-import the full-tuple NOT EXISTS will back-fill rows that were previously collapsed away — so copilot_event_accessed_resources row counts will step up after the upgrade. That is the fix working, not double-counting, but it should be said out loud in the admin notes. The growth is unquantified: the original justification for collapsing ("Action is Read for every access, so nothing real is lost") was removed rather than tested, and the benchmark holds the resolved-row count constant, so it measures cost per row and not the change in row count.
6. CONFIG_VERSION 2.1.0, with a breaking step at 2.0.0. Confirm the removal of GraphUserApps degrades gracefully for existing saved configs.
7. DeprecateTeamsAddons is conditional. A tenant with add-on data keeps the tables and the reporting views; a fresh install loses them. Two shapes in the field from one migration.
8. The wide-window index regression (§3). Not hit today; would be hit by a multi-date backfill.
9. Copilot: import AI interaction history from Graph (opt-in, statistics only) #291 is a new egress path for prompt text (§6). Worth an explicit sign-off rather than being read as just another importer.
10. test_dotnet (Release) now runs on every PR and does nothing (~10 s) when src/AnalyticsEngine/** is untouched. Deliberate: the alternative is unenforceable required checks.

---

8. Multi-model review outcome

This release has been through two independent multi-model reviews.

Round 1 — the diff as originally opened

Three models (Claude Opus 4.8, GPT-5.6 Sol, Gemini 3.1 Pro) reviewed it independently. Every finding was verified against the code; one did not survive that check.

Fixed and included here (#284):

• AddCopilotUsageReports.manual.sql named the wrong prerequisite, letting a hand-upgrading DBA skip a migration and still pass the gate.
• CopilotUsageUserDetailLoader dropped unkeyable rows with an O(N²) RemoveAt loop.

Rejected: a reported .ToLowerInvariant() rule violation in SentEmailImporter — those values are persisted, not just used as set keys, so removing them would change stored data.

Round 2 — #288, #289, #291 and #292

Those four were added after round 1, so they were reviewed separately by four models (Claude Opus 4.8, GPT-5.6 Sol, Gemini 3.1 Pro, Grok 4.6), alongside a synthetic-scale benchmark of the interaction-history dedup path. Two findings did not survive verification and were dropped.

It found four release blockers, all fixed in #293, now merged into dev:

Blocker	Effect
Adoption report snapshot selected by date only	copilot_usage_user_activity_log is keyed (date, user_id, report_period_days), so every licensed user was duplicated per stored period — adoption figures inflated up to 4x, able to exceed the licensed population
Drift check published raw config values	--result-format FullResourcePayloads into a world-readable job summary on a public repo — would have printed the Application Insights connection string
Interaction-history cadence gate stamped on failure	Enabling the feature before AiEnterpriseInteraction.Read.All consent silently did nothing for 24h
Transient Graph errors fed the 72h unlicensed back-off	Two throttling blips parked the entire active pilot group for three days

Plus seven correctness/scale fixes, including a dedup re-key that fixes a cross-user data drop and measures 1.8x fewer logical reads / 5.7x faster, and $orderby=createdDateTime asc (flagged independently by three of the four models) without which a truncated first-run backfill can permanently lose history.

The benchmark also supplied the measurement #291 shipped without: its (session_id, graph_interaction_id) index takes the dedup from 48,328 logical reads to 1,323 (seek, not scan), so the shipped index shape is proven correct and needs no change.

Four further findings were filed with their measurements rather than fixed in #293. Two have since been fixed and are in this release — #295 (adoption trend double scan, fixed in #303) and #297 (pilot-scope call brake, fixed in #306). Two remain deferred: #294 and #296 — both are now fixed and in this release via #307, each carrying the measured before/after the gate requires (see the gate classification table in §2).

Round 3 — the easy-issue sweep

After round 2, the remaining open issues were triaged for what could be safely completed for this release. Four were genuinely low-risk and are now merged: #286 (#304), #295 (#303), #297 (#306) and #302 (#305).

Deliberately not taken, despite looking easy:

• Review the profiling weekly-boundary change to usp_UpsertCopilot (reverted from the pending release) #300 / profiling SQL — excluded from this release by explicit instruction. Revert the profiling weekly-boundary change out of this release (tracked in #300) #301 reverts the weekly-boundary change; the [Ignore]d test proving the bug is real is kept as the repro.
• Copilot interaction history: dedup re-reads whole session history every cycle (needs a bounded window + index) #294, copilot_interaction_keywords has no index leading on keyword_id; orphan cleanup scans the link table #296 — now fixed and in this release via Copilot RC fixes: Graph paging failures, accessed-resource dedup correctness, and two bounded reads (#285, #287, #294, #296) #307. The gate did its job rather than blocking indefinitely: the naive fix for Copilot interaction history: dedup re-reads whole session history every cycle (needs a bounded window + index) #294 (bound the read, add no index) was measured as clearly worse, because created_utc is not in the existing unique index; the shipped fix bounds the read and adds the supporting index, which is what turns it into a 7.5x improvement. Two separate measurements of that naive fix exist on different synthetic datasets — "4,713 → 16,687 reads (3.5x)" here, and "5,484 → 15,555 (2.8x)" in the migration doc. Different data, same conclusion; neither is wrong.

---

9. Known issues shipping with this release

Tracked, not blocking, but they should appear in the customer-facing notes:

• Graph paging failures are recorded as successful empty imports #285 — Graph paging failures recorded as successful empty imports. Fixed in Copilot RC fixes: Graph paging failures, accessed-resource dedup correctness, and two bounded reads (#285, #287, #294, #296) #307 for the Copilot reports, and in Release critique follow-ups: the same #285 hole in the legacy daily reports, a fail-closed dedup guard, and three doc errors #309 for the legacy daily usage reports (SharePoint/Teams/OneDrive/Yammer/Exchange), which turned out to have the identical hole. A 403 now fails the report import with the HTTP status on the Health page, instead of reading as "no Copilot licences" and suppressing retry for 24h. A genuinely empty report is still a clean success, and a 404 (report absent outside the global cloud) is tolerated with the reason recorded.
• Copilot accessed-resource actions are collapsed by independent MIN(), and pairings can be fabricated #287 — accessed-resource actions collapsed by independent MIN(). Fixed in Copilot RC fixes: Graph paging failures, accessed-resource dedup correctness, and two bounded reads (#285, #287, #294, #296) #307. De-duplication now keys on the full seven-column tuple, so distinct actions survive and action/list-item pairings can no longer be fabricated.
• Review the profiling weekly-boundary change to usp_UpsertCopilot (reverted from the pending release) #300 — a Copilot chat at 23:59 on a Sunday is dropped from its own profiling week. The fix was reverted out of this release (Revert the profiling weekly-boundary change out of this release (tracked in #300) #301) to keep profiling SQL untouched; an [Ignore]d test in the repo reproduces it. Investigated since: the half-open window is correct and the original predicate does lose Sunday after midnight, so this is a genuine bug awaiting its own PR.

---

10. Deferred

• Copilot: Power BI adoption funnel and licence-waste view on the Graph Copilot usage report tables #261 (Power BI adoption funnel / licence-waste view on the usage-report tables) — reporting follow-up, not in this release.
• Copilot interaction history: dedup re-reads whole session history every cycle (needs a bounded window + index) #294 and copilot_interaction_keywords has no index leading on keyword_id; orphan cleanup scans the link table #296 — now included via Copilot RC fixes: Graph paging failures, accessed-resource dedup correctness, and two bounded reads (#285, #287, #294, #296) #307, each with the measured before/after the gate requires (see the classification table in §2). Copilot interaction history: dedup re-reads whole session history every cycle (needs a bounded window + index) #294's earlier "obvious fix" measured 3.5x worse; the shipped version bounds the read and adds the supporting index, which is what makes it 7.5x faster instead. The gate did its job.
• Review the profiling weekly-boundary change to usp_UpsertCopilot (reverted from the pending release) #300 — profiling weekly-boundary fix, reverted out of this release by instruction (Revert the profiling weekly-boundary change out of this release (tracked in #300) #301).
• ci.yml still omits src/SPO/AITracker/** from its paths:, so a tracker-only change to main produces no release build. Worth a follow-up issue.
• Copilot: import AI interaction history from Graph (opt-in, statistics only) #291 ships with response latency paired only within an import batch, and no distributed lock on its cadence gate (single web-job instance assumed).

---

11. On merge

• Fourteen issues close: CI: PRs opened non-draft never run the required checks #270, CI: detect EF migration snapshot conflicts between PRs before they reach dev #271, Detect Azure configuration drift between the Bicep templates and deployed resources #272, Add integration tests for the telemetry service using WebApplicationFactory #276, Deprecate Teams add-on tracking (teams_addons*) and disable the import #278, copilot_usage_user_activity_log has no retention bound #286, Copilot: import AI interaction history from Graph (opt-in, statistics only) #290, Copilot adoption: weekly trend scans the same six-month range twice #295, Copilot interaction history: wildcard pilot scope and group-discovery cap weaken the call brake #297, Installer fetches the latest release unauthenticated, so GitHub rate limiting breaks CI and can break real installs #302 (already closed by hand once their fixes reached dev) and Graph paging failures are recorded as successful empty imports #285, Copilot accessed-resource actions are collapsed by independent MIN(), and pairings can be fabricated #287, Copilot interaction history: dedup re-reads whole session history every cycle (needs a bounded window + index) #294, copilot_interaction_keywords has no index leading on keyword_id; orphan cleanup scans the link table #296 from Copilot RC fixes: Graph paging failures, accessed-resource dedup correctness, and two bounded reads (#285, #287, #294, #296) #307. main is the default branch, so the keywords fire on this merge.
• Two milestones become empty and should be closed: Build & release hygiene (CI: PRs opened non-draft never run the required checks #270 + CI: detect EF migration snapshot conflicts between PRs before they reach dev #271) and Telemetry service hardening (Detect Azure configuration drift between the Bicep templates and deployed resources #272 + Add integration tests for the telemetry service using WebApplicationFactory #276).
• Attach all seven .manual.sql scripts to the generated stable release — ci.yml uploads **/*.zip only, so they are never attached automatically and their absence is invisible until a DBA needs one. They form a strict prerequisite chain and must be run in migration-id order; the predecessor of the first is 202608131055001_IndexReportDateQueries. Each hard-fails with RAISERROR severity 16 if its predecessor is not stamped in __MigrationHistory.
  • 202608190622001_CopilotDroppedAuditFields.manual.sql
  • 202608190725064_AddCopilotUsageReports.manual.sql
  • 202608191533567_DeprecateTeamsAddons.manual.sql
  • 202608200600001_AddCopilotInteractionHistory.manual.sql
  • 202608210700001_WidenCopilotAccessedResourceDedupIndex.manual.sql
  • 202608210700002_IndexCopilotInteractionKeywordsByKeyword.manual.sql
  • 202608210700003_IndexCopilotInteractionsDedupWindow.manual.sql
• Rewrite the generated release notes to the admin-facing level-300 format.
• Admin-facing note required for Add a retention bound to the Copilot usage-report tables (#286) #304: tenants already running the Copilot usage-report import have been accumulating rows with no retention bound. The first run of the age-based purge after upgrading may delete a large number of rows, so it is worth scheduling in a maintenance window on a large tenant.

---

Added since this PR was last updated

Two further changes merged into dev.

Copilot Adoption: explainability, parity, correctness and scale (#315)

Follow-up to the original tool (#292). The user-visible additions are info tips on every assertion, an agents inventory with Keep/Review/Retire/New verdicts, an unlicensed-usage view, executive visuals and a 12-sheet Excel export with live charts.

The part that matters for a release decision is the scale work. Four hand-written queries asked for several COUNT(DISTINCT …) in one GROUP BY. SQL Server streams a single distinct aggregate cheaply; two or more force a spool. Measured on a synthetic 200,000-user / 12M-interaction tenant, all four exceeded the 90-second command timeout — and the failure mode is a warning on the page rather than an error, so on a large tenant the report was not slow, it was quietly broken, and nothing in production reported it.

Query	Elapsed before	after
LicensedUsersSql 28d	281 s	73 s
LicensedUsersSql 365d	303 s	135 s
AgentUsageSql	128 s	23 s
UnlicensedUsageRowsSql	131 s	54 s
WeeklyAdoptionTrendSql	315 s	38 s

No schema change, no migration, no new index — every fix is a change to how the query was written. Each rewrite was gated by an old-vs-new row-for-row comparison before performance was considered (33,999 / 200 / 33,999 / 27 rows, zero differing).

Two honest caveats, both carried over from #315 rather than hidden:

• LicensedUsersSql at a 365-day window is still 135 s at that scale. No longer failing, but not comfortable.
• Two of the rewrites trade more logical reads for much less elapsed time. Normally the wrong way round; shipped because the alternative returned no report at all.

Also in #315: several correctness fixes (capped-denominator rates that capped a 200k tenant at 25% adoption; an agent KPI inflated ~4×; a "last N days" window that spanned N+1 days; an agent user count inflated by unattributed audit events), a new CopilotAdoptionAnalysis App Insights event with per-step durations as measurements, and a terminology pass from "seat" to "licence" in everything a user reads.

Installer: handle new-tenant Graph Reports readiness (#317)

A newly created tenant's Microsoft 365 reporting backend may not have onboarded yet. In that state every /reports/get* endpoint returns HTTP 404 carrying the nested Graph code UnknownTenantId, while all other Graph endpoints are healthy. The installer now detects that condition and reports it, instead of the admin seeing a successful install whose usage reports silently return nothing.

Related, NOT in this release: the importer-side half of the same problem (the activity-report phase stamping itself complete after recording consecutive empty days) is still outstanding on branch sambetts/graph-report-404-silent-failure. Release notes must not imply the silent-failure path is fully closed.

Migrations in this release

Seven migrations. All are additive or index-only — new tables and nullable columns for the Copilot usage reports, interaction history and dropped audit fields, plus three index changes:

• CoverCopilotAccessedResourceDedup, WidenCopilotAccessedResourceDedupIndex — measured, documented in their doc comments
• IndexCopilotInteractionKeywordsByKeyword, IndexCopilotInteractionsDedupWindow — on tables new in this release, so instant

Per the release policy each needs its <migrationid>.manual.sql attached to the stable GitHub release — CI does not upload these, so they must be added by hand, in migration-id order.

[sambetts]

Multi-model review of this release

Three models reviewed the full main...dev diff independently, with the same prompt and the repo's own rules (200k-user baseline, EF6 automatic-migration risk, Unicode/nvarchar, migration safety, public-repo data hygiene):

Model	Verdict
Claude Opus 4.8	Ship
GPT-5.6 Sol	Don't ship
Gemini 3.1 Pro	Don't ship

I verified every finding against the code rather than relaying it. Two did not survive that check. The table below is my adjudication, not a merge of their opinions.

#	Finding	Found by	Verified?	Status
1	Graph failures recorded as successful empty imports	GPT	✅ confirmed	Open — highest value
2	Accessed-resource actions collapsed by independent MIN()	GPT	✅ confirmed	Open
3	New usage table missing from retention cleanup	GPT	✅ confirmed	Open
4	Manual script named the wrong prerequisite	Opus + GPT	✅ confirmed	✅ Fixed in #284
5	RemoveAt in reverse loop is O(N²)	Gemini + GPT	✅ confirmed	✅ Fixed in #284
6	Unbounded per-user Graph calls under a group filter	Gemini + GPT	✅ confirmed	Open — pre-existing pattern
7	User-data lookup omits the new workload	GPT	✅ confirmed	Open
8	.ToLowerInvariant() are redundant allocations	Gemini	❌ wrong	No action

---

1. Graph failures are recorded as successful empty imports — open

The most valuable finding of the exercise, and no amount of unit testing would have surfaced it.

PageableGraphLoaderExtensions treats any non-gateway-timeout HttpRequestException as end-of-paging:

logger.LogWarning($"Unexpected HTTP error on page {pageCount}: {ex.Message}. " +
    "Will not retry page & returning results upto current page.");
nextUrl = null;

It returns the rows accumulated so far and does not throw. For the Copilot reports that means a 403 from a missing Reports.Read.All grant returns zero rows with no exception. CopilotUsageUserDetailLoader then takes the parsed.Count == 0 branch, logs "Expected on a tenant with no Microsoft 365 Copilot licences", writes an import log with no error, and returns 0. The cadence gate records success, so it will not retry for 24 hours.

The existing shape-change guard does not catch it either, because it is conditioned on reports.Count > 0:

if (reports.Count > 0 && parsed.Count == 0) { throw ... }

So a permissions misconfiguration, a 5xx mid-paging, or an exhausted 429 all present to an admin as "you have no Copilot licences". That is exactly the misleading-symptom class this project's release notes are supposed to call out.

Suggested fix: a strict paging mode that rethrows HTTP failures, used by all three Copilot reports, with the deliberate 404/non-global-cloud case handled separately and recorded in CopilotUsageReportImportLog.Error.

2. Accessed-resource actions are collapsed, and pairings can be fabricated — open

common_upsert_copilot_agents.sql:302-312 groups on the original five-column resource tuple and then takes two independent minima:

MIN(raction.id)    AS action_id,
MIN(rlistitem.id)  AS list_item_unique_id_id

Two consequences:

1. Two accesses of the same resource with different actions (Read, Write) collapse into one row, keeping only the lower id. That directly undercuts Copilot: persist parsed-but-dropped audit fields #262's purpose, which is to stop dropping these fields.
2. Because the two MIN()s are evaluated independently, the surviving row can pair an action_id from one source row with a list_item_unique_id_id from another — a combination that never appeared in the payload.

The in-code comment justifies this by keeping the dedup identity unchanged so IX_..._dedup still covers the anti-join. That is a real and well-reasoned constraint. But note the stated 1700-byte index-limit rationale does not apply here: both columns are int foreign keys, so adding them to the key costs ~8 bytes, not customer-text width.

Not a crash and not loss of pre-existing data, so defensible for a test release — but it should not reach stable without either widening the tuple or an explicit decision that the fidelity loss is acceptable.

3. copilot_usage_user_activity_log has no retention bound — open

Clean Old Data Data.sql prunes every other usage-report table (onedrive_user_activity_log, outlook_user_activity_log, sharepoint_user_activity_log, teams_user_activity_log, the Yammer set, and the Teams add-on tables under OBJECT_ID guards). The new table is absent.

At the 200k-user baseline that is up to 200,000 rows per report refresh per period, with nothing ever removing them. The table this release removes (teams_addons_user_installed_log) was deprecated in #279 precisely for becoming the largest table in the product — so it would be unfortunate to introduce another unbounded one in the same release.

Fix: a guarded batched delete on the indexed date column, alongside the others.

4 and 5 — fixed in #284

Both confirmed and already addressed:

• AddCopilotUsageReports.manual.sql gated on 202608131055001_IndexReportDateQueries rather than its immediate predecessor 202608190622001_CopilotDroppedAuditFields, so a DBA upgrading by hand could skip a migration and still pass the check. Installer/EF path unaffected (it applies in id order).
• CopilotUsageUserDetailLoader dropped unkeyable rows with RemoveAt in a reverse loop — O(N²) element moves when removals are scattered. Now RemoveAll.

On severity for #5: both models overstated it slightly. The quadratic case needs request.PeriodDays to be null (period ALL) and a mix of keyable and unkeyable rows. On a normal D7/D28 import nothing is removed at all and the old loop was O(N). Fixed anyway — it is free.

6. Unbounded per-user Graph calls under a group filter — open, pre-existing

FilterToUsersInScope awaits one /users/{upn}/memberOf per report row, sequentially. Two models called this a blocker. I would not block on it, for three reasons: it only applies when a group filter is configured, it is the same pattern every other per-user usage-report loader uses, and the code already logs a loud warning naming the call count.

It is still the right thing to fix — resolve the filter groups once and intersect UPNs in memory — but it is a pre-existing architectural pattern, not a regression introduced here.

7. User-data lookup omits the new workload — open

UserDataLookupAPIController does not expose copilot_usage_user_activity_log, and drops the Teams app-install category even though #279 deliberately retains populated teams_addons_user_installed_log tables. Deletion remains correct (the new table cascades on user delete, and cleanup handles retained Teams rows dynamically); it is discovery that is incomplete. Worth fixing before stable given what that endpoint is for.

8. .ToLowerInvariant() — reported, but wrong

Gemini flagged SentEmailImporter.cs:728,743 as violating the "don't .ToLower() into an OrdinalIgnoreCase collection" rule. The collections are OrdinalIgnoreCase, so the report looks right at a glance — but the lowercased values are assigned back and persisted:

toAddr = toAddr.ToLowerInvariant();
recipientAddresses.Add(toAddr);   // -> stored
...
FromAddress = fromAddr,           // -> stored

They normalise stored email addresses; they are not merely set keys. Removing them would change what is written to the database. No action taken. Recording it because a plausible-looking rule violation that turns out to be load-bearing is exactly the kind of "fix" that causes an incident.

---

Where all three models agreed the code is clean

Worth stating, since it is most of the diff:

• All three migration .resx snapshots match their manual-script model blobs; entity/snapshot parity is clean.
• Migration DDL is guarded, idempotent and resumable; suppressTransaction: true used correctly; ONLINE attempts go through sp_executesql with an offline fallback and edition gating.
• The Teams deprecation uses EXISTS not COUNT(*), retains all three tables if any holds data, and only rewrites dependent views on the drop path.
• No added varchar anywhere; customer text is bounded nvarchar, including indexed nvarchar(850).
• The Copilot persistence path follows the repo's scale rules: GetRange batching, Contains chunked to 1,000, OrdinalIgnoreCase in-memory resolution with no SQL LOWER(), bounded EF change tracking.
• CONFIG_VERSION correctly records the additive 1.10.0 step and the breaking 2.0.0 removal; legacy GraphUserApps tokens are ignored rather than throwing.
• Public-data scan: only synthetic values (Contoso, zeroed GUIDs, well-known Azure role IDs, public service hostnames). No customer or tenant identifiers.
• Telemetry service auth, role/scope enforcement, signed upload, Cosmos caps and deploy gating all clean.

My recommendation

Ship as a test release, with findings 1, 2 and 3 tracked as issues and fixed before stable.

Reasoning: nothing here corrupts or loses pre-existing customer data, the upgrade path is verified on a populated database in both of DeprecateTeamsAddons' branches, and the migration chain is sound. Findings 2 and 3 are fidelity and growth concerns that a test tenant will tolerate. Finding 1 is the one to fix soonest — not because it breaks the import, but because it makes a broken import look like a healthy one, which is the hardest kind of problem to diagnose in the field.

If the test tenant shows zero Copilot usage rows after deployment, check Reports.Read.All before believing the "no licences" message.