Problem
We have two Copilot data sources and neither can describe the shape of a conversation:
| Source |
What it gives us |
What it cannot tell us |
Audit.General feed (copilot_chats) |
Accessed resources, agent, app host, opaque message sizes |
Which prompt produced which response; how long Copilot took |
| Graph Copilot usage reports (#260) |
Per-user prompt counts, aggregated over 7/28/90/180 days |
Anything per-turn; anything intraday |
So we can say how much Copilot is being used, and what it touched, but nothing about turn structure, responsiveness, or what people were actually trying to do.
Proposal
Optional, opt-in import of Copilot interaction history via Microsoft Graph:
GET /copilot/users/{userId}/interactionHistory/getAllEnterpriseInteractions
The valuable field is requestId, which pairs a user prompt with the response it produced — that is what makes response latency, true turn counts and prompt-to-response ratios possible at all. sessionId is the same identifier the audit feed stores as copilot_chats.thread_id, so interaction shape joins back to the audit-derived accessed resources for the same conversation.
Privacy: no prompt or response text is stored
The Graph payload contains the user's literal prompt and Copilot's literal answer. That is the most sensitive data this product can see. The importer reads each body, derives statistics from it, and discards it — there is no column anywhere in this feature capable of holding a message body, and the raw response body is never written to logs either.
Stored per interaction: request_id, derived response_latency_ms, conversation type, app class, device, locale, body character/word counts, attachment/link/mention/context counts, interaction type (userPrompt / aiResponse) and created_utc.
Stored only when cognitive services are configured, and only for user prompts: sentiment score, detected language, and extracted key phrases. Copilot responses are never scored — they are model output rather than a signal about the user, and they would dominate the per-character cognitive bill.
Cost: this endpoint is one HTTP call per user
getAllEnterpriseInteractions has no tenant-wide form and no delta form. At this product's stated design target of ~200,000 users, an unscoped run would be 200,000 Graph calls per import cycle — a non-starter as an always-on import. Four independent brakes are therefore required:
- Feature toggle —
ImportTaskSettings.CopilotInteractionHistory, off by default.
- Group scope —
UserGroupsFilter must be set; the import refuses to run without it unless explicitly overridden. Scope is resolved group-first (list the pilot groups, page their members, intersect with users), because the obvious alternative — asking "is this user in the pilot group?" per user — costs one Graph call per tenant user just to decide who to import.
- Per-cycle ceiling — default 500 users, least-recently-run first, so a pilot group larger than the cap is still fully covered across consecutive cycles.
- Per-user back-off — users returning nothing twice running (almost always missing the
M365_COPILOT_BUSINESS_CHAT service plan) are skipped for 72h.
Plus per-user watermarks so a steady-state cycle only asks for interactions created since the last successful run, a daily cadence gate, and a 50-page cap per user per cycle.
Permission
AiEnterpriseInteraction.Read.All, application permission, with no delegated form. It is not part of the installer's base consent and needs explicit admin consent, so "the admin hasn't consented yet" will be the single most common reason for this import to do nothing. The importer should check the token's roles claim up front and log one clear warning rather than emitting a 403 per user per cycle.
Acceptance criteria
Out of scope
No reports or UI. This lands the import and the queryable tables only; the reporting surface is a follow-up.
Problem
We have two Copilot data sources and neither can describe the shape of a conversation:
Audit.Generalfeed (copilot_chats)So we can say how much Copilot is being used, and what it touched, but nothing about turn structure, responsiveness, or what people were actually trying to do.
Proposal
Optional, opt-in import of Copilot interaction history via Microsoft Graph:
The valuable field is
requestId, which pairs a user prompt with the response it produced — that is what makes response latency, true turn counts and prompt-to-response ratios possible at all.sessionIdis the same identifier the audit feed stores ascopilot_chats.thread_id, so interaction shape joins back to the audit-derived accessed resources for the same conversation.Privacy: no prompt or response text is stored
The Graph payload contains the user's literal prompt and Copilot's literal answer. That is the most sensitive data this product can see. The importer reads each body, derives statistics from it, and discards it — there is no column anywhere in this feature capable of holding a message body, and the raw response body is never written to logs either.
Stored per interaction:
request_id, derivedresponse_latency_ms, conversation type, app class, device, locale, body character/word counts, attachment/link/mention/context counts, interaction type (userPrompt/aiResponse) andcreated_utc.Stored only when cognitive services are configured, and only for user prompts: sentiment score, detected language, and extracted key phrases. Copilot responses are never scored — they are model output rather than a signal about the user, and they would dominate the per-character cognitive bill.
Cost: this endpoint is one HTTP call per user
getAllEnterpriseInteractionshas no tenant-wide form and no delta form. At this product's stated design target of ~200,000 users, an unscoped run would be 200,000 Graph calls per import cycle — a non-starter as an always-on import. Four independent brakes are therefore required:ImportTaskSettings.CopilotInteractionHistory, off by default.UserGroupsFiltermust be set; the import refuses to run without it unless explicitly overridden. Scope is resolved group-first (list the pilot groups, page their members, intersect withusers), because the obvious alternative — asking "is this user in the pilot group?" per user — costs one Graph call per tenant user just to decide who to import.M365_COPILOT_BUSINESS_CHATservice plan) are skipped for 72h.Plus per-user watermarks so a steady-state cycle only asks for interactions created since the last successful run, a daily cadence gate, and a 50-page cap per user per cycle.
Permission
AiEnterpriseInteraction.Read.All, application permission, with no delegated form. It is not part of the installer's base consent and needs explicit admin consent, so "the admin hasn't consented yet" will be the single most common reason for this import to do nothing. The importer should check the token'srolesclaim up front and log one clear warning rather than emitting a 403 per user per cycle.Acceptance criteria
gtfilter forces.Out of scope
No reports or UI. This lands the import and the queryable tables only; the reporting surface is a follow-up.