From 07003dd962990c06b7bcdfd09005caf2f64bffb4 Mon Sep 17 00:00:00 2001 From: Deepak J V Date: Mon, 18 May 2026 16:35:31 +0530 Subject: [PATCH 1/2] Add Quality Metrics section and business metric updates in monitoring-plan - Add new Section 5.2 Quality Metrics consolidating QoS/success-rate, query counts, failure-reason classification, query execution time, and parallel-query telemetry - Bump Engineering Health Reports to Section 5.3 - Add 'Data volume processed (bytes of data scanned)', 'Average collaborators per collaboration', and 'Collaboration distribution by workload type' to Section 5.1 Business Metrics --- .../tsg/monitoring-plan.md | 18 +++++++++++++++++- 1 file changed, 17 insertions(+), 1 deletion(-) diff --git a/demos/analytics-using-managedcleanroom/tsg/monitoring-plan.md b/demos/analytics-using-managedcleanroom/tsg/monitoring-plan.md index 9343557..c77d3ed 100644 --- a/demos/analytics-using-managedcleanroom/tsg/monitoring-plan.md +++ b/demos/analytics-using-managedcleanroom/tsg/monitoring-plan.md @@ -419,14 +419,30 @@ CSS should use the [Customer TSG](customer/index.md) plus these additional tools | Active collaborations (by subscription, tenant, region) | Cosmos DB | Daily | TBD | | New collaborations created (count, trend) | RP Worker telemetry | Daily | TBD | | Collaborations deleted | RP Worker telemetry | Daily | TBD | +| Collaboration distribution by workload type | RP Worker telemetry | Daily | TBD | | Unique tenants/subscriptions using ACCR | ARM activity log | Weekly | TBD | | Query execution count | Frontend Service telemetry | Daily | TBD | | Data volume processed (rows read/written) | Frontend Service telemetry | Daily | TBD | +| Data volume processed (bytes of data scanned) | Frontend Service telemetry | Daily | TBD | +| Average collaborators per collaboration | Frontend Service telemetry / Membership Manager | Weekly | TBD | | Average collaboration lifetime | Cosmos DB + ARM | Weekly | TBD | | Regional distribution | RP Worker telemetry | Weekly | TBD | | Workload types enabled | RP Worker telemetry | Weekly | TBD | -### 5.2 Engineering Health Reports — Status: TBD +### 5.2 Quality Metrics — Status: TBD + +> **Note**: Some of these metrics may have been mentioned elsewhere in this document; this section consolidates all quality metrics under one head. + +| # | Metric | Description / Notes | Status | +|---|---|---|---| +| 1 | QoS / success rate of Cleanroom RP CRUD operations (including pause/resume collaboration) | Per-operation success/failure rate for all ARM CRUD ops (Create, Get, List, Update, Delete, Pause, Resume, EnableWorkload, AddCollaborator). Each failure tagged with client-side vs server-side classification, error code, and failure reason. | TBD | +| 2 | QoS / success rate of Analytics Frontend operations | Per-operation success/failure for all Frontend APIs (PublishDataset, PublishQuery, SubmitQuery, VoteOnProposal, AddCollaborator, DownloadOutput, ListAuditEvents, GetCollaboration, GetRun, ListRuns). Each failure tagged with client-side vs server-side classification, code and reason. Client-side examples: publishing a dataset or query without mandatory fields, invalid input, unauthorized identity. | TBD | +| 3 | #queries published / submitted / successfully run / failed to run at a collaboration level | Counts per collaboration. Dimensions: collaborationId, queryName, finalState (COMPLETED / FAILED / SUBMISSION_FAILED). | TBD | +| 4 | Query execution failure reason (per query) | Failure reason classification for each submitted query — client-side (syntax issues, SKR issue, storage or Key Vault access issue) and server-side (data loading issue, executor crash, OOM, etc.). | TBD | +| 5 | Query execution time and average query execution time at a collaboration level | Per-query duration plus p50/p95/p99 and average aggregated at the collaboration level. | TBD | +| 6 | #parallel queries executing in a collaboration | Concurrent in-flight query count per collaboration over time, derived from execution-commencement timestamps (SUBMITTED → COMPLETED interval). Query IDs of the in-flight queries should also be returned. | TBD | + +### 5.3 Engineering Health Reports — Status: TBD > **Note**: Engineering reports will be sourced from the Geneva → Kusto pipeline once operational. Exact queries and dashboards TBD. From bcb38fb46429f6591294cd58474711fd2eb1c8b3 Mon Sep 17 00:00:00 2001 From: Deepak J V Date: Mon, 18 May 2026 18:45:20 +0530 Subject: [PATCH 2/2] Add Business Need column to Quality Metrics table --- .../tsg/monitoring-plan.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/demos/analytics-using-managedcleanroom/tsg/monitoring-plan.md b/demos/analytics-using-managedcleanroom/tsg/monitoring-plan.md index c77d3ed..17fc3a3 100644 --- a/demos/analytics-using-managedcleanroom/tsg/monitoring-plan.md +++ b/demos/analytics-using-managedcleanroom/tsg/monitoring-plan.md @@ -433,14 +433,14 @@ CSS should use the [Customer TSG](customer/index.md) plus these additional tools > **Note**: Some of these metrics may have been mentioned elsewhere in this document; this section consolidates all quality metrics under one head. -| # | Metric | Description / Notes | Status | -|---|---|---|---| -| 1 | QoS / success rate of Cleanroom RP CRUD operations (including pause/resume collaboration) | Per-operation success/failure rate for all ARM CRUD ops (Create, Get, List, Update, Delete, Pause, Resume, EnableWorkload, AddCollaborator). Each failure tagged with client-side vs server-side classification, error code, and failure reason. | TBD | -| 2 | QoS / success rate of Analytics Frontend operations | Per-operation success/failure for all Frontend APIs (PublishDataset, PublishQuery, SubmitQuery, VoteOnProposal, AddCollaborator, DownloadOutput, ListAuditEvents, GetCollaboration, GetRun, ListRuns). Each failure tagged with client-side vs server-side classification, code and reason. Client-side examples: publishing a dataset or query without mandatory fields, invalid input, unauthorized identity. | TBD | -| 3 | #queries published / submitted / successfully run / failed to run at a collaboration level | Counts per collaboration. Dimensions: collaborationId, queryName, finalState (COMPLETED / FAILED / SUBMISSION_FAILED). | TBD | -| 4 | Query execution failure reason (per query) | Failure reason classification for each submitted query — client-side (syntax issues, SKR issue, storage or Key Vault access issue) and server-side (data loading issue, executor crash, OOM, etc.). | TBD | -| 5 | Query execution time and average query execution time at a collaboration level | Per-query duration plus p50/p95/p99 and average aggregated at the collaboration level. | TBD | -| 6 | #parallel queries executing in a collaboration | Concurrent in-flight query count per collaboration over time, derived from execution-commencement timestamps (SUBMITTED → COMPLETED interval). Query IDs of the in-flight queries should also be returned. | TBD | +| # | Metric | Business Need | Description / Notes | Status | +|---|---|---|---|---| +| 1 | QoS / success rate of Cleanroom RP CRUD operations (including pause/resume collaboration) | QoS measurements — track service health and reliability of the Cleanroom RP; drive SLO/SLA reporting and isolate client- vs server-side regressions. | Per-operation success/failure rate for all ARM CRUD ops (Create, Get, List, Update, Delete, Pause, Resume, EnableWorkload, AddCollaborator). Each failure tagged with client-side vs server-side classification, error code, and failure reason. | TBD | +| 2 | QoS / success rate of Analytics Frontend operations | QoS measurements — track service health and reliability of the Analytics Frontend; distinguish customer-induced errors (bad input) from platform errors. | Per-operation success/failure for all Frontend APIs (PublishDataset, PublishQuery, SubmitQuery, VoteOnProposal, AddCollaborator, DownloadOutput, ListAuditEvents, GetCollaboration, GetRun, ListRuns). Each failure tagged with client-side vs server-side classification, code and reason. Client-side examples: publishing a dataset or query without mandatory fields, invalid input, unauthorized identity. | TBD | +| 3 | #queries published / submitted / successfully run / failed to run at a collaboration level | Analyze the state of a collaboration — understand activity, throughput, and failure trends per collaboration to inform customer engagement and product improvements. | Counts per collaboration. Dimensions: collaborationId, queryName, finalState (COMPLETED / FAILED / SUBMISSION_FAILED). | TBD | +| 4 | Query execution failure reason (per query) | Identify and correct errors that are blocking customers — triage failures faster, attribute to the right component, and reduce mean-time-to-resolution. | Failure reason classification for each submitted query — client-side (syntax issues, SKR issue, storage or Key Vault access issue) and server-side (data loading issue, executor crash, OOM, etc.). | TBD | +| 5 | Query execution time and average query execution time at a collaboration level | Work on improvements based on observed performance — detect regressions, benchmark workloads, and guide query/cluster optimization. | Per-query duration plus p50/p95/p99 and average aggregated at the collaboration level. | TBD | +| 6 | #parallel queries executing in a collaboration | Verify whether parallel queries were executed as expected — validate scheduling/concurrency behavior, detect contention, and inform capacity planning. | Concurrent in-flight query count per collaboration over time, derived from execution-commencement timestamps (SUBMITTED → COMPLETED interval). Query IDs of the in-flight queries should also be returned. | TBD | ### 5.3 Engineering Health Reports — Status: TBD