Skip to content

[ci] [dashboard]: add explicit benchmark cohort comparison mode #1636

Description

@Satyam-53

Problem

Users sometimes need to compare performance across hardware or software environments.

Currently, selecting All cohorts displays an independent set of charts for every cohort. This creates significant visual clutter and makes direct comparison difficult because:

  • Cohorts are shown in separate charts.
  • Axis ranges may differ.
  • Colors do not consistently represent cohort identity.
  • Users cannot deliberately choose which cohorts to compare.
  • Incompatible benchmark environments may appear comparable.

Proposed solution

Add an explicit Compare cohorts mode.

Users should be able to select two cohorts, with an optional maximum of three. Each metric should then be displayed in a shared chart with one series per selected cohort.

Comparison should use the canonical cohort identity and readable labels introduced by the cohort-selector ticket.

Entry points

Compare mode should be accessible from:

  • The main dashboard controls.
  • The all-cohort overview.
  • Optionally, the detailed single-cohort view.

When entering from a selected cohort, that cohort should be preselected.

Cohort selection

  • Require at least two cohorts to render a comparison.
  • Allow no more than three cohorts.
  • Constrain available cohorts using the active Model, GPU, Source, date, and advanced filters.
  • Show readable configuration summaries as primary labels.
  • Keep hardware, software, and recipe IDs available as secondary details.
  • Explain why a cohort is unavailable or incompatible when possible.

Compatibility rules

The implementation must define when two cohorts are suitable for direct comparison.

At minimum, compatibility should consider:

  • Model or benchmark identity
  • Benchmark schema/version
  • Metric definitions and units
  • Benchmark recipe
  • Input shape or workload configuration
  • Any schedule or execution property that affects comparability

Hardware and software profiles may differ because comparing those profiles is a primary use case.

Legacy and v2 data should not be combined without an explicit warning. If their metric definitions are incompatible, the comparison should be rejected.

Compatibility rules should be centralized rather than implemented separately in individual chart components.

Chart behavior

  • Render one shared chart per metric.
  • Render one series per selected cohort.
  • Assign each cohort a stable color.
  • Use the same cohort color across all metrics.
  • Display a persistent legend with readable cohort labels.
  • Use a common axis for every series within a metric.
  • Include exact hardware, software, recipe, commit, source, and timestamp details in tooltips.
  • Represent missing observations accurately.
  • Do not interpolate across missing records in a misleading way.
  • Clearly indicate metrics that are unavailable for one or more selected cohorts.

URL state

Encode Compare mode and the selected cohort keys in the URL so comparisons can be bookmarked and shared.

The dashboard should handle:

  • Missing cohort keys
  • Duplicate cohort keys
  • Stale or unknown keys
  • More than the maximum number of cohorts
  • Cohorts excluded by another active filter
  • A comparison URL containing fewer than two valid cohorts

Dependency

Blocked by the ticket that introduces canonical cohort identity, readable cohort labels, and the Benchmark cohort selector.

Preferably follows the all-cohort overview ticket so Compare mode can reuse its cohort-selection interaction.

Shared multi-series chart infrastructure may be developed earlier, but final integration should use the canonical cohort and overview contracts.

Wait till #1634 and #1635 lands

Acceptance criteria

  • Compare mode can be enabled explicitly.
  • Users can select a minimum of two and a maximum of three cohorts.
  • The current cohort is preselected when entering from a detailed view.
  • Comparisons use one shared chart per metric.
  • Every selected cohort has a stable color across all charts.
  • Charts include a persistent, readable cohort legend.
  • Tooltips identify the exact cohort and benchmark environment.
  • Compatibility rules are defined and centralized.
  • Incompatible cohorts are rejected or accompanied by a clear warning.
  • Legacy and v2 comparisons are handled explicitly.
  • Missing records and metrics are represented accurately.
  • Compare mode and selected cohorts are encoded in the URL.
  • Invalid comparison URLs fall back safely.
  • Exiting Compare mode returns to a normal single-cohort or overview state.
  • The comparison UI is keyboard-accessible.
  • Tests cover cohort limits, compatibility, stable colors, URL round-tripping, missing data, and legacy behavior.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    platformPlatform-specific (Windows/macOS)scope: attentionAttention backends (VSA, STA, Flash, etc.)

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions