Expand on limitations of existing speech recognition events in Alternatives and refactor Timestamps to use Seconds (#205)

* Expand on limitations of existing speech recognition events

Added detailed explanations regarding the limitations of existing API surfaces for tracking latency in speech recognition in the Alternatives Considered section, including issues with `speechstart` and `speechend` events and the implications of modifying `event.timeStamp`.

* Refactor speech recognition timestamps to use seconds

Updated the speech recognition result timestamps to use seconds instead of milliseconds, improving consistency with other Web APIs. Added detailed explanations for the choice of time representation, proposed behavior, and security considerations. Based off of comments from #205

Author: alan33d

explainers/speech-recognition-result-timestamps.md

@@ -6,70 +6,103 @@
 
 The Web Speech API currently does not expose the start and end timestamps of the source audio corresponding to a given transcription result (`SpeechRecognitionResult`). This limitation creates two major challenges for API clients and end users:
 
-- **Timeline Association:** Developers cannot readily associate transcribed text with specific segments of the audio source, making it difficult to map generated captions to media timelines or audio tracks.
+- **Timeline Association:** Developers cannot readily associate transcribed text with specific segments of the audio source, making it difficult to map generated captions to media timelines, audio tracks, or video frames.
 - **Latency Tracking & Backend Failover:** With the adoption of on-device Automatic Speech Recognition (ASR) to improve privacy and reduce server costs, processing performance becomes heavily dependent on local client hardware resources. The Web Speech API acts as a "black box" regarding local processing delays. Developers cannot programmatically calculate transcription latency or detect when on-device models fall behind real-time. This leads to poor user experiences (e.g. caption lag during live video conferencing) and deprives applications of the signal needed to seamlessly fail over to high-performance cloud backends.
 
 ### Proposed Solution
 
-We propose extending the `SpeechRecognitionResult` interface to include optional (nullable) `audioStartTime` and `audioEndTime` attributes.
+We propose extending the `SpeechRecognitionResult` interface to include `audioStartTime` and `audioEndTime` attributes.
 
 #### Web IDL Definition
 
 ```webidl
 partial interface SpeechRecognitionResult {
-    // Start timestamp of the audio segment in milliseconds (relative to the start of the audio stream)
-    readonly attribute DOMHighResTimeStamp? audioStartTime;
+    // Start timestamp of the audio segment in seconds relative to the start of the audio stream (0.0s).
+    readonly attribute double audioStartTime;
 
-    // End timestamp of the audio segment in milliseconds (relative to the start of the audio stream)
-    readonly attribute DOMHighResTimeStamp? audioEndTime;
+    // End timestamp of the audio segment in seconds relative to the start of the audio stream.
+    readonly attribute double audioEndTime;
 };
 ```
 
+### Choice of Time Representation: Seconds as `double`
+
+The timestamps `audioStartTime` and `audioEndTime` are defined as `double` representing **seconds**, rather than `DOMHighResTimeStamp` (milliseconds). This design choice is based on the following considerations:
+
+1. **Consistency with Adjacent Web Audio & Media APIs:**
+   * In adjacent W3C media specifications, media-local stream timelines are universally represented in **seconds** as a `double`:
+     * **Web Audio API:** [`BaseAudioContext.currentTime`](https://webaudio.github.io/web-audio-api/#dom-baseaudiocontext-currenttime) (seconds)
+     * **HTML Media Elements:** [`HTMLMediaElement.currentTime`](https://html.spec.whatwg.org/multipage/media.html#dom-media-currenttime) (seconds)
+     * **AudioParam Scheduling:** [`AudioParam.setValueAtTime()`](https://webaudio.github.io/web-audio-api/#dom-audioparam-setvalueattime) (seconds)
+   * Using seconds ensures seamless interoperability when developers route audio between `<audio>`/`<video>` elements, Web Audio graphs, and `SpeechRecognition`, avoiding repetitive and error-prone unit conversions ($1000\times / \div 1000$).
+
+2. **Semantic Inaccuracy of `DOMHighResTimeStamp` for Media Streams:**
+   * Under the [W3C High Resolution Time Level 3](https://www.w3.org/TR/hr-time-3/#sec-domhighrestimestamp) specification, `DOMHighResTimeStamp` is strictly defined as a time coordinate in **milliseconds** measured relative to the global execution context's time origin (`performance.timeOrigin`).
+   * Because `audioStartTime` and `audioEndTime` represent a **media-local timeline** (elapsed time starting at $0.0\text{s}$ at the beginning of the audio stream) rather than document uptime, using `DOMHighResTimeStamp` would be semantically incorrect.
+
+3. **Numerical Precision:**
+   * A standard 64-bit IEEE 754 floating-point number (`double`) in seconds provides sub-nanosecond resolution across hours of continuous audio streaming, ensuring sample-accurate precision at any audio sample rate (e.g. 16 kHz to 96 kHz).
+
+4. **W3C TAG Design Principles Alignment:**
+   * This design adheres to the [W3C TAG Design Principles on Times and Dates](https://w3ctag.github.io/design-principles/#times-and-dates), ensuring consistency across media stream APIs on the web platform.
+
+---
+
 ### Proposed Behavior & Example Usage
 
-The `audioStartTime` and `audioEndTime` properties represent the audio duration bounds (in milliseconds) corresponding to the transcribed segment. If the underlying recognition engine backend does not support segment timestamps, these attributes return `null`.
+The `audioStartTime` and `audioEndTime` properties represent the audio duration bounds (in seconds) corresponding to the transcribed segment.
 
-Developers can programmatically compute processing latency by comparing `audioEndTime` against the standard DOM event generation timestamp (`Event.timeStamp`):
+Developers can programmatically compute processing latency by converting the stream timestamp to the document timeline and comparing against `Event.timeStamp`:
 
 ```javascript
 const recognition = new SpeechRecognition();
 recognition.continuous = true;
 recognition.interimResults = true;
 
+let audioOriginMs = 0;
+
+// 1. Capture the audio stream's start timestamp on the document timeline
+recognition.onaudiostart = (event) => {
+  audioOriginMs = event.timeStamp;
+};
+
 recognition.onresult = (event) => {
   const result = event.results[event.resultIndex];
 
-  if (result.audioEndTime !== null && result.audioEndTime !== undefined) {
-    // Calculate on-device processing latency
-    const processingLatencyMs = event.timeStamp - result.audioEndTime;
+  // 2. Convert stream-relative seconds to document timeline milliseconds
+  const absoluteAudioEndMs = audioOriginMs + (result.audioEndTime * 1000);
+
+  // 3. Calculate on-device processing latency
+  const processingLatencyMs = event.timeStamp - absoluteAudioEndMs;
 
-    // Trigger seamless failover to cloud backend if latency breaches acceptable threshold
-    if (processingLatencyMs > 1500) {
-      console.warn(`ASR processing lag detected (${processingLatencyMs}ms). Transitioning to cloud provider.`);
-      switchToCloudBackend();
-    }
+  // 4. Trigger seamless failover to cloud backend if latency breaches acceptable threshold
+  if (processingLatencyMs > 1500) {
+    console.warn(`ASR processing lag detected (${processingLatencyMs.toFixed(0)}ms). Transitioning to cloud provider.`);
+    switchToCloudBackend();
   }
 };
 
 recognition.start();
 ```
 
-## Converting Stream Timestamps to Document Time Origin
+---
 
-`audioStartTime` and `audioEndTime` are defined as media-local offsets in milliseconds relative to the start of the audio stream ($t = 0.0\text{ms}$).
+### Converting Stream Timestamps to Document Time Origin
 
-For real-time applications such as **live translation**, **subtitling overlays**, and **audio-visual sync**, developers often need to map these stream offsets to the document's global timeline (`DOMHighResTimeStamp` / `performance.now()`).
+`audioStartTime` and `audioEndTime` are defined as media-local offsets in seconds relative to the start of the audio stream ($t = 0.0\text{s}$).
 
-### Pattern: Capturing the Audio Timeline Origin
+For real-time applications such as **live translation**, **subtitling overlays**, and **audio-visual sync**, developers often need to map these stream offsets to the document's global timeline (`performance.timeOrigin` / `performance.now()`).
+
+#### Pattern: Capturing the Audio Timeline Origin
 
 To convert stream-relative timestamps to document time coordinates:
-1. Record the baseline timestamp when the `audiostart` event fires (`event.timeStamp` is a `DOMHighResTimeStamp` relative to `timeOrigin`).
-2. Add the result's `audioStartTime` and `audioEndTime` offsets to that baseline.
+1. Record the baseline timestamp when the `audiostart` event fires (`event.timeStamp` is in milliseconds relative to `timeOrigin`).
+2. Add the result's `audioStartTime` and `audioEndTime` offsets (converted to milliseconds) to that baseline:
 
-$$\text{absoluteStartTime} = \text{audioOrigin} + \text{result.audioStartTime}$$
-$$\text{absoluteEndTime} = \text{audioOrigin} + \text{result.audioEndTime}$$
+$$\text{absoluteStartTimeMs} = \text{audioOriginMs} + (\text{result.audioStartTime} \times 1000)$$
+$$\text{absoluteEndTimeMs} = \text{audioOriginMs} + (\text{result.audioEndTime} \times 1000)$$
 
-### Measuring Live Translation Latency Example
+#### Measuring Live Translation Latency Example
 
 In live speech translation workflows, measuring both **Speech-to-Text (STT) latency** and **Machine Translation (MT) end-to-end latency** is essential:
 
@@ -78,23 +111,22 @@ const recognition = new SpeechRecognition();
 recognition.continuous = true;
 recognition.interimResults = true;
 
-let audioOriginTime = 0;
+let audioOriginTimeMs = 0;
 
 // 1. Capture the audio stream's time origin on the document timeline
 recognition.onaudiostart = (event) => {
-  audioOriginTime = event.timeStamp;
+  audioOriginTimeMs = event.timeStamp;
 };
 
 recognition.onresult = async (event) => {
   const result = event.results[event.resultIndex];
-  if (result.audioEndTime === null || result.audioEndTime === undefined) return;
 
-  // 2. Convert stream offsets to document time origin coordinates
-  const absoluteAudioStart = audioOriginTime + result.audioStartTime;
-  const absoluteAudioEnd = audioOriginTime + result.audioEndTime;
+  // 2. Convert stream offsets (seconds) to document timeline (milliseconds)
+  const absoluteAudioStartMs = audioOriginTimeMs + (result.audioStartTime * 1000);
+  const absoluteAudioEndMs = audioOriginTimeMs + (result.audioEndTime * 1000);
 
   // 3. Compute ASR recognition latency
-  const asrLatencyMs = event.timeStamp - absoluteAudioEnd;
+  const asrLatencyMs = event.timeStamp - absoluteAudioEndMs;
 
   // 4. Perform live translation
   const text = result[0].transcript;
@@ -103,23 +135,44 @@ recognition.onresult = async (event) => {
   const translationEndTime = performance.now();
 
   // 5. Total end-to-end latency from speaker utterance to translated subtitle
-  const totalE2ELatencyMs = translationEndTime - absoluteAudioEnd;
+  const totalE2ELatencyMs = translationEndTime - absoluteAudioEndMs;
 
   console.log(`ASR Processing Time: ${asrLatencyMs.toFixed(1)}ms`);
   console.log(`Total Live Translation Delay: ${totalE2ELatencyMs.toFixed(1)}ms`);
 };
+
+recognition.start();
 ```
+
 ---
+
 ### Security and Privacy Considerations
 
 #### Fingerprinting Risk
-Exposing sub-millisecond or precise micro-architectural timing information enables hardware profiling (measuring CPU execution speed, thermal throttling, and system load), creating a tracking vector for cross-origin user fingerprinting.
+Exposing sub-millisecond or precise micro-architectural timing information enables hardware profiling (measuring CPU execution speed, thermal throttling, and system load), creating a potential tracking vector for cross-origin user fingerprinting.
 
 #### Mitigation Strategy
-To mitigate fingerprinting vectors, browser implementations MUST apply timestamp fuzzing and precision reduction before exposing timing attributes to web scripts. We propose mirroring the precision capping strategy used in [`HTMLMediaElement.currentTime`](https://developer.mozilla.org/en-US/docs/Web/API/HTMLMediaElement/currentTime), rounding exposed timestamps to **2ms** precision (or matching the site-wide timer resolution policy).
+To mitigate potential side-channel and fingerprinting vectors:
+* **Precedent:** Implementations (such as Chromium) follow the security posture established by [`HTMLMediaElement.currentTime`](https://developer.mozilla.org/en-US/docs/Web/API/HTMLMediaElement/currentTime), coarsening raw engine timestamps to discrete resolution buckets (e.g. 2ms resolution) in non-isolated execution contexts.
+* **Specification Precedent:** In accordance with [W3C High Resolution Time Level 3](https://www.w3.org/TR/hr-time-3/#privacy-security), the specification does not mandate a hardcoded quantization value, allowing user agents to adjust timer resolution or introduce jitter according to their security and cross-origin isolation policies.
+
+---
 
 ### Alternatives Considered
 
 - **Browser-Generated Warning Events (`onprocessinglag`):** Simple for web applications to catch, but fails to accommodate varying latency thresholds across different use cases (e.g. real-time meeting captioning requires <200ms latency, while dictation tools tolerate multi-second delays).
 - **Internal Processing Queue Metric (`queueDepth`):** Directly exposes engine backlogs, but is difficult to standardize across fragmented engine architectures, model types, and buffering strategies.
 - **Binary Status Flag (`isRealTime`):** Simple boolean check, but lacks numerical precision for applications seeking to track progressive latency degradation trendlines.
+- **Existing API Surfaces (Events):** Existing events were deemed insufficient because:
+  1. **`speechstart` and `speechend` Events:**
+     The Web Speech API specification defines `speechstart` and `speechend` events on the `SpeechRecognition` interface. However, these events cannot solve the continuous latency tracking problem:
+     * **Session-level vs. Result-level Granularity:** In continuous recognition mode, `speechstart` and `soundstart` fire once when voice activity is first detected at the beginning of the session. They do not fire for every individual phrase or sentence returned in subsequent `SpeechRecognitionResult` events.
+     * **Inequality with Result Audio Boundaries:** Because `speechstart` only marks initial voice activity, `speechstart.timeStamp` is not equal to `result.audioStartTime` for any subsequent utterance emitted throughout a session.
+     * **Fragile Event Correlation:** Even if engines fired `speechstart`/`speechend` around each phrase, associating separate asynchronous DOM events with streaming interim and final `SpeechRecognitionResult` objects requires complex, error-prone client-side state tracking (poor ergonomics).
+  2. **Overloading `event.timeStamp` on Result Events:**
+     Another alternative considered was modifying `event.timeStamp` on the `result` event to match the speech timing:
+     * **Eliminates Latency Calculation:** `event.timeStamp` indicates when the browser dispatched the DOM event on the document timeline. Keeping `event.timeStamp` intact while providing `result.audioEndTime` allows web applications to measure processing delay:
+       $$\text{latencyMs} = \text{event.timeStamp} - (\text{audioOriginMs} + \text{result.audioEndTime} \times 1000)$$
+     * Overwriting `event.timeStamp` would conflate acoustic timing with main-thread dispatch time, eliminating the ability to detect processing lag.
+
+Attaching `audioStartTime` and `audioEndTime` directly to `SpeechRecognitionResult` provides a 1:1 association between the recognized transcript text and its corresponding acoustic timeline.