Repository navigation
feat(ai-components): add VoiceWave and VoiceGlow voice-agent visualizations - #3339
Merged
Merged
Conversation
…ations Port the voice agent visualizations from stream-chat-android (GetStream/stream-chat-android#6848, #6849, #6851) to the ai-components entry point, with the same motion maths. - VoiceWave: a string carrying a wave packet, drawn on a 2D canvas. - VoiceGlow: a soft globe drawn by a WebGL shader, with a 2D canvas fallback chosen by `rendering`. - useAudioLevel: a pull-based level getter for a MediaStream, for the `audioLevel` prop. - Both follow the theme through the voice-color and voice-backdrop tokens, honour prefers-reduced-motion, pause off screen and are labelled for assistive technology in all 12 locales. The ai-chatbot example gains a hands-free voice mode built on them, and its send path moves into a shared useSendAIMessage hook.
…ow to the globe The glow is all soft gradients, so a 2x backing store filled four times the pixels for no visible difference; it now draws at no more than 1x, while the wave keeps up to 2x for a crisp line. The Canvas glow clips its color masses to the globe instead of painting past it and masking the excess away, and computes the orbit angles once a frame.
`rendering='css'` composes the glow the way the shader does, as CSS: an opaque color field (a base and the masses mixed over it in order, with the shader's smoothstep falloffs, diffuse shade and fresnel rim) seen through an alpha that only the first two masses carry, a faint static body plus one mask riding each of their orbits, with inverse turns inside each mask keeping the colors still. Every gradient is painted once and only transform and opacity animate, so the compositor runs the motion: it keeps moving while the main thread is busy, needs no canvas or WebGL, and runs no script per frame unless an audio level is given. The palette is computed with the shader's color math and follows the theme; each state sets playback rates, so state changes ease instead of jumping. WebGL stays the default for `auto`.
…h, speech rhythm and fade-in - Only the shape takes the pointer: the root lets events through to whatever is behind it, and a hit area the size of the globe or the string takes them and hands them to the root's handlers. - The wave and the Canvas glow animate speech with the shader's simplex envelope, ported in float32, instead of a 1-D CPU noise, so every renderer and iOS share the same rhythm. - The visualizations fade in over 0.3s once their first frame is drawn (the CSS glow once its palette is set), instantly with reduced motion.
…ghter CSS glow `rendering` is now 'auto' | 'webgl' | 'css'. Where WebGL cannot start or loses its context, VoiceGlow falls back to the CSS renderer instead of a Canvas2D drawing: closer to the shader in look, and it keeps moving while the main thread is busy. Standing in for WebGL, the GPU is likely off and the compositor blends every layer on the CPU, so the fallback draws a lighter CSS glow: the still body and the body riding the first orbit, without the stretches, shade and rim. With GPU acceleration off it holds 60 fps at about half the CPU of the full CSS glow, close to the old Canvas2D cost, while leaving the main thread nearly idle. An explicit rendering='css' keeps the full look. Both skip masses the palette leaves empty. The Canvas2D renderer and the 1-D noise only it used are removed.
…t at up to 3x Stroke the wave at 0.6% of the shorter side with Android's 1.5 device-pixel floor, instead of a 1.5 CSS-pixel floor that made it heavier in boxes under 250px, and draw it at the screen's own density up to 3x, as iOS does. Each visualization now sets its own pixel-ratio cap. Give the ai-chatbot voice mode's wave a square box, since the wave is sized from the shorter side.
In loud speech the wave's crests rose into near-vertical spikes and its height swung sharply with every syllable. The crests now spread out instead of steepening past a slope of 6, with the cap on their spacing eased so it breathes, and the speech height rises quickly and settles slowly, with shallower syllables. While idle, connecting or thinking the wave is still exactly the Android and iOS wave. useAudioLevel now measures over 2048 samples (about 43 ms) instead of 256, a few pitch periods of a voice, so the level holds steady through a vowel. Renderers get the time since the last frame so the smoothing runs at the same speed at any frame rate. The voice-mode demo uses the default expressivity, and the CSS glow stylesheet uses list.nth in place of the deprecated global nth.
Contributor
|
Size Change: +49.1 kB (+4.34%) Total Size: 1.18 MB 📦 View Changed
ℹ️ View Unchanged
|
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## release-v14 #3339 +/- ##
===============================================
+ Coverage 85.89% 86.45% +0.55%
===============================================
Files 524 540 +16
Lines 16649 17516 +867
Branches 5229 5401 +172
===============================================
+ Hits 14301 15143 +842
- Misses 2348 2373 +25 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configuration
You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
…ow and fix review findings - Every WebGL glow now draws with one shared offscreen context and copies its frame onto its own 2D canvas, so the shader compiles once and many glows stay under the browser's context limit. A lost context drops every glow to CSS; the last glow to unmount releases it. - The CSS glow has one composition, the former fallback: no stretches, shade, rim or second riding mass. Off screen its animations pause and its theme and audio reads stop. - `rendering` is `'auto' | 'css'`; `'webgl'` behaved exactly like `'auto'`. - Colors the browser rejects are skipped instead of read as black, and an unreadable backdrop is no longer taken for dark. - Both glows share the theme reading, the pointer conversion and the driver's easing constants. - The driver computes its color slots once per palette and eases its own copies in place. - The canvas follows a change of pixel ratio alone, as on a move to another monitor. - Ripple timers are dropped once they fire. - The voice-mode demo ends a turn the agent is never seen working on after 20 seconds.
MartinCupela
approved these changes
Oct 22, 2026
oliverlaz
added a commit
that referenced
this pull request
Oct 22, 2026
Resolve the ai-components README conflict with #3339 (VoiceWave and VoiceGlow): keep both sides' exports in the component, hook, context and type lists, and both hook sections (useHeardText and useAudioLevel).
github-actions Bot
pushed a commit
that referenced
this pull request
Oct 22, 2026
## [14.14.0](v14.13.3...v14.14.0) (2026-10-22) ### Features * **ai-components:** add VoiceWave and VoiceGlow voice-agent visualizations ([#3339](#3339)) ([da6f72f](da6f72f)), closes [GetStream/stream-chat-android#6848](GetStream/stream-chat-android#6848) [GetStream/stream-chat-android#6849](GetStream/stream-chat-android#6849) [GetStream/stream-chat-android#6851](GetStream/stream-chat-android#6851) [GetStream/stream-chat-swift#4338](GetStream/stream-chat-swift#4338) * **Message:** add SpokenMessageText for voice-paced AI message text ([#3343](#3343)) ([22abe90](22abe90)), closes [GetStream/Vision-Agents#1128](GetStream/Vision-Agents#1128)
|
🎉 This PR is included in version 14.14.0 🎉 The release is available on: Your semantic-release bot 📦🚀 |
oliverlaz
added a commit
that referenced
this pull request
Oct 22, 2026
…ardText hook (#3344) ### 🎯 Goal `stream-chat-react/ai-components` must work standalone, without the core SDK. `SpokenMessageText` (added in #3343) rendered through the core `Message` components (`MessageText`, the `StreamedMessageText` fallback, and the message and channel contexts), so it doesn't belong in the plugin. This keeps only the utility hook. The drop-in component becomes a recipe in the docs (GetStream/getstream.io#761). This is the last commit of #3343 (`57e84ca2e`), which was pushed just before the PR was merged and didn't make it in. >⚠️ **14.14.0 already shipped `SpokenMessageText`, `SpokenTextProvider`, `SpokenTextContext` and `useSpokenTextContext`** (with the message-based `useHeardText`) from `stream-chat-react/ai-components`. This PR removes those exports and changes `useHeardText`'s signature, and ships as a fix (14.14.1): they were released minutes ago without docs, so it is treated as correcting a mistaken release rather than a major change. ### 🛠 Implementation details - **Removed:** `SpokenMessageText`, `SpokenTextProvider`, `SpokenTextContext`, `useSpokenTextContext`, their types and their tests. - **`useHeardText`** now lives in `src/plugins/AIComponents/hooks/useHeardText.ts`. Its only import is `react`; I checked this in the source, the built ESM and the CJS bundle. - `useHeardText({ text, heard, subscribe?, generating?, live? }) => { heardText, isPaced, skip }`. - `heard(now)` returns `{ chars, done } | undefined`, where `chars` counts Unicode code points. - `heard` and `subscribe` are read through refs, so new function identities on each render don't restart pacing. - Types: `HeardTextProgress`, `HeardTextFunction`, `HeardTextSubscribe`, `UseHeardTextParams`, `UseHeardTextResult`. - **Behaviour:** - Reveals the first `floor(chars)` code points and never goes backwards; surrogate pairs are never split. - Ticks on `requestAnimationFrame` only while paced and not done. - The full text appears once `done && !generating`, when `live` is false, or after `skip()`. - Falls back to the full text when unpaced. - **README:** documents the hook with a plain-element example and points to the docs for the drop-in. The VoiceWave/VoiceGlow docs from #3339 are untouched. - **Checks:** 21 `renderHook` tests. Full suite 3175/3175 passes, as do `yarn types`, `yarn lint`, `yarn build` and `validate-cjs` / `validate-esm` / `validate-entry-isolation`. In the built output, `ai-components` exports `useHeardText` and nothing `Spoken*`, and the main entry exports neither. - **Verified** end to end in a Next.js app against a local Stream Agents router, using the docs' `SpokenMessageText` recipe on this hook: each voice reply is revealed as its audio plays. ### 🎨 UI Changes None.
github-actions Bot
pushed a commit
that referenced
this pull request
Oct 22, 2026
## [14.14.1](v14.14.0...v14.14.1) (2026-10-22) ### Bug Fixes * **ai-components:** replace SpokenMessageText with a standalone useHeardText hook ([#3344](#3344)) ([ad16506](ad16506)), closes [#3343](#3343) [GetStream/getstream.io#761](https://github.com/GetStream/getstream.io/issues/761) [#3343](#3343) [#3339](#3339)
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Goal
Android (GetStream/stream-chat-android#6848, GetStream/stream-chat-android#6849, GetStream/stream-chat-android#6851) and iOS (GetStream/stream-chat-swift#4338) added two visualizations of a voice agent to their AI components. This brings them to
stream-chat-react/ai-componentson v14, with the same motion, so the three platforms look and move alike, and shows them in a new voice mode in theai-chatbotsample.Implementation details
Public API, all from
stream-chat-react/ai-components:VoiceWave: a single string carrying a wave packet that erupts with speech, its colors flowing along it.VoiceGlow: a soft globe whose color masses orbit inside it and swell with speech.rendering?: 'auto' | 'css'(defaultauto).useAudioLevel(stream): a stable getter for the live level of aMediaStream(a microphone or a remote track), for theaudioLevelprop.VoiceAgentState,VoiceGlowRendering,VoiceVisualizationProps,VoiceWaveProps,VoiceGlowProps.Both components take
state(idle | connecting | listening | thinking | speaking),colors(one to five CSS colors, defaulting to the theme accent),expressivity(0 to 2) andaudioLevel(a getter read once a frame, so a changing level re-renders nothing;nullanimates speech on its own). Any otherdivprop goes to the root.Shared motion. A port of Android's driver: the per-state motion table, frame-rate-independent easing, reduced motion slowing time to 0.07x, the tap bounce and the HSV palette. The speech envelope runs the shader's simplex noise in float32, so the CPU-drawn wave and the GPU glow speak with the same rhythm as iOS.
VoiceWave is a Canvas2D port of Android's
WaveDrawing. Its line is 0.6% of the shorter side with Android's 1.5 device-pixel floor, drawn at the screen's density up to 3x as on iOS. In speech it is smoothed on the web: the crests spread out rather than steepen past a slope of 6, and the height rises quickly and settles slowly. While idle, connecting or thinking it is exactly the native wave, and it matches Android's Paparazzi snapshots down to antialiasing.VoiceGlow draws with a GLSL ES 1.00 port of the AGSL shader, at 1x (it is all soft gradients). Every glow on a page draws with one shared offscreen WebGL context, as iOS shares one renderer, and copies its frame onto its own 2D canvas. The shader compiles once, and a page with many glows stays under the browser's limit on live contexts. If the context is lost, every glow falls back to CSS and the next glow to mount starts a fresh context; the last glow to unmount releases it.
rendering='css', also the fallback where WebGL is unavailable or lost, is one CSS composition of the shader's glow: a color field seen through a mask riding the first mass's orbit, without the shader's fine marbling or its second, fainter riding mass. Its gradients are painted once and onlytransformandopacityanimate, so the browser's compositor runs the motion and it keeps moving while the main thread is busy. Off screen, its animations pause and its theme and audio reads stop. With GPU acceleration disabled, it holds 60 fps at about 18 to 22% CPU at 300px in headless Chrome.Behavior shared with iOS:
onClickstill fires.--str-chat__ai-voice-colorand--str-chat__ai-voice-backdrop, re-read about once a second.role="img"with a name such as "Voice agent: Speaking", translated in all 12 locales.useAudioLevelmeasures RMS on demand over the last 2048 samples (about 43 ms), so the level holds steady through a vowel. The audio graph is torn down when the stream changes and on unmount.Sample app.
ai-chatbotgets a voice mode: a full-panel overlay with the glow for the agent and the wave for the user's microphone. It usesuseSpeechToTextfor dictation andspeechSynthesisto read replies aloud, and returns to listening hands-free. A turn the agent is never seen working on ends after 20 seconds, with its answer if one came in. The send path moved into a shareduseSendAIMessage. The sample's README andimplementation.mddescribe it.Known differences from native:
Tests:
useAudioLevel.UI Changes
Screenshots and a recording of voice mode are to be added. In the meantime, the sample's voice mode shows both components: run
yarn build, thenyarn start:ai-chatbot, and start voice mode from the composer. Microphone access needs Chrome, Edge or Safari.