Repository navigation
feat(elevenlabs): support eleven_v3 models via text-to-dialogue websocket - #2
Conversation
…cket The multi-stream-input endpoint rejects eleven_v3 and eleven_v3_conversational with 403 at the websocket handshake. Route these models over the Realtime Text to Dialogue websocket instead, keeping the public TTS class and constructor unchanged. - one _DialogueConnection per TTS instance, reused across synthesis turns and kept open with keep_alive (the server idles out ~20s after the last client message); dropped after inactivity_timeout without streams - completion is tracked by counting is_final_audio_for_turn markers, which the server emits exactly once per flush that had pending text; flushes with no pending text are skipped so the count stays exact - the endpoint has no cancel message and a new turn never stops in-flight synthesis, so interruption closes the socket and rewarms a replacement in the background - sync_alignment is honored: the endpoint returns character timings with snake_case keys - ChunkedStream runs as a one-shot turn on the same connection - existing voice_settings, language_code, base_url (EU residency), pronunciation dictionaries, and encodings carry over unchanged Ref livekit#4901
Mirrors the existing suite's structure: model routing, dialogue stream URL, init packet shape (voices as plain ID strings, first-message-only voice_settings), send-side new_turn flagging and dirty-flag flush counting, and recv-loop behavior (audio routing, flush-marker completion, snake_case alignment parsing, benign idle timeout, poisoned-connection close, error on unexpected close with an active turn).
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
|
@codex review |
|
To use Codex here, create a Codex account and connect to github. |
|
@codex review |
1 similar comment
|
@codex review |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6ceb9468b1
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
…e sockets Address review findings: - SynthesizeStream and ChunkedStream now select the protocol from their snapshotted model instead of the live TTS options, so a concurrent update_options() model switch can no longer send one request down a path that disagrees with its payload; the connection-type check turns the rare cross-over into a clear retryable error - a dialogue connection marked non-current now closes as soon as it is idle (immediately from mark_non_current, after drain from finish_turn, with a keep-alive-loop backstop) instead of lingering for inactivity_timeout; per-turn voice or model switches no longer accumulate orphaned websockets against provider connection limits
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ab0b874e88
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
A request keeps its snapshotted model, so when update_options() switches between eleven_v3 and other models while the request is starting, no retry can ever match the new connection type. Raise a non-retryable APIConnectionError with an actionable message instead of burning the retry budget on a deterministic mismatch.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d1185512ee
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
Two more review findings on the dialogue path: - TTS.aclose() no longer cancels in-flight connection close tasks (a close cancelled after marking the connection closed would leak the websocket and its recv/keep-alive tasks past shutdown). Close tasks are spawned through the TTS instance and awaited at shutdown; only the prewarm task, which may sit in a slow handshake, is cancelled. - once a turn's input is done, a stall timer (re-armed per audio message) bounds the wait for the final marker. The client keep_alive defeats the server's own idle close, so a mid-generation stall would otherwise hang the waiter forever. Firing discards the connection, which also stops the stalled generation.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 690d3c7aa0
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
synthesize() streams are not tracked in _streams, so a ChunkedStream woken by shutdown runs its discard path after aclose() has already cancelled the prewarm task, scheduling a fresh websocket on a closed instance. A _closing flag now makes discard skip prewarming and _current_connection refuse new connections once shutdown begins.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8dae50be68
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
The _closing guard ran only before awaiting connect(), and aclose() does not hold the connection lock, so a shutdown starting mid-handshake found nothing to close while the connect went on to publish a live websocket on a closed instance. Recheck after the handshake and close the fresh connection instead of publishing it. Covers the multi-stream path too, since the acquisition is shared.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 79eb99ccdc
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
update_options() can run while connect() is awaiting the handshake; it mutates the live options but cannot mark the not-yet-published connection non-current, so a wrong-family connection would be published with is_current=True and wedge every subsequent stream until something else closed it. Revalidate the family after the handshake, under the connection lock, and discard the mismatched connection with a retryable error - the retry rebuilds from the current options.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8d0997e40b
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
… up failed init Three more review findings on the dialogue path: - start_turn() now applies the request timeout while waiting for the active turn, so an idle constructed stream can no longer starve later requests indefinitely - synthesize() streams are tracked in _streams like stream() ones, so TTS.aclose() cancels active chunked requests instead of leaving a detached dialogue websocket synthesizing past shutdown - a failed or cancelled init send right after the websocket handshake closes the socket before propagating, instead of leaking one open websocket per retry
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9a0abd372b
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
ChunkedStream discarded the acquisition duration and reuse flag, so its TTSMetrics always reported acquire_time=0.0 and connection_reused=False on the dialogue path.
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: aba5851123
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
…uisition The blanket exception handler around _current_connection() wrapped the deliberate non-retryable closed-instance error into a fresh retryable APIConnectionError, so requests racing aclose() retried against a closed instance. Re-raise APIError instances unchanged in both acquire paths.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: da9d70fa37
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
If the input or sentence task dies (for example while sending the final flush after audio already arrived), no timer is armed and the stream would hang holding the active turn, wedging the connection for later requests. Forward producer exceptions to the waiter so the error surfaces and the connection is discarded.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3b261cab76
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
… handshake The post-handshake guard only compared the protocol family, so a same-family switch (eleven_v3 to eleven_v3_conversational, or a voice/settings change) during the handshake published a socket negotiated with stale options that persisted as current. An options-generation counter now invalidates any connection whose options changed while connecting; this subsumes the family check.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8c37394f42
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
…e writes - the stale-connection closes inside _current_connection() are now spawned through the tracked background set instead of awaited inline, so a caller's acquisition timeout cannot cancel them halfway and leave a half-closed connection; TTS.aclose() sweeps the set again after closing the current connection - all dialogue websocket writes are bounded (turn writes by the request timeout, keep-alives by their interval); a write stalled by transport backpressure now raises APITimeoutError, which fails the producer task and discards the connection instead of holding the turn forever
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5b7a35e497
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
… on superseded connections A self-audit of the accumulated review fixes against an explicit invariant set found one hole and one gap: - start_turn() accepted superseded connections: a stream queued on the turn lock while the previous turn drained could start its turn on a connection update_options() had already replaced, sending text to a socket guaranteed to close moments later. Turn starts now require a current connection; the retry acquires a fresh one. - the lifecycle rules the review rounds converged on existed only implicitly across commits; they are now spelled out as five invariants in the _DialogueConnection docstring (publish, supersede, close, turns, bounded waits) so reviewers can check changes against the model instead of hunting interleavings one at a time.
The write timeout only covered the websocket send, so a backpressured writer holding the send lock (e.g. a keep-alive) could extend another turn's write beyond its budget before its bounded window even started. One wait_for now bounds lock acquisition and the write together.
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 25ab999fcc
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
aclose() treated the _closed flag as proof of finished cleanup, so a close racing a self-initiated one (e.g. shutdown colliding with a server-side close) could return while the socket and tasks were still winding down. Concurrent closers now wait on a completion event set at the end of cleanup; the recv/keep-alive finallys skip aclose() once _closed is set, so no task waits on its own completion.
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5be25ad9c7
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| if self._prewarm_task is not None: | ||
| self._prewarm_task.cancel() |
There was a problem hiding this comment.
When shutdown catches _prewarm_task after its WebSocket handshake has succeeded but while the initialization send is pending, cancel() starts the asynchronous cleanup in _DialogueConnection.connect(), which itself waits for the socket to close. If there are no background tasks or published current connection, TTS.aclose() then returns without another await, leaving the prewarm task and its socket alive after shutdown. Cancel and join this task before returning so its cancellation cleanup is guaranteed to finish.
Useful? React with 👍 / 👎.
…cket (#2) * feat(elevenlabs): support eleven_v3 models via text-to-dialogue websocket The multi-stream-input endpoint rejects eleven_v3 and eleven_v3_conversational with 403 at the websocket handshake. Route these models over the Realtime Text to Dialogue websocket instead, keeping the public TTS class and constructor unchanged. - one _DialogueConnection per TTS instance, reused across synthesis turns and kept open with keep_alive (the server idles out ~20s after the last client message); dropped after inactivity_timeout without streams - completion is tracked by counting is_final_audio_for_turn markers, which the server emits exactly once per flush that had pending text; flushes with no pending text are skipped so the count stays exact - the endpoint has no cancel message and a new turn never stops in-flight synthesis, so interruption closes the socket and rewarms a replacement in the background - sync_alignment is honored: the endpoint returns character timings with snake_case keys - ChunkedStream runs as a one-shot turn on the same connection - existing voice_settings, language_code, base_url (EU residency), pronunciation dictionaries, and encodings carry over unchanged Ref livekit#4901 * test(elevenlabs): cover the text-to-dialogue websocket path Mirrors the existing suite's structure: model routing, dialogue stream URL, init packet shape (voices as plain ID strings, first-message-only voice_settings), send-side new_turn flagging and dirty-flag flush counting, and recv-loop behavior (audio routing, flush-marker completion, snake_case alignment parsing, benign idle timeout, poisoned-connection close, error on unexpected close with an active turn). * fix(elevenlabs): route on snapshotted model, close superseded dialogue sockets Address review findings: - SynthesizeStream and ChunkedStream now select the protocol from their snapshotted model instead of the live TTS options, so a concurrent update_options() model switch can no longer send one request down a path that disagrees with its payload; the connection-type check turns the rare cross-over into a clear retryable error - a dialogue connection marked non-current now closes as soon as it is idle (immediately from mark_non_current, after drain from finish_turn, with a keep-alive-loop backstop) instead of lingering for inactivity_timeout; per-turn voice or model switches no longer accumulate orphaned websockets against provider connection limits * fix(elevenlabs): fail fast on model-family change during request start A request keeps its snapshotted model, so when update_options() switches between eleven_v3 and other models while the request is starting, no retry can ever match the new connection type. Raise a non-retryable APIConnectionError with an actionable message instead of burning the retry budget on a deterministic mismatch. * fix(elevenlabs): survive shutdown races, bound post-input stalls Two more review findings on the dialogue path: - TTS.aclose() no longer cancels in-flight connection close tasks (a close cancelled after marking the connection closed would leak the websocket and its recv/keep-alive tasks past shutdown). Close tasks are spawned through the TTS instance and awaited at shutdown; only the prewarm task, which may sit in a slow handshake, is cancelled. - once a turn's input is done, a stall timer (re-armed per audio message) bounds the wait for the final marker. The client keep_alive defeats the server's own idle close, so a mid-generation stall would otherwise hang the waiter forever. Firing discards the connection, which also stops the stalled generation. * fix(elevenlabs): never open replacement connections during TTS shutdown synthesize() streams are not tracked in _streams, so a ChunkedStream woken by shutdown runs its discard path after aclose() has already cancelled the prewarm task, scheduling a fresh websocket on a closed instance. A _closing flag now makes discard skip prewarming and _current_connection refuse new connections once shutdown begins. * fix(elevenlabs): recheck shutdown after the connection handshake The _closing guard ran only before awaiting connect(), and aclose() does not hold the connection lock, so a shutdown starting mid-handshake found nothing to close while the connect went on to publish a live websocket on a closed instance. Recheck after the handshake and close the fresh connection instead of publishing it. Covers the multi-stream path too, since the acquisition is shared. * fix(elevenlabs): revalidate model family after the connection handshake update_options() can run while connect() is awaiting the handshake; it mutates the live options but cannot mark the not-yet-published connection non-current, so a wrong-family connection would be published with is_current=True and wedge every subsequent stream until something else closed it. Revalidate the family after the handshake, under the connection lock, and discard the mismatched connection with a retryable error - the retry rebuilds from the current options. * fix(elevenlabs): bound turn acquisition, track chunked streams, clean up failed init Three more review findings on the dialogue path: - start_turn() now applies the request timeout while waiting for the active turn, so an idle constructed stream can no longer starve later requests indefinitely - synthesize() streams are tracked in _streams like stream() ones, so TTS.aclose() cancels active chunked requests instead of leaving a detached dialogue websocket synthesizing past shutdown - a failed or cancelled init send right after the websocket handshake closes the socket before propagating, instead of leaking one open websocket per retry * fix(elevenlabs): record connection metrics for chunked dialogue requests ChunkedStream discarded the acquisition duration and reuse flag, so its TTSMetrics always reported acquire_time=0.0 and connection_reused=False on the dialogue path. * fix(elevenlabs): preserve deliberate APIErrors through connection acquisition The blanket exception handler around _current_connection() wrapped the deliberate non-retryable closed-instance error into a fresh retryable APIConnectionError, so requests racing aclose() retried against a closed instance. Re-raise APIError instances unchanged in both acquire paths. * fix(elevenlabs): fail the dialogue waiter when a producer task errors If the input or sentence task dies (for example while sending the final flush after audio already arrived), no timer is armed and the stream would hang holding the active turn, wedging the connection for later requests. Forward producer exceptions to the waiter so the error surfaces and the connection is discarded. * fix(elevenlabs): discard connections whose options changed during the handshake The post-handshake guard only compared the protocol family, so a same-family switch (eleven_v3 to eleven_v3_conversational, or a voice/settings change) during the handshake published a socket negotiated with stale options that persisted as current. An options-generation counter now invalidates any connection whose options changed while connecting; this subsumes the family check. * fix(elevenlabs): make discard closes cancellation-safe, bound dialogue writes - the stale-connection closes inside _current_connection() are now spawned through the tracked background set instead of awaited inline, so a caller's acquisition timeout cannot cancel them halfway and leave a half-closed connection; TTS.aclose() sweeps the set again after closing the current connection - all dialogue websocket writes are bounded (turn writes by the request timeout, keep-alives by their interval); a write stalled by transport backpressure now raises APITimeoutError, which fails the producer task and discards the connection instead of holding the turn forever * fix(elevenlabs): document dialogue lifecycle invariants, reject turns on superseded connections A self-audit of the accumulated review fixes against an explicit invariant set found one hole and one gap: - start_turn() accepted superseded connections: a stream queued on the turn lock while the previous turn drained could start its turn on a connection update_options() had already replaced, sending text to a socket guaranteed to close moments later. Turn starts now require a current connection; the retry acquires a fresh one. - the lifecycle rules the review rounds converged on existed only implicitly across commits; they are now spelled out as five invariants in the _DialogueConnection docstring (publish, supersede, close, turns, bounded waits) so reviewers can check changes against the model instead of hunting interleavings one at a time. * fix(elevenlabs): include send-lock acquisition in the write timeout The write timeout only covered the websocket send, so a backpressured writer holding the send lock (e.g. a keep-alive) could extend another turn's write beyond its budget before its bounded window even started. One wait_for now bounds lock acquisition and the write together. * fix(elevenlabs): join an in-progress dialogue close before returning aclose() treated the _closed flag as proof of finished cleanup, so a close racing a self-initiated one (e.g. shutdown colliding with a server-side close) could return while the socket and tasks were still winding down. Concurrent closers now wait on a completion event set at the end of cleanup; the recv/keep-alive finallys skip aclose() once _closed is set, so no task waits on its own completion.
1 participant
Problem
The plugin's websocket path speaks only
multi-stream-input. ElevenLabs rejectseleven_v3andeleven_v3_conversationalon that endpoint with HTTP 403 at the websocket handshake (verified live 2026-08-20; a flash-model control on the same key/voice passes). The v3 family streams only over the Realtime Text to Dialogue websocket. Upstream issue: livekit#4901.Design
Public
TTSclass and constructor unchanged. Models starting witheleven_v3route to a new_DialogueConnection; every other model keeps the existingmulti-stream-inputpath untouched. This mirrors the model-based protocol routing this plugin's own STT already does (scribe_v2_realtime), and openai's STT (_REALTIME_ONLY_MODELS).The dialogue endpoint differs from multi-stream in three ways that shape the implementation (all verified against production, EU and US hosts):
isFinalis_final_audio_for_turnper flush that had pending textclose_contextcancels one streaminactivity_timeoutkeep_aliveTherefore:
TTSinstance, reused across turns. EachSynthesizeStreamis one turn (new_turn: trueon its first input). Sentences stream unflushed (the server emits naturally past its ~40-char buffer); flush happens at end of input only.keep_aliveevery 10s while the socket is open; the socket is dropped afterinactivity_timeout(existing option, default 180s) without streams.sync_alignmentkeeps working: the dialogue endpoint returns character timings with snake_case keys; the recv loop accepts both key families.ChunkedStreamruns as a one-shot turn on the same connection — one code path.voice_settings(all five fields, including off-grid values),language_code, EU residencybase_url, pronunciation dictionaries, and the defaultmp3_22050_32encoding were all verified accepted on the dialogue endpoint unchanged.Verification
make checkgreen (format, lint, mypy strict).uv run pytest --plugin elevenlabs: 24 passed (10 existing + 14 new mirroring the existing suite's structure).synthesize()one-shot; EU residency host +language="de".Notes
eleven_v3_conversationaladded toTTSModels(eleven_v3was already present).Concurrency model (added after review)
Three actors touch the TTS instance's shared state: requests,
update_options(), andaclose(). The dialogue path follows five invariants, spelled out in the_DialogueConnectiondocstring:finish_turn, keep-alive loop as backstop.acloseis idempotent; discard closes run as tracked background tasks that are awaited (never cancelled) at shutdown, with a final sweep.Review findings map onto these invariants; changes should be checked against the model rather than individual interleavings.