feat: adds support for speak fallback#548
Merged
Merged
Conversation
Contributor
WalkthroughThe changes introduce support for configuring multiple Text-to-Speech (TTS) providers with fallback in the agent's "speak" configuration, updating both the backend data models and the README usage example. Comprehensive unit tests are added to verify correct handling, serialization, and deserialization of both single and multiple provider configurations. Changes
Sequence Diagram(s)sequenceDiagram
participant User
participant AgentOptions
participant Speak
participant Provider
User->>AgentOptions: Configure agent with speak (single or list)
AgentOptions->>AgentOptions: __post_init__ (normalize speak data)
AgentOptions->>Speak: Instantiate Speak object(s)
Speak->>Provider: Set provider attributes
User->>AgentOptions: Serialize/deserialize configuration
AgentOptions->>Speak: (If needed) Convert dict/list to Speak object(s)
Note over AgentOptions: AgentOptions now supports both single and multiple Speak providers
📜 Recent review detailsConfiguration used: CodeRabbit UI 📒 Files selected for processing (1)
🚧 Files skipped from review as they are similar to previous changes (1)
⏰ Context from checks skipped due to timeout of 90000ms (3)
✨ Finishing Touches🧪 Generate Unit Tests
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
jpvajda
commented
Jul 1, 2025
jpvajda
commented
Jul 1, 2025
jpvajda
commented
Jul 1, 2025
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (1)
tests/unit_test/test_unit_agent_speak_provider.py (1)
7-8: Remove unused imports for cleaner code.The
jsonandpytestimports are not used in this test file and can be removed to improve code cleanliness.-import json -import pytest
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (4)
README.md(2 hunks)deepgram/clients/agent/v1/websocket/options.py(2 hunks)examples/agent/fallback_tts/main.py(1 hunks)tests/unit_test/test_unit_agent_speak_provider.py(1 hunks)
🧰 Additional context used
🧠 Learnings (5)
📓 Common learnings
Learnt from: dvonthenen
PR: deepgram/deepgram-python-sdk#424
File: deepgram/clients/speak/v1/response.py:48-209
Timestamp: 2024-10-09T02:19:48.728Z
Learning: User dvonthenen prefers to defer certain suggestions, specifically regarding error handling and documentation enhancements in new data classes of `deepgram/clients/speak/v1/response.py`, and may revisit them later.
Learnt from: dvonthenen
PR: deepgram/deepgram-python-sdk#424
File: deepgram/clients/speak/v1/response.py:48-209
Timestamp: 2024-06-27T00:06:23.128Z
Learning: User dvonthenen prefers to defer certain suggestions, specifically regarding error handling and documentation enhancements in new data classes of `deepgram/clients/speak/v1/response.py`, and may revisit them later.
README.md (9)
Learnt from: dvonthenen
PR: deepgram/deepgram-python-sdk#472
File: examples/text-to-speech/websocket/output_to_wav/main.py:16-17
Timestamp: 2024-10-18T00:30:20.224Z
Learning: In `examples/text-to-speech/websocket/output_to_wav/main.py`, the hardcoded values for `AUDIO_FILE` and `TTS_TEXT` are intentional and should remain as is.
Learnt from: dvonthenen
PR: deepgram/deepgram-python-sdk#468
File: examples/text-to-speech/websocket/complete/main.py:26-30
Timestamp: 2024-09-27T15:21:34.197Z
Learning: In `examples/text-to-speech/websocket/complete/main.py`, commented-out options like `auto_flush_speak_delta` and `verbose` are intentional in examples and should not be flagged in reviews.
Learnt from: dvonthenen
PR: deepgram/deepgram-python-sdk#426
File: deepgram/clients/listen/v1/rest/options.py:12-12
Timestamp: 2024-10-09T02:19:46.087Z
Learning: Unused imports in `deepgram/clients/listen/v1/rest/options.py` are retained to maintain backwards compatibility.
Learnt from: dvonthenen
PR: deepgram/deepgram-python-sdk#426
File: deepgram/clients/listen/v1/rest/options.py:12-12
Timestamp: 2024-07-01T19:12:36.972Z
Learning: Unused imports in `deepgram/clients/listen/v1/rest/options.py` are retained to maintain backwards compatibility.
Learnt from: dvonthenen
PR: deepgram/deepgram-python-sdk#426
File: deepgram/clients/listen/__init__.py:6-6
Timestamp: 2024-10-09T02:19:46.086Z
Learning: Imports for DeepgramClientOptions and ClientOptionsFromEnv in deepgram/clients/listen/__init__.py should not be flagged as unused in reviews.
Learnt from: dvonthenen
PR: deepgram/deepgram-python-sdk#426
File: deepgram/clients/listen/__init__.py:6-6
Timestamp: 2024-07-01T18:18:02.415Z
Learning: Imports for DeepgramClientOptions and ClientOptionsFromEnv in deepgram/clients/listen/__init__.py should not be flagged as unused in reviews.
Learnt from: dvonthenen
PR: deepgram/deepgram-python-sdk#472
File: deepgram/clients/speak/v1/websocket/async_client.py:643-646
Timestamp: 2024-10-18T00:29:32.961Z
Learning: In `deepgram/clients/speak/v1/websocket/async_client.py`, the double call to `self._speaker.finish()` in the `finish` method of the `AsyncSpeakWSClient` class is intentional and required for proper cleanup.
Learnt from: dvonthenen
PR: deepgram/deepgram-python-sdk#472
File: examples/text-to-speech/websocket/simple/main.py:41-50
Timestamp: 2024-10-18T00:25:58.316Z
Learning: In the `examples/text-to-speech/websocket/simple/main.py` file, the code is intended as a simple example and may include unused handlers or placeholder code meant for illustration purposes.
Learnt from: dvonthenen
PR: deepgram/deepgram-python-sdk#472
File: examples/text-to-speech/websocket/output_to_wav/main.py:52-56
Timestamp: 2024-10-18T00:30:04.220Z
Learning: In `examples/text-to-speech/websocket/output_to_wav/main.py`, the code is intended as an example demonstrating functionality, and refactoring suggestions may not be necessary.
tests/unit_test/test_unit_agent_speak_provider.py (1)
Learnt from: dvonthenen
PR: deepgram/deepgram-python-sdk#468
File: examples/text-to-speech/websocket/complete/main.py:26-30
Timestamp: 2024-09-27T15:21:34.197Z
Learning: In `examples/text-to-speech/websocket/complete/main.py`, commented-out options like `auto_flush_speak_delta` and `verbose` are intentional in examples and should not be flagged in reviews.
examples/agent/fallback_tts/main.py (10)
Learnt from: dvonthenen
PR: deepgram/deepgram-python-sdk#472
File: examples/text-to-speech/websocket/simple/main.py:41-50
Timestamp: 2024-10-18T00:25:58.316Z
Learning: In the `examples/text-to-speech/websocket/simple/main.py` file, the code is intended as a simple example and may include unused handlers or placeholder code meant for illustration purposes.
Learnt from: dvonthenen
PR: deepgram/deepgram-python-sdk#472
File: examples/text-to-speech/websocket/output_to_wav/main.py:52-56
Timestamp: 2024-10-18T00:30:04.220Z
Learning: In `examples/text-to-speech/websocket/output_to_wav/main.py`, the code is intended as an example demonstrating functionality, and refactoring suggestions may not be necessary.
Learnt from: dvonthenen
PR: deepgram/deepgram-python-sdk#472
File: examples/text-to-speech/websocket/simple/main.py:101-101
Timestamp: 2024-10-18T00:26:33.968Z
Learning: In the `examples/text-to-speech/websocket/simple/main.py` file, it's acceptable for `wait_for_complete()` to lack exception handling, as it is intended to demonstrate functionality in an example context.
Learnt from: dvonthenen
PR: deepgram/deepgram-python-sdk#468
File: examples/text-to-speech/websocket/complete/main.py:26-30
Timestamp: 2024-09-27T15:21:34.197Z
Learning: In `examples/text-to-speech/websocket/complete/main.py`, commented-out options like `auto_flush_speak_delta` and `verbose` are intentional in examples and should not be flagged in reviews.
Learnt from: dvonthenen
PR: deepgram/deepgram-python-sdk#472
File: examples/text-to-speech/websocket/output_to_wav/main.py:16-17
Timestamp: 2024-10-18T00:30:20.224Z
Learning: In `examples/text-to-speech/websocket/output_to_wav/main.py`, the hardcoded values for `AUDIO_FILE` and `TTS_TEXT` are intentional and should remain as is.
Learnt from: dvonthenen
PR: deepgram/deepgram-python-sdk#472
File: examples/text-to-speech/websocket/output_to_wav/main.py:38-41
Timestamp: 2024-10-18T00:30:01.884Z
Learning: In the `examples/text-to-speech/websocket/output_to_wav/main.py` file, the code is intended as a simple example, and additional error handling for file operations is not required.
Learnt from: dvonthenen
PR: deepgram/deepgram-python-sdk#472
File: examples/speech-to-text/websocket/replay/main.py:78-80
Timestamp: 2024-10-18T16:20:17.742Z
Learning: In the `examples` directory of the Deepgram Python SDK, code is intended as illustrative examples and doesn't require production-level error handling.
Learnt from: dvonthenen
PR: deepgram/deepgram-python-sdk#426
File: deepgram/clients/listen/v1/websocket/__init__.py:8-8
Timestamp: 2024-10-09T02:19:46.086Z
Learning: Unused imports in `deepgram/clients/listen/v1/websocket/__init__.py` are retained to maintain backward compatibility and should not be flagged for removal in reviews.
Learnt from: dvonthenen
PR: deepgram/deepgram-python-sdk#426
File: deepgram/clients/listen/v1/websocket/__init__.py:8-8
Timestamp: 2024-07-01T19:21:39.778Z
Learning: Unused imports in `deepgram/clients/listen/v1/websocket/__init__.py` are retained to maintain backward compatibility and should not be flagged for removal in reviews.
Learnt from: dvonthenen
PR: deepgram/deepgram-python-sdk#426
File: deepgram/clients/speak/__init__.py:16-19
Timestamp: 2024-07-01T19:12:57.715Z
Learning: Unused imports in `deepgram/clients/speak/__init__.py` are retained for backward compatibility and should not be flagged for removal in reviews.
deepgram/clients/agent/v1/websocket/options.py (13)
Learnt from: dvonthenen
PR: deepgram/deepgram-python-sdk#431
File: deepgram/clients/listen/v1/websocket/client.py:30-30
Timestamp: 2024-10-09T02:19:46.087Z
Learning: The `LiveOptions` import in `deepgram/clients/listen/v1/websocket/client.py` is intentionally present for future use and should not be flagged for removal in reviews.
Learnt from: dvonthenen
PR: deepgram/deepgram-python-sdk#431
File: deepgram/clients/listen/v1/websocket/client.py:30-30
Timestamp: 2024-07-11T14:10:17.231Z
Learning: The `LiveOptions` import in `deepgram/clients/listen/v1/websocket/client.py` is intentionally present for future use and should not be flagged for removal in reviews.
Learnt from: dvonthenen
PR: deepgram/deepgram-python-sdk#426
File: deepgram/clients/listen/v1/rest/options.py:12-12
Timestamp: 2024-10-09T02:19:46.087Z
Learning: Unused imports in `deepgram/clients/listen/v1/rest/options.py` are retained to maintain backwards compatibility.
Learnt from: dvonthenen
PR: deepgram/deepgram-python-sdk#426
File: deepgram/clients/listen/v1/rest/options.py:12-12
Timestamp: 2024-07-01T19:12:36.972Z
Learning: Unused imports in `deepgram/clients/listen/v1/rest/options.py` are retained to maintain backwards compatibility.
Learnt from: dvonthenen
PR: deepgram/deepgram-python-sdk#426
File: deepgram/clients/listen/v1/websocket/__init__.py:8-8
Timestamp: 2024-10-09T02:19:46.086Z
Learning: Unused imports in `deepgram/clients/listen/v1/websocket/__init__.py` are retained to maintain backward compatibility and should not be flagged for removal in reviews.
Learnt from: dvonthenen
PR: deepgram/deepgram-python-sdk#426
File: deepgram/clients/listen/v1/websocket/__init__.py:8-8
Timestamp: 2024-07-01T19:21:39.778Z
Learning: Unused imports in `deepgram/clients/listen/v1/websocket/__init__.py` are retained to maintain backward compatibility and should not be flagged for removal in reviews.
Learnt from: dvonthenen
PR: deepgram/deepgram-python-sdk#468
File: examples/text-to-speech/websocket/complete/main.py:26-30
Timestamp: 2024-09-27T15:21:34.197Z
Learning: In `examples/text-to-speech/websocket/complete/main.py`, commented-out options like `auto_flush_speak_delta` and `verbose` are intentional in examples and should not be flagged in reviews.
Learnt from: dvonthenen
PR: deepgram/deepgram-python-sdk#426
File: deepgram/clients/speak/__init__.py:16-19
Timestamp: 2024-07-01T19:12:57.715Z
Learning: Unused imports in `deepgram/clients/speak/__init__.py` are retained for backward compatibility and should not be flagged for removal in reviews.
Learnt from: dvonthenen
PR: deepgram/deepgram-python-sdk#426
File: deepgram/clients/speak/__init__.py:16-19
Timestamp: 2024-10-09T02:19:46.087Z
Learning: Unused imports in `deepgram/clients/speak/__init__.py` are retained for backward compatibility and should not be flagged for removal in reviews.
Learnt from: dvonthenen
PR: deepgram/deepgram-python-sdk#426
File: deepgram/clients/speak/v1/websocket/response.py:23-30
Timestamp: 2024-07-01T19:12:00.190Z
Learning: When using the `dataclasses.field` with default values in the `deepgram/clients/speak/v1/websocket/response.py`, avoid using mutable default arguments. Instead, use `field(default_factory=...)`.
Learnt from: dvonthenen
PR: deepgram/deepgram-python-sdk#445
File: deepgram/clients/manage/v1/response.py:179-194
Timestamp: 2024-08-14T19:04:32.758Z
Learning: The conversion of lists to specific object types in the `__getitem__` method is required by `dataclasses_json` for serialization/deserialization purposes, even if they are already lists of those objects.
Learnt from: dvonthenen
PR: deepgram/deepgram-python-sdk#445
File: deepgram/clients/manage/v1/response.py:179-194
Timestamp: 2024-10-09T02:19:46.086Z
Learning: The conversion of lists to specific object types in the `__getitem__` method is required by `dataclasses_json` for serialization/deserialization purposes, even if they are already lists of those objects.
Learnt from: dvonthenen
PR: deepgram/deepgram-python-sdk#426
File: deepgram/clients/speak/v1/websocket/response.py:23-30
Timestamp: 2024-10-09T02:19:48.728Z
Learning: When using the `dataclasses.field` with default values in the `deepgram/clients/speak/v1/websocket/response.py`, avoid using mutable default arguments. Instead, use `field(default_factory=...)`.
🧬 Code Graph Analysis (2)
tests/unit_test/test_unit_agent_speak_provider.py (1)
deepgram/clients/agent/v1/websocket/options.py (7)
SettingsOptions(368-398)UpdateSpeakOptions(418-436)Agent(254-322)Speak(219-250)Provider(72-85)Endpoint(89-106)Header(21-27)
deepgram/clients/agent/v1/websocket/options.py (1)
deepgram/client.py (2)
speak(503-507)listen(489-493)
🪛 Flake8 (7.2.0)
tests/unit_test/test_unit_agent_speak_provider.py
[error] 7-7: 'json' imported but unused
(F401)
[error] 8-8: 'pytest' imported but unused
(F401)
[error] 9-9: 'deepgram.clients.agent.v1.websocket.options.Agent' imported but unused
(F401)
[error] 9-9: 'deepgram.clients.agent.v1.websocket.options.Provider' imported but unused
(F401)
🔇 Additional comments (9)
deepgram/clients/agent/v1/websocket/options.py (4)
276-276: LGTM! Well-designed type annotation change.The
Union[Speak, List[Speak]]type properly supports both single provider (backward compatible) and multiple provider (new fallback feature) configurations while maintaining the existing default behavior.
281-310: Excellent implementation of polymorphic type conversion.The
__post_init__method correctly handles all scenarios:
- List of dicts → List of Speak objects
- Single dict → Single Speak object
- Existing Speak objects remain unchanged
The pattern is consistent with existing
listenandthinkattribute handling, ensuring maintainability.
317-322: Correct deserialization logic for polymorphic speak attribute.The updated
__getitem__method properly handles both list and single dict formats during deserialization, maintaining consistency with the__post_init__conversion logic.
424-436: Consistent implementation across option classes.The
UpdateSpeakOptionsclass correctly mirrors theAgentclass implementation, enabling dynamic speak configuration updates with both single and multiple provider formats. The code consistency is excellent.README.md (1)
253-310: Excellent documentation with clear examples.The README effectively demonstrates both single and multiple TTS provider configurations:
- Maintains backward compatibility by showing existing patterns
- Provides realistic, complete examples with proper endpoint and header configuration
- Clear section organization with descriptive headings
- Educational value is excellent for developers implementing fallback TTS
examples/agent/fallback_tts/main.py (3)
1-27: Well-structured example with clear documentation.The file header and imports are properly organized with excellent documentation explaining the purpose and scope of the fallback TTS feature. The global warning variable pattern is acceptable for example code.
102-143: Realistic and educational provider configuration.The TTS provider setup demonstrates best practices:
- Primary/fallback provider strategy with different types (Deepgram/OpenAI)
- Proper custom endpoint configuration with authentication headers
- Realistic model selections and voice parameters
- Clear explanatory comments and console output
145-162: Proper connection management and user interaction.The WebSocket connection lifecycle is correctly managed with comprehensive error handling and clear user feedback. The example maintains simplicity while demonstrating all necessary aspects of the fallback TTS feature.
tests/unit_test/test_unit_agent_speak_provider.py (1)
20-359: Comprehensive test coverage for multi-provider functionality.The test suite excellently covers all aspects of the new feature:
- Backward compatibility: Single provider format tests ensure existing code continues working
- New functionality: Array provider tests validate multi-provider configurations
- Data integrity: Round-trip serialization/deserialization tests ensure no data loss
- Edge cases: Custom attributes, multiple endpoints, and mixed configurations
- Dynamic updates: UpdateSpeakOptions testing for runtime configuration changes
The test organization is logical with clear class separation by functionality and descriptive test method names.
jpvajda
changed the title
jpvajda
changed the title
Contributor
Author
|
I built a simple test app locally to test this out, everything looks good. ✅ 🎉 Excellent! All tests passed successfully! The test app worked perfectly and validated all the agent speak provider functionality: Test Results Summary:✅ All 6 tests PASSED:
Key Validations:
Future Steps: The agent speak provider functionality is now fully tested and validated. The code works correctly beyond the unit tests, supporting both single and array provider formats, proper API key handling, and dynamic provider updates. You can now confidently use this functionality in production applications. |
naomi-lgbt
approved these changes
Jul 10, 2025
lukeocodes
reviewed
Jul 11, 2025
lukeocodes
left a comment
lukeocodes
left a comment
Contributor
There was a problem hiding this comment.
I've reviewed this again. I think it is possible this is technically breaking, despite it supporting the existing interface
I have requested some input from the Python community. I will update again when I know more.
lukeocodes
approved these changes
Jul 11, 2025
lukeocodes
left a comment
lukeocodes
left a comment
Contributor
There was a problem hiding this comment.
The interface changing would technically break type-checking if it was built to be used in that way.
But seeing as this is our internal types, and the tests specifically test the old and new way of defining providers, this is a good change.
LGTM.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Types of changes
Agent speak now supports:
{"provider": {...}}[{"provider": {...}}, {"provider": {...}}]What types of changes does your code introduce to the community Python SDK?
Put an
xin the boxes that applyChecklist
Put an
xin the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code.Further comments
Summary by CodeRabbit
New Features
Documentation
Tests