Repository navigation
feat(media): agents speak and make video, with per-clip choices - #2346
Merged
Merged
Conversation
Groundwork for text to speech and video generation, which store their clips like generated images and are sold per character and per second. - Generated media is no longer image-only: save_generated_media takes the file name, so a clip is typed by its extension like an image is. - MP3, WAV, Ogg, Opus, M4A, MP4 and WebM join the inline allowlist, so the file viewer plays them instead of offering a download. Media cannot execute, and nosniff plus the sandbox CSP still apply. - The ledger books a unit-priced call (record_ambient_cost), and the affordability checks take a known cost, refusing a call that would carry the run past a cap rather than only one made after it. The viewer loads bytes as a blob, so seeking works without Range support on the generated-media route. Part-of: #2259 Refs: #2260
What a text-to-speech binding may choose, as data: OpenAI, ElevenLabs and Gemini, each with its models, the formats it returns and the voices every account has. GET /providers/text-to-speech-models serves it to the Builder, gated like the image catalog. - A provider is a vault purpose rather than a model provider, so ElevenLabs, which runs no chat model, joins services.json. - ElevenLabs' shared voices expire on 2026-12-31, so its list is empty and a binding names one of the account's own voices by id. - Prices are per million characters and dated: Gemini's double on 2027-01-01, and the budget has to see that on the day it happens. Part-of: #2259
The provider clients behind text to speech, one per API shape the catalog names, and synthesize(), which turns a text of any length into one clip. - A text over a model's limit is cut at a paragraph, sentence or word boundary and spoken in order. MP3 pieces concatenate; WAV is spoken as 24 kHz PCM by every provider and wrapped in one header, since two WAV files back to back are not one. - The key travels only in a header. An ElevenLabs voice id is encoded as one path segment, so a binding's value cannot reach another endpoint of the account. - A failure says what kind it was (key, unknown voice, refused text, rate limit, unavailable) and never the provider's own words, which are logged where they arrive. Pieces already spoken are discarded. - httpx only; no provider SDK is added. Part-of: #2259
…ech-foundation # Conflicts: # CHANGELOG.md # docs/channels.de.md # docs/channels.es.md # docs/channels.pl.md
…t-to-speech-catalog
…o-speech-providers
A generate_speech tool that reads a text aloud through the provider, model and voice an agent's author chose, and behaves for the user the way generate_image does: the clip is stored per organization, played in the conversation, opened in the file viewer and written under output/ when a workspace is open. - Configuration is validated against the catalog: the model, the voice the model can speak in, and a format the provider returns. ElevenLabs needs a voice id, since its voices belong to the account. - The key is required when provider is any catalog provider, so the Builder narrows the picker to the chosen provider's keys (#2258). - The budget is asked with the clip's per-character price before the call; the cost is booked after, and on a partial failure the pieces already spoken are booked too. - Failures go to whoever can fix them: bad or refused text and a provider that is down are steered; a refused key, an unknown voice and a spent budget are answered. - Storage and the workspace copy move into capabilities/_generated.py, shared with image generation, so both behave identically. - Builder: the provider and model selects are shared with the image panel (provider-model-fields.tsx), plus a voice select, or a voice id field for ElevenLabs. Chat: an audio step kind and a player. - Docs in four languages (capability reference, glossary, data protection), CHANGELOG. Closes: #2259
What a video-generation binding may choose, as data, and the clients that make one clip through it: Google Gemini Omni, ByteDance Seedance on BytePlus ModelArk, xAI Grok Imagine and Alibaba Wan 3.0. - Each model states what it can make - its length range, aspect ratios, resolutions, whether clips carry sound and whether it starts from an image - and a dated per-second price per resolution. GET /providers/video-models serves it, gated like the other media catalogs. BytePlus joins services.json as a vault purpose. - generate_video() checks a request against that, then submits it. Gemini answers in the call and hands back a Files API file; the others take a job that is polled until it settles. A job past the binding's deadline is cancelled where the provider allows it (BytePlus, while queued) and the call times out. - The clip is downloaded before the provider's address expires, capped in size, and the key is never sent to a storage host. - Failures say what kind they were - key, content policy, refused request, rate limit, unavailable, timed out - and never carry the provider's own words. Part-of: #2260
A generate_video tool that makes a short clip through the provider and model an agent's author chose, and behaves for the user the way generate_image does: the clip is stored per organization, played in the conversation, opened in the file viewer and written under output/ when a workspace is open. - Configuration is validated against what the chosen model can make: its resolutions, aspect ratios and length range. A usual length of five seconds is clamped to the model; Gemini Omni decides its own. - A clip can start from an image: a file this organization generated (generate_image's filename) or a workspace path, PNG, JPEG or WebP. - The budget is asked with the clip's per-second price before the job is submitted - at the model's longest where it decides its length - and the length the provider reports is booked after. - Failures go to whoever can fix them: a prompt, length, shape or image the model can fix, a content-policy refusal and a provider that is down are steered; a refused key, a spent budget and a job past the time limit are answered, since a retry is billed again. - The key picker narrows to the chosen provider (#2258), as for speech. - Builder: the shared provider and model fields plus resolution and aspect-ratio selects from the chosen model. Chat: a video step kind and player. BytePlus gets its brand mark. - Docs in four languages (capability reference, glossary, data protection), CHANGELOG. Closes: #2260
make audit reported 9 advisories against backend/uv.lock: eight against pillow 12.3.0 and one against certifi 2026.7.22. Every pull request failed the Security Scan on them. None of the Pillow 13 breaking changes reach the backend's image paths (uploads, workspace thumbnails, channel chart PNGs): they save RGB, not I mode, and use no removed API. The licences are unchanged, so only the versions in THIRD_PARTY_NOTICES.md move. Verified: make audit is clean, license_inventory.py check passes, and 742 image-related tests pass on Pillow 13, including the chat-attachment-formats integration suite. Closes #2279
Review fixes for the generated-media groundwork.
- GET /generated/{filename} answers a Range request with 206 and the
slice, and the Next.js proxy passes the range both ways. A player
points straight at that address, and WebKit - Safari and the macOS
desktop app - plays an MP4 only from ranged responses. A multi-range
or another unit gets the whole file; a range past the end gets 416.
- docs/configuration.md (and pl/de/es) and three frontend comments
still said only images and PDFs are served for display.
- The affordability check notes that asking reserves nothing, so
concurrent priced calls can overshoot, as RunSpendState accepts.
Part-of: #2259
…t-to-speech-providers
Review fixes for the speech provider clients. - A failure is classed by the provider's reason as well as its status: Gemini answers a bad key with 400 API_KEY_INVALID and ElevenLabs a spent quota with 401 quota_exceeded, which read as "text refused" and "bad key". 408 is unavailable, not a refused request. The body is still only logged. - synthesize() refuses a voice the model does not speak in, and an account voice id that is not [A-Za-z0-9_-]+: "." or ".." in the ElevenLabs path is resolved away and sends the key elsewhere. - An empty answer, or PCM with an odd byte count, is unavailable rather than a silent or corrupted clip. - A long text is cut after Chinese and Japanese sentence marks too, and MP3 pieces after the first lose their ID3v2 tag when joined. - Drop the unused is_offered(); pin that every catalog API shape has a client; correct what TIMEOUT_SECONDS bounds. Part-of: #2259
Review fixes for the video_generation capability. - first_frame reaches the organization-wide generated store only for an image generate_image returned in this conversation, by parsed result rather than any mention. An agent driven by a channel visitor could otherwise send another chat's image to a provider by its file name. - The starting image is typed by its bytes (sniff_image_header), a large workspace file is refused from its stat before it is read, and a refused or failed workspace read is said as such rather than as "no image called". - The usual length is clamped to the binding's longest clip, so a cap below five seconds no longer refuses every call without a length; an explicit 0 is refused rather than read as none. - A length or longest clip is refused for a model that decides its own length, and a sound setting where the model's sound is fixed; the Builder hides those fields and offers sound as provider default, on or off. - One clip at a time (sequential), since the price check reserves nothing. Docs and the data-protection label say video. Part-of: #2260
make audit-frontend failed on GHSA-587h-7wmj-h5pm (high): vite 8.2.0, which vitest and @vitejs/plugin-react pull in, bundles an affected launch-editor. 8.2.4 is inside the ranges both already ask for, so only frontend/bun.lock changes; vite brings rolldown 1.2.14 with it. Verified: make audit-frontend passes, make lint-frontend passes, and 125 vitest files (1475 tests) pass on vite 8.2.4. Refs #2279
Merges the provider-client fixes and applies the review fixes for the text_to_speech capability. - synthesize() reports each piece as soon as it is spoken (on_spoken) and the tool books it then, so a failure or a cancelled run partway through a long text leaves nothing the provider billed unbooked. This replaces SpeechProviderError.spoken_characters. - One clip at a time (sequential): the price check reserves nothing. - instructions is capped at 1,000 characters; it is unpriced and sent with every piece. The data-protection row now names it. - An ElevenLabs voice id must be [A-Za-z0-9_-]+ at the binding too. - Catalog formats are mp3 or wav only, refused at load otherwise. - A refused request is worded as text, voice or setting, not text. - SpeechCall keeps its api_key out of its repr. - Builder: a model change drops a voice the new model lacks (marin on tts-1 was kept and refused at publish), and the file format is a select over the formats the provider returns. - Channels say what a speech call is doing while it runs. - README says publish does not yet enforce the key's purpose (#2258) and no longer claims a dropped model is answered as a result. Part-of: #2259
…eo-generation-providers
Merges the text-to-speech fixes and applies the review fixes for the video provider clients. - Downloads follow redirects by hand: httpx strips only Authorization on a cross-host redirect, so Gemini's x-goog-api-key reached the storage host. The key goes to the first address only, every hop must be https, and a provider's video address is parsed before use. - A refused storage download is unavailable, not a refused key. - The download ends a grace period after the job's deadline, and a status read never waits past it. - A status read that times out, cannot connect, or is answered 5xx or 429 is read again until the deadline instead of abandoning a job that is still running and billed. - A single slow request before the deadline is unavailable, not a job timeout; a Gemini 400 API_KEY_INVALID is a refused key; 408 is unavailable. - A nested field of the wrong shape is unavailable rather than an AttributeError; Alibaba is polled every 15 s, as it asks. - VideoCall keeps its key out of its repr; the download buffers into one bytearray. docs/secrets.md lists ElevenLabs and BytePlus. Part-of: #2260
Resolves the overlapping data-protection rows, message keys and tool-card imports, keeps the video field copy equal to its schema, and lets channels say what a video call is doing while it runs. The BytePlus row in docs/secrets.md now links to the capability section. Part-of: #2260
…eat/2259-text-to-speech-foundation # Conflicts: # CHANGELOG.md
…t-to-speech-providers
…-to-speech-capability
…eo-generation-providers
…deo-generation-capability
Re-review fixes for the speech provider clients. - ElevenLabs answers a voice the account no longer has with 400 voice_not_found; it was a refused request, which tells the model to rephrase. It is now not-found, which tells the author to choose another voice. 402 (a plan the account lacks) is the key owner's to fix, like a refused key. - At most 200 characters of a provider's error body are logged: enough for its error code, short of the text it may echo back. - CHANGELOG names the new text-to-speech catalog endpoint. Part-of: #2259
Re-review fixes for the generated-media groundwork. - A Range number is at most 18 digits: Python's int() refuses one past 4,300 digits, so a 5 KB header was a 500 with a stack trace. Such a header is no range to honour and gets the whole file. - The proxy passes a backend 416 through with its Content-Range rather than turning it into FILE_NOT_FOUND for a file that exists. - Accept-Ranges is sent only by a route that serves ranges; the workspace, conversation and project routes do not pass one on. - CHANGELOG says a generated clip answers a player's byte ranges. Part-of: #2259
…t-to-speech-providers
Merges the provider and groundwork fixes, and applies the re-review fixes for the text_to_speech capability. - Changing the model on ElevenLabs dropped the author's voice id: the catalog lists no account voices, so the id was never "offered" by the new model. A provider whose voices belong to the account now keeps it across its models. - A test pins that clips are spoken one at a time; without the flag the budget check, which reserves nothing, could be passed together. - on_spoken books each piece whose audio came back. The docstrings no longer claim a piece in flight at cancel, or answered with something that is not audio, is booked. - One voice-id pattern, exported by the service and read by the binding validator, instead of two copies. - The capability reference (en/pl/de/es) says pieces are booked as they come back, calls run one at a time, and instructions is capped at 1,000 characters and unpriced. CHANGELOG wording follows. Part-of: #2259
…eo-generation-providers
Merges the text-to-speech fixes and applies the re-review fixes for the video provider clients. - The Gemini key stays on a redirect that keeps the first address's scheme, host and port, and is dropped for good at the first that does not. Dropping it on every hop failed a same-host redirect. - One classifier for what is transient: timeouts, connect, read and protocol errors, 408, 429, 500, 502, 503, 504, and a status read that is not JSON. A status read or a download retries those until its deadline - a finished clip is billed - and fails at once on anything else, so a proxy refusal no longer waits out the job. - Failures are classed by status before body: an outage or a rate limit that echoes "invalid api key" is not a refused key, and "case-sensitive" is not a content-policy refusal. - An xAI job done with no address field is malformed, not moderated; an inline Gemini clip is bounded before it is decoded; a host that trickles bytes past the grace period times out. - Downloads are kept out of tracing: a signed clip address in a span lets anyone with the traces fetch the clip. - Logged provider bodies and codes are capped at 200 characters; the cause of a job's timeout is logged; redirect exhaustion says so. Part-of: #2260
RR-3a71: a hosted page's socket now signs a media result only when the file it names is in the run's own AgentDeps.generated_files, read off the run as it starts streaming (EmbedSession._turn). The tool name no longer decides it, so an MCP tool or a renamed one called generate_image cannot mint a link to a file the run did not store. Publish also refuses renaming another tool onto generate_image, generate_speech or generate_video; MCP-prefixed names are not visible at publish, which the stored-set authority covers. RR-b5c2: the public generated-media route is rate-limited per signed link (by digest), 120 fetches a minute for ranged playback, and listed in test_rate_limited_surfaces. The frontend proxy passes a 429 and its Retry-After through. docs/security.md row updated in all four locales. RR-e94d: setup_logging puts the redaction filter on the uvicorn.access and uvicorn.error loggers and their handlers, so a signed ?token= in an access line is redacted whichever order logging is configured in. docs/security.md says reverse-proxy access logs still carry the tokens. Nitpicks: C2 is moot (_media_calls is gone); Q2 the bare HMAC's reason is documented; Q3 payloads are read through parse_generated_* with unknown keys kept; Q4 the backend url is documented as for a client driving a page's socket directly.
Review fixes for #2396. RR-4c19: deleting generated bytes is a job queued in the deleting transaction instead of a spawn_after_commit task. CleanupExternalState gains a defaulted `prefixes` field (a job queued by an older release still parses); the org purge puts generated_<org> into the job it already queues, and a prefix the store refuses fails the job so its retries re-run it. Conversation and project delete (GeneratedMediaService.forget_conversations) drop the rows and queue the paths through queue_external_state_cleanup. Generated media is now row-backed, so generated_ leaves UNREFERENCED_BY_DESIGN and GeneratedMedia.storage_path joins MEDIA_PATH_COLUMNS (its coverage test was failing). Files from before the table have no row and are reported as such, documented in docs/data-protection.md. RR-7d02: the sweep runs each batch in its own worker session, committed before the next is read, and starts no batch after a 10-minute budget (job timeout is 15), so a timeout keeps the work already done. RR-a8e6: no break on a partial batch; expired() reads past a (created_at, id) cursor, so stuck rows are skipped and retried next run. RR-1f5b: keep_generated_media unlinks on BaseException, shielded. RR-6e30: UserService.delete drops the generated rows of the person's cascading conversations (after project chats are handed on) and queues their unlink; the DELETE /users/{id} row says so in four locales. RR-c247: bytes_used counts only rows with a conversation; the refusal names GENERATED_MEDIA_QUOTA_MB and the deployment operator. RR-58d1: docstring and configuration docs say the quota may be passed by up to one file per call in flight. RR-9b7a: CHANGELOG splits streaming (Fixed) from quota, retention and sweep (Added), with the opt-out for operators. Typed feat here; whether the new defaults warrant ! and BREAKING CHANGE is the maintainer's call at squash. RR-0e9d: S3 size() reuses _head_blocking, which returns the length. RR-f41a: comment on why streamed_file_response is not FileResponse. RR-2a6c not done (optional); RR-d3f8 left for its own issue.
Second-round review nitpicks on #2395: - S-r2-1, N2: the token expiry now forbids a leading zero ([1-9][0-9]{0,11}) in the backend `_TOKEN` and the frontend proxy's TOKEN_PATTERN, so `0<expiry>` no longer opens the file and one link cannot claim a fresh rate-limit bucket per zero. The `embed_generated_media_allowed` docstring now says why the link cannot be varied, and the rate_limit module docstring no longer claims every surface reads a row first; it is rewrapped to its neighbours' width. - S-r2-2: `limit_embed_generated_media` counts a token only when `token_opens` accepts it for the path's key and file. A forged, expired or padded token falls through to the route's same 404 and spends no Redis key. - C-r2-1: the changelog entry on renames onto media tool names covers rollback and the draft, and says what to do. - C-r2-2, N3: docs/channels.md and its pl/de/es versions describe the trigger as a generated image, speech or video result naming a file the run stored, not three tool names; the long line is rewrapped and the translations restamped. Tests: a zero-padded token is refused (service and vitest), and a forged, padded or other-file token is not counted by the limiter.
RR-6b14: CleanupExternalState leaves an empty `prefixes` out of its dump (`exclude_if`), so a previous-release worker, whose payload forbids extra fields, still runs every cleanup job without prefixes. An org purge's prefix job needs the new worker; CHANGELOG tells operators to upgrade workers with the API and not roll back while one is queued. RR-8c3e: media with no conversation counts against the quota again. The sweep deletes it after UNLINKED_MEDIA_LIFETIME (7 days), whatever GENERATED_MEDIA_RETENTION_DAYS is, 0 included. `expired` takes an `unlinked_cutoff` and an optional `cutoff`. Docs name previews, /btw asides and API runs with no conversation. RR-2d97: a partial prefix delete raises - local when the directory is still there after rmtree, S3 (ExternalServiceError) when delete_objects lists Errors. The best-effort wrappers still log; the cleanup job fails and retries. RR-3c58: a storage error in keep_generated_media's cancel cleanup is logged, so the CancelledError goes on; the double-cancel gap is noted. RR-7e21, RR-a4d6, RR-e07c, RR-91bf: data-protection docs (4 locales) record that a failed per-file unlink is logged, not retried; the orphan note says "mostly" pre-table files; the stale attachment sentence now matches the deletion table; the account-deletion "Leaves" cell names media made with no conversation. Refs #2310
The shipped Nginx config wrote the default combined access log, which records a hosted page's media link, an embed socket's and an artifact link's ?token= in clear. It now logs $uri without the query string. Traefik's access log cannot drop only the query, so docs/security.md still names it as one to keep as holding live credentials.
CI on 67b2e9e failed two gates. The coverage gate missed the branch of SecretRequirement.purpose_for where the gating field is empty. And the secret-picker e2e spec picked the condition's option by its raw value: image generation, now gated on its provider (#2258), lists openai as OpenAI. That left the shared draft agent unpublishable for the MCP spec after it. The option is now matched case-insensitively.
Addresses three Codex review comments on #2396. P1, quota admission not serialized: each call read SUM(size_bytes) in its own transaction and recorded its file only after the paid call, so concurrent calls all passed. Admission now takes a per-organization advisory lock (LockScope.GENERATED_MEDIA_PER_ORG), measures recorded bytes plus the claims of calls in flight against the quota with this call's estimate, and writes a claim to the new generated_media_reservations table. Recording the file replaces the claim in the same transaction; a refusal, failure or cancellation releases it (MediaSlot, shielded); a claim older than an hour stops counting and the daily sweep removes it. Estimates: 16 MiB per image, 8 KiB per character of speech, 2 MiB per second of video capped at the clip download limit. P1, generated-file unlinks not retried: a deleted conversation's or account's generated media now rides CleanupExternalState.required_paths, whose failures raise so the job retries. The field is defaulted and left out of the dump when empty, like prefixes. P2, filename unique across the deployment: the unique constraint is now (organization_id, filename). No repository query looks a row up by filename alone. Migration 0160_generated_media.py is unreleased and edited in place.
Since #2258 gates image generation's key on its provider, a binding somebody only switched on stores no provider, so the picker stayed hidden while the panel showed the catalog's first provider and publish demanded its key. The panel now passes that provider as resolved config, as the speech and video panels do. The e2e secret spec opens the condition only when the picker is not already offered.
… into fix/2309-hosted-generated-media # Conflicts: # CHANGELOG.md
… into fix/2310-stream-generated-media # Conflicts: # CHANGELOG.md # backend/app/agents/capabilities/text_to_speech/_toolset.py # backend/app/agents/capabilities/video_generation/_toolset.py # docs/reference/capabilities.de.md # docs/reference/capabilities.es.md # docs/reference/capabilities.pl.md
platform-proxy.test.ts holds every hand-rolled BFF proxy to encoding the segments it interpolates into the backend URL. The hosted-media proxy checked them against anchored patterns first but interpolated them bare, which failed test-frontend on 4bcb966.
test_security_marker.py failed on a96d5c2: three new tests that prove another tenant's generated media is out of reach matched the 'tenant' keyword without @pytest.mark.security, and the sweep's time budget test matched 'budget' by coincidence.
CI on 6ab0f20 failed in two places this PR reaches. The platform route guard requires every route reachable without a session to be declared with its reason, and the hosted-media route was not. And the surface-transcript tests' stand-in run had no ctx, which the hosted page now reads its run's stored files from.
fetchConversations read window.location after awaiting the coalesced list invalidation. The sidebar calls it from a mount effect without awaiting it, so when a test file finished inside that window the read ran against a torn-down jsdom and vitest failed test-frontend with an unhandled 'window is not defined' (seen on #2395). The id is now read when the call is made, which is also the address it was made for.
… into fix/2309-hosted-generated-media
… into fix/2310-stream-generated-media
…-and-video # Conflicts: # CHANGELOG.md
… into fix/2309-hosted-generated-media
… into fix/2310-stream-generated-media
… into feat/2259-2260-speech-and-video # Conflicts: # CHANGELOG.md # backend/pyproject.toml
#2395 added the hosted-page route reading the whole file for each request, which #2310 had just removed from the console's route: a link may be fetched 120 times a minute, so a long clip was held once per range. The service now opens the file (path and size) and the route streams only the range asked for, with the same 404 for a file gone between being measured and being opened.
…-and-video Brings in 1.13.1 to 1.15.0 (#2324 among them). Main took 0160 to 0169, so this branch's 0160_generated_media becomes 0170_generated_media on 0169_starters_per_membership. The registry, worker and capability tables keep both sides: speech and video beside follow-ups, the media sweep beside the proactive sweeps.
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
ElevenLabs had a mark among the services but none among the model providers ProviderIcon reads, so the text-to-speech picker drew a bare "E". It joins PROVIDERS, from lobehub like the others; black, so it keeps the text colour.
Merged
DEENUU1
added a commit
that referenced
this pull request
Oct 23, 2026
Promotes `[Unreleased]` to `[1.16.0] - 2026-10-23` and bumps `backend/pyproject.toml`, `backend/uv.lock` and `frontend/package.json`. Ships #2346 with #2395 and #2396 folded in: agents speak and make video; generated media load on hosted pages, are streamed, and are bounded by a quota. New capabilities, so MINOR.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
Summary
Add two capabilities:
text_to_speech, which reads text aloud through OpenAI, ElevenLabs or Gemini, andvideo_generation, which makes short clips through Gemini (Veo/Omni), BytePlus Seedance, xAI Grok Imagine or Alibaba Wan. Both are provider abstractions shaped like image generation. Each one stores what it makes in the organization's generated-file store, plays it inline in chat, and books its cost to the run budget.Business Context
Agents could write and draw, but they could not speak or film (#2259, #2260). Authors want the provider, model and spending decisions in the spec. The agent should still be able to pick a voice or a resolution per clip, within limits the author sets.
Changes
core/cataloggains the speech and video providers and their models, voices, resolutions, aspect ratios, lengths and dated unit prices.GET /providers/speech-modelsandGET /providers/video-modelsexpose them to the Builder.services/text_to_speechandservices/video_generationhold one client per provider.provider_error_code.py). The body never reaches the logs, the exception or the model._choices.py,schema_for). The tool schema is narrowed to what the chosen model makes, withinmax_resolutionand the allowedvoiceslist.max_resolutionmeans any resolution.voiceslist means any voice; on ElevenLabs that is any voice id in the account.can_afford_ambient_call). After it, the reported units are booked.submitted,made). Once a job was submitted, a timed-out or cancelled job is booked at its estimate.customand legacy keys are still accepted.alibaba_workspaceandalibaba_region. Wan 3.0 is then called athttps://{workspace}.{region}.maas.aliyuncs.com/api/v1; left empty, the binding keeps the shared DashScope host. The workspace must be a single DNS label, because it becomes part of the host the key is sent to.AgentDeps.generated_files). That set survives a park/resume. In a voice call, video is ahand_offcapability, and the twin shares the call's set.docs/reference/capabilities.md,docs/voice.md,docs/data-protection.mdanddocs/features.md, the READMEs, and both capability READMEs, each in four languages.Architecture
A clip is booked whenever the provider may have billed it, meaning on every path after
submitted. Only a refusal before submission is free.Verification
make lintpass, as doescheck_docs_i18n.py.UNAVAILABLE, and the MP3 length is read from the Xing header. Both are covered against recorded shapes, not against the live APIs.capabilities.webp,figures.webp) still say 29 capabilities and need a redraw; the text lists say 31.Linked Issues
Closes #2258
Closes #2350
Closes #2351
Closes #2095
Closes #2309
Closes #2310
Refs #2259, #2260
Folded in: #2395 and #2396 (2026-10-23)
This PR now carries both stacked PRs, so it lands and releases as one:
GitHub marks both as merged into this branch.
Found while putting them together and fixed here:
/embed/{public_key}/generated/{filename}read the whole file for every request. That is exactly what feat(media)!: stream generated media and bound what is kept #2396 removed from the console's route, and this route allows 120 fetches a minute per link. It now opens the file (path and size) and streams only the range asked for, like the console's route (26d7ce4f1).0160to0169in 1.15.0, so0160_generated_mediais now0170_generated_mediaon0169_starters_per_membership. The chain was applied, reverted and reapplied on a scratch pgvector database, withalembic checkclean.7829eda60).The sync with main kept both sides in the capability registry (speech and video beside follow-ups), the worker (the media sweep beside the proactive sweeps) and the capability art. Verified locally: 1128 unit tests and 71 integration tests across the merged areas, plus
make lint-backendandmake lint-frontend.