[MichalZelasko]

Changes

Summary

Generated images, speech and video are now streamed rather than loaded, on the API and in the console's BFF, and a Range request reads only its range. Each generated file is recorded, counted against a per-organization quota, and deleted with its conversation, after a retention period, and with its organization.

Business Context

GET /api/v1/generated/{filename} read the whole file into memory before answering, Range requests included, and the BFF buffered it again with arrayBuffer(). A generated video can be up to 300 MiB, and the route has no rate limiter. So a member who knew a clip's name could make every concurrent request hold the whole clip in both processes. Generated files were also recorded nowhere, so deleting a conversation left its media in storage, with no quota and no retention. Raised by the Codex security review on #2265 and #2267.

Changes

Streaming reads

• BaseFileStorage gains size() and open_range(path, start, end). LocalFileStorage stats the file, then reads the span from disk in STREAM_CHUNK_BYTES chunks on the file pool. S3FileStorage uses HeadObject for the size and a ranged GetObject (Range: bytes=a-b) for the read, through the existing chunked reader.
• GET /api/v1/generated/{filename} calls open_generated_media (storage path and size, scoped to the caller's organization) and _workspace_bytes.streamed_file_response. That function decides the span before anything is read and returns a StreamingResponse with Content-Length. Content type from the suffix allowlist, inline vs download, nosniff, CSP: sandbox, Accept-Ranges, Content-Range, the 416 answer and "a multi-range is answered whole" behave as before. Header building is shared with file_response, which keeps its range parameters for callers that already hold the bytes. A file deleted between the size and the read is still a 404.
• The tenant boundary is unchanged: the directory comes from ctx.organization_id, and a leaf name with a separator is a 404. load_generated_media keeps its signature and scoping for callers that need the bytes, such as the video tool's first frame and Generated images, speech and video do not load on hosted pages and embeds #2309.
• The BFF route (frontend/src/app/api/generated/[filename]/route.ts) passes response.body through. It also forwards Content-Length (only when the body was not content-encoded) and X-Content-Type-Options: nosniff. Range, 200/206/416 and the other headers are unchanged.

Bounded and retained

• New generated_media table (migration 0160_generated_media.py) with: organization (CASCADE), conversation (SET NULL, linked only when the run's conversation belongs to the same organization), leaf name (unique per organization), storage path, size, and created_at.
• store_generated now calls keep_generated_media. It saves the bytes and records the row in a session of its own. If the row cannot be written, or the call is cancelled first, it removes the bytes again (shielded, and a failed unlink never replaces the cancellation).
• GENERATED_MEDIA_QUOTA_MB (default 5120, 0 = off). quota_refusal checks it in generate_image, generate_speech and generate_video before the paid call. Over the quota, the tool answers in plain text and names the setting an operator can raise. Admission is serialized per organization. Under an advisory lock, a call reserves its estimated size in generated_media_reservations before the paid call, so concurrent runs cannot all pass the check. The estimates are 16 MiB for an image, 8 KiB per character for speech, and 2 MiB/s for video, capped at 300 MiB. The stored file replaces the reservation. A failure or cancel releases it, and a reservation left by a crashed call stops counting after an hour and is swept. Every row counts, including media made outside a conversation: previews, test runs, channel /btw asides, and API runs that name no conversation.
• GENERATED_MEDIA_RETENTION_DAYS (default 90, 0 = off): a new daily sweep-generated-media job. Rows with no conversation also go after a fixed 7 days (UNLINKED_MEDIA_LIFETIME), even with retention off.
  • Each batch commits in its own worker session, so a timeout keeps the batches already done.
  • A (created_at, id) cursor reads past files the store refuses to delete. Those keep their rows and are retried on the next run.
  • A 10-minute time budget stops new batches before the 15-minute job timeout.
• Deletion is queued as a job in the deleting transaction (queue_external_state_cleanup), not best-effort after the commit:
  • Conversation, project and a2a deletion: GeneratedMediaService.forget_conversations drops the rows and queues their paths in required_paths. A path the store refuses fails the job, which then retries.
  • Account deletion: forget_user runs after project chats are handed to the team, and covers the conversations that cascade with the person.
  • Organization deletion: the generated_<org> prefix rides the purge's cleanup job (new CleanupExternalState.prefixes). A prefix the store does not fully remove raises, so the job retries; both delete_prefix backends now report a partial delete.
  • The retention sweep's conversation purge unlinks generated media and drops its rows too.
• An empty prefixes or required_paths is left out of the job payload, so a worker from the previous release still reads every cleanup job that has no prefix.
• data-protection report no longer exempts generated_. Generated files now have rows, so the report lists unrecorded files there: mostly files from before this table.
• Docs: data-protection.md (table row, "Generated media" section, deletion and controls rows, the erasure table), configuration.md (new settings) and reference/capabilities.md, with pl/de/es translations restamped. CHANGELOG: streaming under Fixed; quota, retention and sweep under Added, with an operator note.

BREAKING CHANGE (MINOR release, per AGENTS.md): by default, generated media is now deleted after 90 days (7 days for media made outside a conversation), and generation is refused past 5 GiB per organization. To keep the previous behaviour, set GENERATED_MEDIA_RETENTION_DAYS=0 and GENERATED_MEDIA_QUOTA_MB=0. Media made outside a conversation still goes after 7 days. Files generated before the upgrade have no row, so they are neither counted nor swept. Upgrade workers together with the API: an organization purge's cleanup job carries prefixes, which a previous-release worker cannot read, so do not roll back while such a job is queued.

Verification

• cd backend && uv run pytest tests/test_generated_media.py tests/api/test_generated_media_routes.py tests/test_workspace_bytes.py tests/test_image_generation.py tests/test_video_generation_capability.py tests/test_text_to_speech_capability.py tests/test_retention.py tests/test_services_conversation.py tests/test_project_service.py tests/test_file_storage_s3.py tests/test_worker_main.py tests/test_coverage_gate.py tests/test_media_offload.py tests/api/test_workspace_routes.py tests/integration/test_generated_media.py -q: 534 passed, 12 skipped. The skipped tests are the new integration file, because there was no database.
• The two fix commits ran 803 and 670 targeted tests respectively, all passing.
• With --cov=app, app/services/generated_media.py, app/worker/tasks/generated_media_tasks.py, app/api/routes/v1/generated_media.py, _workspace_bytes.py and the touched capability modules are at 100%. app/repositories/generated_media.py and the model's __repr__ are covered only by tests/integration/test_generated_media.py, which CI runs.
• cd frontend && bunx vitest run src/app/api/binary-routes.test.ts: 72 passed. This includes a test where the upstream body never ends: a route that read the whole body would hang there.
• make lint-backend and make lint-frontend pass.

Not verified locally:

• No database was available. The migration, make db-check, test_migrations.py and the integration tests (repository SQL, the SET NULL and CASCADE behaviour, the retention union query) were not run. CI runs them.
• Ranged S3 reads were not tested against a real bucket or MinIO, only against a mocked client.
• Memory use with a real 200 MB clip was not measured. The tests show that only the requested span is read and that the BFF does not wait for the end of the body.

Review: two rounds of structured review (correctness, security, quality, verification) found and fixed these:

• durable deletion;
• a sweep that rolled back on timeout, or stalled on one stuck file;
• a cancelled save that left a file behind;
• account erasure;
• the quota exemption;
• skew between job versions;
• partial prefix deletes.

Each fix has a regression test (test_generated_media.py, test_external_state_cleanup.py, test_file_storage*.py, test_org_purge_reservation.py, test_services.py).

Limitations:

• Files generated before this change have no row. They are still served, but they are not counted or swept and they go only with their organization. The data-protection report lists them.
• Account erasure reaches media through the person's conversations. Media they made with no conversation goes after 7 days, not at once, because the table has no user column.
• A file the store still refuses after the cleanup job's last attempt is left without a row, and the data-protection report lists it.
• Pre-existing and left for follow-ups: the retention purge reads the expiring conversations twice. A queued API run also discards the conversation it opens.

Linked Issues

Closes #2310
Refs #2346

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-22T12:47:52.474427Z	fe09fa5	Manual request
🔒 Security Review	✅ Completed 2026-10-22T12:45:16.887528Z	fe09fa5	Manual request

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d8008ce5c3

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[MichalZelasko]

@codex review

[chatgpt-codex-connector]

🛡️ Codex Security Review · Automatically triggered

Security review completed. No security issues were found in this pull request.

Reviewed commit: fe09fa51e0

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

• You comment "@codex security review"
• A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: fe09fa51e0

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".