diff --git a/docs/API.md b/docs/API.md index 06bf6dd0..d597db55 100644 --- a/docs/API.md +++ b/docs/API.md @@ -343,6 +343,65 @@ player.addChapters([ ]); ``` +## `generateScenes(assetId, languageCode, options?)` + +Generates AI-powered scene boundaries by combining shot detection with timestamped transcript cues. Uses shot boundaries as candidate visual anchors, then groups adjacent shot windows into higher-level scenes with titles. + +**Parameters:** + +- `assetId` (string) - Mux video asset ID +- `languageCode` (string) - Language code for captions (e.g., 'en', 'es', 'fr') +- `options` (optional) - Configuration options + +**Options:** + +- `provider?: 'openai' | 'anthropic' | 'google'` - AI provider (default: 'openai') +- `model?: string` - AI model to use (defaults: `gpt-5.1`, `claude-sonnet-4-5`, or `gemini-3-flash-preview`) +- `promptOverrides?: object` - Override specific sections of the scenes prompt + - `task?: string` - Override the main scene segmentation task + - `outputFormat?: string` - Override the expected output format description + - `sceneGuidelines?: string` - Override scene count and formatting guidance + - `boundaryGuidelines?: string` - Override how shot anchors and transcript continuity should influence boundaries + - `titleGuidelines?: string` - Override scene title style guidance +- `minScenesPerHour?: number` - Minimum scenes to generate per hour of content (default: 6) +- `maxScenesPerHour?: number` - Maximum scenes to generate per hour of content (default: 20) +- `outputLanguageCode?: string | 'auto'` - Language for generated scene titles (default: transcript language) +- `pollIntervalMs?: number` - Poll interval passed to `waitForShotsForAsset()` +- `maxAttempts?: number` - Max poll attempts passed to `waitForShotsForAsset()` +- `createShotsIfMissing?: boolean` - Whether to request shots before polling (default: `true`) +- `minShotWindowDurationSeconds?: number` - Minimum duration before short low-signal shot windows are merged (default: `2`) + +**Returns:** + +```typescript +{ + assetId: string; + languageCode: string; + scenes: Array<{ + startTime: number; // Scene start time in seconds + endTime: number; // Scene end time in seconds + title: string; // Descriptive scene title + }>; + usage?: TokenUsage; // Token usage from the AI provider +} +``` + +**Requirements:** + +- Asset must be a video asset +- Asset must have a ready caption/transcript track in the specified language +- Workflow requires completed shots and will request/poll for them by default + +**Example Output:** + +```typescript +[ + { startTime: 0, endTime: 18.4, title: "Opening Introduction" }, + { startTime: 18.4, endTime: 47.2, title: "Product Overview" }, + { startTime: 47.2, endTime: 91, title: "Hands-On Demonstration" } +] +``` + ## `translateAudio(assetId, toLanguageCode, options?)` Creates AI-dubbed audio tracks from existing media content using ElevenLabs voice cloning and translation. Uses the default audio track on your asset. Source language is auto-detected unless `fromLanguageCode` is provided. diff --git a/docs/EXAMPLES.md b/docs/EXAMPLES.md index c158435d..d3fe6028 100644 --- a/docs/EXAMPLES.md +++ b/docs/EXAMPLES.md @@ -27,6 +27,10 @@ You can run examples directly from the project root without installing dependenc npm run example:chapters [language-code] [provider] npm run example:chapters:compare [language-code] +# Scenes +npm run example:scenes -- --language en --provider openai +npm run example:scenes:compare -- --language en + # Burned-in Caption Detection npm run example:burned-in [provider] npm run example:burned-in:compare @@ -72,6 +76,12 @@ npm run example:ask-questions:multiple abc123 "Does this video contain people?" # Compare OpenAI vs Anthropic chapter generation npm run example:chapters:compare abc123 en +# Generate scenes with broader segmentation +npm run example:scenes abc123 -- --language en --provider openai --broad + +# Compare scene generation across providers +npm run example:scenes:compare abc123 -- --language en + # Run moderation analysis with Hive npm run example:moderation abc123 hive @@ -183,6 +193,28 @@ npm run chapters:basic [language-code] [provider] npm run compare [language-code] ``` +## Scene Generation Examples + +- **Basic Scenes**: Generate shot-guided scene boundaries with titles +- **Provider Comparison**: Compare OpenAI vs Anthropic vs Google scene generation + +```bash +cd examples/scenes +npm install +npm run basic -- --language en --provider openai +npm run compare -- --language en +``` + +The basic example also supports: + +```bash +# Generate titles in another language +npm run example:scenes abc123 -- --language en --output-language es + +# Ask for broader, less granular sceneing +npm run example:scenes abc123 -- --language en --broad +``` + ## Caption Translation Examples - **Basic Translation**: Translate captions and upload to Mux diff --git a/docs/PROMPT-CUSTOMIZATION.md b/docs/PROMPT-CUSTOMIZATION.md index 1157fcee..2253f766 100644 --- a/docs/PROMPT-CUSTOMIZATION.md +++ b/docs/PROMPT-CUSTOMIZATION.md @@ -41,6 +41,7 @@ const result = await getSummaryAndTags(assetId, { | --- | --- | | `getSummaryAndTags` | `task`, `title`, `description`, `keywords`, `qualityGuidelines` | | `generateChapters` | `task`, `titleGuidelines`, `chapterGuidelines` | +| `generateScenes` | `task`, `outputFormat`, `sceneGuidelines`, `boundaryGuidelines`, `titleGuidelines` | | `hasBurnedInCaptions` | `task`, `positiveIndicators`, `negativeIndicators`, `confidenceGuidelines` | ## Presets: Common Override Patterns @@ -110,6 +111,18 @@ const result = await generateChapters(assetId, "en", { }); ``` +### Scene Boundary Style + +```ts +const result = await generateScenes(assetId, "en", { + provider: "openai", + promptOverrides: { + boundaryGuidelines: "Prefer broader scenes unless there is a clear narrative or setting change.", + titleGuidelines: "Use short cinematic titles under 5 words.", + }, +}); +``` + ## Advanced: Full Section Objects For more control, you can pass a full `PromptSection` object instead of a string to customize the XML tag name and attributes: diff --git a/docs/WORKFLOWS.md b/docs/WORKFLOWS.md index efa127cd..5300b812 100644 --- a/docs/WORKFLOWS.md +++ b/docs/WORKFLOWS.md @@ -246,6 +246,56 @@ player.addChapters(result.chapters); - Asset must have a ready caption/transcript track in the specified language - Uses existing auto-generated or uploaded captions/transcripts +## Scene Generation + +Generate AI-powered scene boundaries by combining shot detection with timestamped transcript cues. + +Unlike chaptering, scene generation is grounded in visual structure first. The workflow waits for Mux shot detection, aligns each shot span to overlapping WebVTT cues, and asks the model to group those windows into coherent scenes. This helps avoid treating every camera cut as a new scene while still using visual anchors to detect meaningful transitions. + +```typescript +import { generateScenes } from "@mux/ai/workflows"; + +const result = await generateScenes("your-mux-asset-id", "en", { + provider: "openai", + promptOverrides: { + boundaryGuidelines: "Prefer broader scenes unless the narrative clearly shifts.", + titleGuidelines: "Use short, cinematic titles under 5 words." + } +}); + +console.log(result.scenes); +// Array of {startTime: number, endTime: number, title: string} +``` + +### Requirements + +- Asset must be a video asset +- Asset must have a ready caption/transcript track in the specified language +- Asset must have available shots, or allow the workflow to create them + +### Prompt Customization + +`generateScenes` supports `promptOverrides` so consumers can steer segmentation behavior without replacing the whole workflow prompt. + +Available sections: + +- `task` +- `outputFormat` +- `sceneGuidelines` +- `boundaryGuidelines` +- `titleGuidelines` + +Example: + +```typescript +const result = await generateScenes(assetId, "en", { + promptOverrides: { + sceneGuidelines: "Favor fewer, broader scenes for continuous talking-head content.", + titleGuidelines: "Use concise editorial titles with no more than 4 words." + } +}); +``` + ## Embeddings Generate vector embeddings for semantic search over video or audio transcripts. diff --git a/examples/scenes/basic-example.ts b/examples/scenes/basic-example.ts new file mode 100644 index 00000000..cdc25fb6 --- /dev/null +++ b/examples/scenes/basic-example.ts @@ -0,0 +1,83 @@ +import { Command } from "commander"; + +import { secondsToTimestamp } from "@mux/ai/primitives"; +import { generateScenes } from "@mux/ai/workflows"; + +import "../env"; + +type Provider = "openai" | "anthropic" | "google"; + +const program = new Command(); + +program + .name("scenes") + .description("Generate scene boundaries for a Mux video asset") + .argument("", "Mux asset ID to analyze") + .option("-l, --language ", "Language code for transcript/captions", "en") + .option("-p, --provider ", "AI provider (openai, anthropic, google)", "openai") + .option("--output-language ", "Output language as BCP 47 code (e.g. 'fr', 'ja') or 'auto'") + .option("--broad", "Prefer broader scenes with fewer splits") + .action(async (assetId: string, options: { + language: string; + provider: Provider; + outputLanguage?: string; + broad?: boolean; + }) => { + if (!["openai", "anthropic", "google"].includes(options.provider)) { + console.error("Unsupported provider. Choose from: openai, anthropic, google"); + process.exit(1); + } + + console.log(`Generating scenes for asset: ${assetId}`); + console.log(`Language: ${options.language}`); + console.log(`Provider: ${options.provider}`); + if (options.outputLanguage) + console.log(`Output Language: ${options.outputLanguage}`); + if (options.broad) + console.log("Segmentation Style: broader scenes"); + console.log(); + + try { + const start = Date.now(); + + console.log("Generating scenes..."); + const result = await generateScenes(assetId, options.language, { + provider: options.provider, + outputLanguageCode: options.outputLanguage, + promptOverrides: options.broad ? + { + boundaryGuidelines: "Prefer broader scenes unless the narrative or setting clearly changes.", + titleGuidelines: "Use concise titles under 5 words.", + } : + { + titleGuidelines: "Use concise titles under 5 words.", + }, + }); + + const duration = Date.now() - start; + + console.log("Success!"); + console.log(`Duration: ${duration}ms`); + console.log(`Generated ${result.scenes.length} scenes\n`); + + console.log("Scene List:"); + result.scenes.forEach((scene, index) => { + console.log( + ` ${index + 1}. ${secondsToTimestamp(scene.startTime)} - ${secondsToTimestamp(scene.endTime)} | ${scene.title}`, + ); + }); + + if (result.usage) { + console.log("\nToken Usage:"); + console.log(JSON.stringify(result.usage, null, 2)); + } + + console.log("\nStructured Output:"); + console.log(JSON.stringify(result.scenes, null, 2)); + } catch (error) { + console.error("Error:", error instanceof Error ? error.message : error); + process.exit(1); + } + }); + +program.parse(); diff --git a/examples/scenes/provider-comparison.ts b/examples/scenes/provider-comparison.ts new file mode 100644 index 00000000..e50ac1ed --- /dev/null +++ b/examples/scenes/provider-comparison.ts @@ -0,0 +1,79 @@ +import { Command } from "commander"; + +import { secondsToTimestamp } from "@mux/ai/primitives"; +import { generateScenes } from "@mux/ai/workflows"; + +import "../env"; + +interface ProviderConfig { + name: string; + provider: "openai" | "anthropic" | "google"; +} + +const program = new Command(); + +program + .name("scenes-compare") + .description("Compare scene generation across multiple AI providers") + .argument("", "Mux asset ID to analyze") + .option("-l, --language ", "Language code for transcript/captions", "en") + .action(async (assetId: string, options: { + language: string; + }) => { + console.log(`Comparing scene generation for asset: ${assetId}`); + console.log(`Language: ${options.language}\n`); + + try { + const providers: ProviderConfig[] = [ + { name: "OpenAI", provider: "openai" }, + { name: "Anthropic", provider: "anthropic" }, + { name: "Google", provider: "google" }, + ]; + + const results: Array<{ + config: ProviderConfig; + result: Awaited>; + duration: number; + }> = []; + + for (const config of providers) { + console.log(`Testing ${config.name} scene generation...`); + const start = Date.now(); + const result = await generateScenes(assetId, options.language, { provider: config.provider }); + const duration = Date.now() - start; + + console.log("Results:"); + console.log(` Duration: ${duration}ms`); + console.log(` Generated scenes: ${result.scenes.length}`); + console.log(" Scene breakdown:"); + result.scenes.forEach((scene, index) => { + console.log( + ` ${index + 1}. ${secondsToTimestamp(scene.startTime)} - ${secondsToTimestamp(scene.endTime)} | ${scene.title}`, + ); + }); + if (result.usage) { + console.log(" Token usage:"); + console.log(` Input: ${result.usage.inputTokens ?? "n/a"}`); + console.log(` Output: ${result.usage.outputTokens ?? "n/a"}`); + console.log(` Total: ${result.usage.totalTokens ?? "n/a"}`); + } + console.log(); + + results.push({ config, result, duration }); + } + + console.log("\nProvider Comparison:"); + results.forEach(({ config, result, duration }) => { + console.log(`${config.name}: ${result.scenes.length} scenes (${duration}ms)`); + }); + + const allFirstTitles = results + .map(({ config, result }) => `${config.name}: ${result.scenes[0]?.title ?? "No scenes returned"}`); + console.log(`\nFirst Scene Titles: ${allFirstTitles.join(" | ")}`); + } catch (error) { + console.error("Error:", error instanceof Error ? error.message : error); + process.exit(1); + } + }); + +program.parse(); diff --git a/package.json b/package.json index 87c7fe23..972c337a 100644 --- a/package.json +++ b/package.json @@ -83,6 +83,8 @@ "example:ask-questions:multiple": "npx tsx examples/ask-questions/multiple-questions-example.ts", "example:chapters": "npx tsx examples/chapters/basic-example.ts", "example:chapters:compare": "npx tsx examples/chapters/provider-comparison.ts", + "example:scenes": "npx tsx examples/scenes/basic-example.ts", + "example:scenes:compare": "npx tsx examples/scenes/provider-comparison.ts", "example:burned-in": "npx tsx examples/burned-in-captions/basic-example.ts", "example:burned-in:compare": "npx tsx examples/burned-in-captions/provider-comparison.ts", "example:summarization": "npx tsx examples/summarization/basic-example.ts", diff --git a/src/workflows/index.ts b/src/workflows/index.ts index 00f9297e..39caa28a 100644 --- a/src/workflows/index.ts +++ b/src/workflows/index.ts @@ -3,6 +3,7 @@ export * from "./burned-in-captions"; export * from "./chapters"; export * from "./embeddings"; export * from "./moderation"; +export * from "./scenes"; export * from "./summarization"; export * from "./translate-audio"; export * from "./translate-captions"; diff --git a/src/workflows/scenes.ts b/src/workflows/scenes.ts new file mode 100644 index 00000000..d8e2f046 --- /dev/null +++ b/src/workflows/scenes.ts @@ -0,0 +1,676 @@ +import { generateText, Output } from "ai"; +import dedent from "dedent"; +import { z } from "zod"; + +import { getLanguageName } from "@mux/ai/lib/language-codes"; +import { + getAssetDurationSecondsFromAsset, + getPlaybackIdForAsset, + isAudioOnlyAsset, +} from "@mux/ai/lib/mux-assets"; +import type { PromptOverrides, PromptSection } from "@mux/ai/lib/prompt-builder"; +import { createLanguageSection, createPromptBuilder } from "@mux/ai/lib/prompt-builder"; +import { createLanguageModelFromConfig, resolveLanguageModelConfig } from "@mux/ai/lib/providers"; +import type { ModelIdByProvider, SupportedProvider } from "@mux/ai/lib/providers"; +import { withRetry } from "@mux/ai/lib/retry"; +import { resolveMuxSigningContext } from "@mux/ai/lib/workflow-credentials"; +import { waitForShotsForAsset } from "@mux/ai/primitives/shots"; +import { fetchTranscriptForAsset, getReadyTextTracks, parseVTTCues, secondsToTimestamp } from "@mux/ai/primitives/transcripts"; +import type { MuxAIOptions, TokenUsage, WorkflowCredentialsInput } from "@mux/ai/types"; + +// ───────────────────────────────────────────────────────────────────────────── +// Types +// ───────────────────────────────────────────────────────────────────────────── + +export const sceneSchema = z.object({ + startTime: z.number(), + endTime: z.number(), + title: z.string(), +}); + +export type Scene = z.infer; + +export const scenesSchema = z.object({ + scenes: z.array(sceneSchema), +}); + +export type ScenesType = z.infer; + +/** Structured return payload from `generateScenes`. */ +export interface ScenesResult { + assetId: string; + languageCode: string; + scenes: Scene[]; + /** Token usage from the AI provider (for efficiency/cost analysis). */ + usage?: TokenUsage; +} + +/** + * Shot-aligned transcript window used to scaffold scene generation. + * Exported for focused unit tests around temporal alignment logic. + */ +export interface SceneShotWindow { + startTime: number; + endTime: number; + transcriptText: string; + cueCount: number; + shotCount: number; +} + +/** + * Sections of the scenes user prompt that can be overridden. + * Use these to customize segmentation and titling behavior for your use case. + */ +export type ScenesPromptSections = + "task" | + "outputFormat" | + "sceneGuidelines" | + "boundaryGuidelines" | + "titleGuidelines"; + +/** + * Override specific sections of the scenes prompt. + * + * @example + * ```typescript + * const result = await generateScenes(assetId, "en", { + * promptOverrides: { + * titleGuidelines: "Use short, cinematic titles under 5 words.", + * boundaryGuidelines: "Prefer fewer, broader scenes unless the visual context clearly changes.", + * }, + * }); + * ``` + */ +export type ScenesPromptOverrides = PromptOverrides; + +/** Configuration accepted by `generateScenes`. */ +export interface ScenesOptions extends MuxAIOptions { + /** AI provider used to interpret the shot-aligned transcript windows (defaults to 'openai'). */ + provider?: SupportedProvider; + /** Provider-specific model identifier. */ + model?: ModelIdByProvider[SupportedProvider]; + /** Override specific sections of the user prompt. */ + promptOverrides?: ScenesPromptOverrides; + /** + * Minimum number of scenes to generate per hour of content. + * Defaults to 6. + */ + minScenesPerHour?: number; + /** + * Maximum number of scenes to generate per hour of content. + * Defaults to 20. + */ + maxScenesPerHour?: number; + /** + * BCP 47 language code for scene titles (e.g. "en", "fr", "ja"). + * When omitted, auto-detects from the transcript track's language. + * Falls back to the requested transcript language if no metadata is available. + */ + outputLanguageCode?: string; + /** Poll interval forwarded to `waitForShotsForAsset()`. */ + pollIntervalMs?: number; + /** Max poll attempts forwarded to `waitForShotsForAsset()`. */ + maxAttempts?: number; + /** Whether to request shot generation before polling (default: true). */ + createShotsIfMissing?: boolean; + /** + * Minimum duration for a shot window before it is likely merged into a neighbor. + * Defaults to 2 seconds. + */ + minShotWindowDurationSeconds?: number; +} + +const DEFAULT_PROVIDER = "openai"; +const DEFAULT_MIN_SCENES_PER_HOUR = 6; +const DEFAULT_MAX_SCENES_PER_HOUR = 20; +const DEFAULT_MIN_SHOT_WINDOW_DURATION_SECONDS = 2; + +interface ScenesAnalysisResponse { + scenes: ScenesType; + usage: TokenUsage; +} + +interface NormalizeScenesOptions { + scenes: Scene[]; + sceneStartCandidates: number[]; + assetDurationSeconds: number; +} + +// ───────────────────────────────────────────────────────────────────────────── +// Helpers +// ───────────────────────────────────────────────────────────────────────────── + +function normalizeWhitespace(value: string): string { + return value.replace(/\s+/g, " ").trim(); +} + +function roundToMillisecondPrecision(value: number): number { + return Math.round(value * 1000) / 1000; +} + +function clampTime(value: number, min: number, max: number): number { + return Math.min(Math.max(value, min), max); +} + +function formatSecondsForPrompt(value: number): string { + return `${roundToMillisecondPrecision(value)}s (${secondsToTimestamp(value)})`; +} + +function cueOverlapsWindow( + cue: { startTime: number; endTime: number }, + startTime: number, + endTime: number, +): boolean { + return cue.endTime > startTime && cue.startTime < endTime; +} + +function snapToNearestCandidate(value: number, candidates: number[]): number { + if (candidates.length === 0) { + return value; + } + + let nearest = candidates[0]; + let smallestDistance = Math.abs(value - nearest); + + for (const candidate of candidates.slice(1)) { + const distance = Math.abs(value - candidate); + if (distance < smallestDistance) { + nearest = candidate; + smallestDistance = distance; + } + } + + return nearest; +} + +function formatShotWindowsForPrompt(shotWindows: SceneShotWindow[]): string { + return shotWindows + .map((window, index) => { + const transcriptText = window.transcriptText || "(no overlapping transcript cues)"; + + return dedent` + Window ${index + 1} + Start: ${formatSecondsForPrompt(window.startTime)} + End: ${formatSecondsForPrompt(window.endTime)} + Shots in window: ${window.shotCount} + Overlapping cues: ${window.cueCount} + Transcript: ${transcriptText}`; + }) + .join("\n\n"); +} + +/** + * Builds shot-aligned transcript windows from completed shot anchors and parsed VTT cues. + * Exported for unit tests. + */ +export function buildShotWindowsForScenes( + shots: Array<{ startTime: number }>, + cues: Array<{ startTime: number; endTime: number; text: string }>, + assetDurationSeconds: number, +): SceneShotWindow[] { + const orderedAnchors = Array.from(new Set( + [0, ...shots.map(shot => shot.startTime)] + .filter(startTime => Number.isFinite(startTime)) + .map(startTime => clampTime(roundToMillisecondPrecision(startTime), 0, assetDurationSeconds)), + )).sort((a, b) => a - b); + + if (orderedAnchors.length === 0 || orderedAnchors[0] !== 0) { + orderedAnchors.unshift(0); + } + + const windows: SceneShotWindow[] = []; + + for (let index = 0; index < orderedAnchors.length; index++) { + const startTime = orderedAnchors[index]; + const endTime = orderedAnchors[index + 1] ?? assetDurationSeconds; + + if (endTime <= startTime) { + continue; + } + + const overlappingCues = cues.filter(cue => cueOverlapsWindow(cue, startTime, endTime)); + const transcriptText = normalizeWhitespace( + overlappingCues + .map(cue => cue.text) + .filter(Boolean) + .join(" "), + ); + + windows.push({ + startTime, + endTime, + transcriptText, + cueCount: overlappingCues.length, + shotCount: 1, + }); + } + + return windows; +} + +/** + * Merges obviously low-signal shot windows to reduce prompt noise. + * Exported for unit tests. + */ +export function mergeSceneShotWindows( + shotWindows: SceneShotWindow[], + minShotWindowDurationSeconds: number = DEFAULT_MIN_SHOT_WINDOW_DURATION_SECONDS, +): SceneShotWindow[] { + const mergedWindows: SceneShotWindow[] = []; + + for (const shotWindow of shotWindows) { + const previousWindow = mergedWindows.at(-1); + if (!previousWindow) { + mergedWindows.push({ ...shotWindow }); + continue; + } + + const previousDuration = previousWindow.endTime - previousWindow.startTime; + const currentDuration = shotWindow.endTime - shotWindow.startTime; + const previousIsLowSignal = previousDuration < minShotWindowDurationSeconds || !previousWindow.transcriptText; + const currentIsLowSignal = currentDuration < minShotWindowDurationSeconds && !shotWindow.transcriptText; + + if (previousIsLowSignal || currentIsLowSignal) { + previousWindow.endTime = shotWindow.endTime; + previousWindow.cueCount += shotWindow.cueCount; + previousWindow.shotCount += shotWindow.shotCount; + previousWindow.transcriptText = normalizeWhitespace( + `${previousWindow.transcriptText} ${shotWindow.transcriptText}`, + ); + continue; + } + + mergedWindows.push({ ...shotWindow }); + } + + return mergedWindows; +} + +/** + * Normalizes model output into stable, ordered, non-overlapping scenes. + * Exported for unit tests. + */ +export function normalizeScenesForAsset({ + scenes, + sceneStartCandidates, + assetDurationSeconds, +}: NormalizeScenesOptions): Scene[] { + const validCandidates = Array.from(new Set( + [0, ...sceneStartCandidates] + .filter(candidate => Number.isFinite(candidate)) + .map(candidate => clampTime(roundToMillisecondPrecision(candidate), 0, assetDurationSeconds)), + )).sort((a, b) => a - b); + + const scenesByStartTime = new Map(); + + for (const scene of scenes) { + if (!Number.isFinite(scene.startTime) || typeof scene.title !== "string") { + continue; + } + + const snappedStartTime = snapToNearestCandidate( + clampTime(scene.startTime, 0, assetDurationSeconds), + validCandidates, + ); + + if (snappedStartTime >= assetDurationSeconds) { + continue; + } + + const title = scene.title.trim(); + if (!title) { + continue; + } + + if (!scenesByStartTime.has(snappedStartTime)) { + scenesByStartTime.set(snappedStartTime, title); + } + } + + const normalizedStartEntries = Array.from(scenesByStartTime.entries()) + .map(([startTime, title]) => ({ startTime, title })) + .sort((a, b) => a.startTime - b.startTime); + + if (normalizedStartEntries.length === 0) { + throw new Error("No valid scenes found in AI response"); + } + + if (normalizedStartEntries[0].startTime !== 0) { + normalizedStartEntries[0].startTime = 0; + } + + const normalizedScenes: Scene[] = normalizedStartEntries + .map((entry, index, allEntries) => { + const nextStartTime = allEntries[index + 1]?.startTime ?? assetDurationSeconds; + const endTime = clampTime(nextStartTime, entry.startTime, assetDurationSeconds); + + return { + startTime: entry.startTime, + endTime, + title: entry.title, + }; + }) + .filter(scene => scene.endTime > scene.startTime); + + if (normalizedScenes.length === 0) { + throw new Error("No valid non-overlapping scenes found after normalization"); + } + + return normalizedScenes; +} + +// ───────────────────────────────────────────────────────────────────────────── +// Prompts +// ───────────────────────────────────────────────────────────────────────────── + +export type SceneSystemPromptSections = "role" | "context" | "constraints" | "qualityGuidelines"; + +const scenesSystemPromptBuilder = createPromptBuilder({ + template: { + role: { + tag: "role", + content: "You are a video editor and transcript analyst specializing in segmenting media into coherent scenes.", + }, + context: { + tag: "context", + content: dedent` + You receive ordered shot-aligned transcript windows. + Each window starts at a shot boundary and includes the transcript cues that overlap that span. + Shots are candidate visual anchors for scene starts, but not every shot boundary should become a new scene. + A scene may contain multiple adjacent shot windows when they belong to the same narrative moment.`, + }, + constraints: { + tag: "constraints", + content: dedent` + - Only use information present in the provided shot windows and transcript text + - Only use provided shot-window start times as scene start candidates + - Return structured data that matches the requested JSON schema + - Do not add commentary or extra text outside the JSON + - When a section is provided, all scene titles MUST be written in that language`, + }, + qualityGuidelines: { + tag: "quality_guidelines", + content: dedent` + - Merge adjacent shot windows when they represent one continuous moment + - Prefer stable scene boundaries over overly granular cuts + - Use transcript continuity to avoid splitting one coherent idea into many scenes + - Use visual anchor changes to detect likely scene transitions`, + }, + }, + sectionOrder: ["role", "context", "constraints", "qualityGuidelines"], +}); + +const scenesPromptBuilder = createPromptBuilder({ + template: { + task: { + tag: "task", + content: "Group the shot windows into coherent scenes and provide a concise title for each scene.", + }, + outputFormat: { + tag: "output_format", + content: dedent` + Return valid JSON in this exact shape: + { + "scenes": [ + {"startTime": 0, "endTime": 18.4, "title": "Opening Introduction"}, + {"startTime": 18.4, "endTime": 47.2, "title": "Product Overview"}, + {"startTime": 47.2, "endTime": 91.0, "title": "Hands-On Demonstration"} + ] + }`, + }, + sceneGuidelines: { + tag: "scene_guidelines", + content: "", + }, + boundaryGuidelines: { + tag: "boundary_guidelines", + content: dedent` + - Start a new scene when there is a meaningful narrative transition, not just a camera cut + - Keep adjacent windows in the same scene when the transcript shows one continuous moment + - Use the shot-window start times as visual anchor candidates for scene boundaries + - Ensure the first scene starts at 0 seconds`, + }, + titleGuidelines: { + tag: "title_guidelines", + content: dedent` + - Keep titles concise and descriptive + - Avoid filler or generic labels like "Scene 1" + - Use the transcript's terminology when it helps identify the moment`, + }, + }, + sectionOrder: ["task", "outputFormat", "sceneGuidelines", "boundaryGuidelines", "titleGuidelines"], +}); + +function buildScenesUserPrompt({ + shotWindows, + promptOverrides, + minScenesPerHour = DEFAULT_MIN_SCENES_PER_HOUR, + maxScenesPerHour = DEFAULT_MAX_SCENES_PER_HOUR, + assetDurationSeconds, + languageName, +}: { + shotWindows: SceneShotWindow[]; + promptOverrides?: ScenesPromptOverrides; + minScenesPerHour?: number; + maxScenesPerHour?: number; + assetDurationSeconds: number; + languageName?: string; +}): string { + const contextSections: PromptSection[] = [ + { + tag: "asset_context", + content: dedent` + Duration: ${formatSecondsForPrompt(assetDurationSeconds)} + Shot windows provided: ${shotWindows.length}`, + }, + { + tag: "shot_windows", + content: formatShotWindowsForPrompt(shotWindows), + attributes: { format: "time-aligned transcript windows" }, + }, + ]; + + if (languageName) { + contextSections.push(createLanguageSection(languageName)); + } + + const dynamicSceneGuidelines = dedent` + - Create at least ${minScenesPerHour} and at most ${maxScenesPerHour} scenes per hour of content + - Use start and end times in seconds (not HH:MM:SS) + - Scene start times should be non-decreasing + - Scene end times should be greater than their start times + - Do not include text before or after the JSON`; + + const mergedOverrides: ScenesPromptOverrides = { + sceneGuidelines: dynamicSceneGuidelines, + ...promptOverrides, + }; + + return scenesPromptBuilder.buildWithContext(mergedOverrides, contextSections); +} + +// ───────────────────────────────────────────────────────────────────────────── +// Implementation +// ───────────────────────────────────────────────────────────────────────────── + +async function generateScenesWithAI({ + provider, + modelId, + userPrompt, + systemPrompt, + credentials, +}: { + provider: SupportedProvider; + modelId: string; + userPrompt: string; + systemPrompt: string; + credentials?: WorkflowCredentialsInput; +}): Promise { + "use step"; + + const model = await createLanguageModelFromConfig(provider, modelId, credentials); + + const response = await withRetry(() => + generateText({ + model, + output: Output.object({ schema: scenesSchema }), + messages: [ + { + role: "system", + content: systemPrompt, + }, + { + role: "user", + content: userPrompt, + }, + ], + }), + ); + + return { + scenes: response.output, + usage: { + inputTokens: response.usage.inputTokens, + outputTokens: response.usage.outputTokens, + totalTokens: response.usage.totalTokens, + reasoningTokens: response.usage.reasoningTokens, + cachedInputTokens: response.usage.cachedInputTokens, + }, + }; +} + +export async function generateScenes( + assetId: string, + languageCode: string, + options: ScenesOptions = {}, +): Promise { + "use workflow"; + const { + provider = DEFAULT_PROVIDER, + model, + promptOverrides, + minScenesPerHour, + maxScenesPerHour, + outputLanguageCode, + pollIntervalMs, + maxAttempts, + createShotsIfMissing = true, + minShotWindowDurationSeconds = DEFAULT_MIN_SHOT_WINDOW_DURATION_SECONDS, + credentials, + } = options; + + const modelConfig = resolveLanguageModelConfig({ + ...options, + model, + provider: provider as SupportedProvider, + }); + + const { asset: assetData, playbackId, policy } = await getPlaybackIdForAsset(assetId, credentials); + const assetDurationSeconds = getAssetDurationSecondsFromAsset(assetData); + if (assetDurationSeconds === undefined) { + throw new Error(`Could not determine duration for asset '${assetId}'`); + } + if (isAudioOnlyAsset(assetData)) { + throw new Error("Scene generation is only supported for video assets"); + } + + const signingContext = await resolveMuxSigningContext(credentials); + if (policy === "signed" && !signingContext) { + throw new Error( + "Signed playback ID requires signing credentials. " + + "Set MUX_SIGNING_KEY and MUX_PRIVATE_KEY environment variables.", + ); + } + + const readyTextTracks = getReadyTextTracks(assetData); + const transcriptResult = await fetchTranscriptForAsset(assetData, playbackId, { + languageCode, + cleanTranscript: false, + shouldSign: policy === "signed", + credentials, + required: true, + }); + + if (!transcriptResult.track || !transcriptResult.transcriptText) { + const availableLanguages = readyTextTracks + .map(track => track.language_code) + .filter(Boolean) + .join(", "); + throw new Error( + `No caption track found for language '${languageCode}'. Available languages: ${availableLanguages || "none"}`, + ); + } + + const cues = parseVTTCues(transcriptResult.transcriptText); + if (cues.length === 0) { + throw new Error("No usable VTT cues found in caption track"); + } + + const shotsResult = await waitForShotsForAsset(assetId, { + pollIntervalMs, + maxAttempts, + createIfMissing: createShotsIfMissing, + credentials, + }); + + const rawShotWindows = buildShotWindowsForScenes(shotsResult.shots, cues, assetDurationSeconds); + const shotWindows = mergeSceneShotWindows(rawShotWindows, minShotWindowDurationSeconds); + if (shotWindows.length === 0) { + throw new Error("No shot windows available for scene generation"); + } + + const resolvedLanguageCode = outputLanguageCode && outputLanguageCode !== "auto" ? + outputLanguageCode : + (transcriptResult.track.language_code ?? languageCode); + const languageName = resolvedLanguageCode ? getLanguageName(resolvedLanguageCode) : undefined; + + const userPrompt = buildScenesUserPrompt({ + shotWindows, + promptOverrides, + minScenesPerHour, + maxScenesPerHour, + assetDurationSeconds, + languageName, + }); + + let scenesData: ScenesAnalysisResponse | null = null; + + try { + scenesData = await generateScenesWithAI({ + provider: modelConfig.provider, + modelId: modelConfig.modelId, + userPrompt, + systemPrompt: scenesSystemPromptBuilder.build(), + credentials, + }); + } catch (error) { + throw new Error( + `Failed to generate scenes with ${provider}: ${error instanceof Error ? error.message : "Unknown error"}`, + ); + } + + if (!scenesData?.scenes) { + throw new Error("No scenes generated from AI response"); + } + + const normalizedScenes = normalizeScenesForAsset({ + scenes: scenesData.scenes.scenes, + sceneStartCandidates: shotWindows.map(shotWindow => shotWindow.startTime), + assetDurationSeconds, + }); + + const usageWithMetadata: TokenUsage = { + ...scenesData.usage, + metadata: { + ...scenesData.usage?.metadata, + assetDurationSeconds, + }, + }; + + return { + assetId, + languageCode, + scenes: normalizedScenes, + usage: usageWithMetadata, + }; +} diff --git a/tests/integration/scenes.test.ts b/tests/integration/scenes.test.ts new file mode 100644 index 00000000..8cf15f0b --- /dev/null +++ b/tests/integration/scenes.test.ts @@ -0,0 +1,39 @@ +import { describe, expect, it } from "vitest"; + +import { generateScenes } from "../../src/workflows"; +import { muxTestAssets } from "../helpers/mux-test-assets"; + +describe("scenes Integration Tests", () => { + const assetId = muxTestAssets.chaptersAssetId; + const languageCode = "en"; + + it("should return valid scene boundaries for the test asset", async () => { + const result = await generateScenes(assetId, languageCode, { + provider: "openai", + }); + + expect(result).toBeDefined(); + expect(result).toHaveProperty("assetId", assetId); + expect(result).toHaveProperty("languageCode", languageCode); + expect(Array.isArray(result.scenes)).toBe(true); + expect(result.scenes.length).toBeGreaterThan(0); + + result.scenes.forEach((scene, index) => { + expect(typeof scene.startTime).toBe("number"); + expect(typeof scene.endTime).toBe("number"); + expect(scene.endTime).toBeGreaterThan(scene.startTime); + expect(typeof scene.title).toBe("string"); + expect(scene.title.length).toBeGreaterThan(0); + + if (index === 0) { + expect(scene.startTime).toBe(0); + } + + if (index > 0) { + const previousScene = result.scenes[index - 1]; + expect(scene.startTime).toBeGreaterThanOrEqual(previousScene.startTime); + expect(scene.startTime).toBe(previousScene.endTime); + } + }); + }); +}); diff --git a/tests/unit/scenes.test.ts b/tests/unit/scenes.test.ts new file mode 100644 index 00000000..738ef3df --- /dev/null +++ b/tests/unit/scenes.test.ts @@ -0,0 +1,277 @@ +import { generateText } from "ai"; +import dedent from "dedent"; +import { beforeEach, describe, expect, it, vi } from "vitest"; + +import { getAssetDurationSecondsFromAsset, getPlaybackIdForAsset, isAudioOnlyAsset } from "../../src/lib/mux-assets"; +import { createLanguageModelFromConfig, resolveLanguageModelConfig } from "../../src/lib/providers"; +import { resolveMuxSigningContext } from "../../src/lib/workflow-credentials"; +import { waitForShotsForAsset } from "../../src/primitives/shots"; +import { fetchTranscriptForAsset, getReadyTextTracks } from "../../src/primitives/transcripts"; +import { + buildShotWindowsForScenes, + generateScenes, + mergeSceneShotWindows, + normalizeScenesForAsset, +} from "../../src/workflows/scenes"; + +vi.mock("ai", () => ({ + generateText: vi.fn(), + Output: { + object: vi.fn(({ schema }) => ({ schema })), + }, +})); + +vi.mock("../../src/lib/mux-assets", () => ({ + getAssetDurationSecondsFromAsset: vi.fn(), + getPlaybackIdForAsset: vi.fn(), + isAudioOnlyAsset: vi.fn(), +})); + +vi.mock("../../src/lib/providers", () => ({ + createLanguageModelFromConfig: vi.fn(), + resolveLanguageModelConfig: vi.fn(), +})); + +vi.mock("../../src/lib/workflow-credentials", () => ({ + resolveMuxSigningContext: vi.fn(), +})); + +vi.mock("../../src/primitives/shots", () => ({ + waitForShotsForAsset: vi.fn(), +})); + +vi.mock("../../src/primitives/transcripts", async () => { + const actual = await vi.importActual( + "../../src/primitives/transcripts", + ); + + return { + ...actual, + fetchTranscriptForAsset: vi.fn(), + getReadyTextTracks: vi.fn(), + }; +}); + +describe("scenes workflow helpers", () => { + it("builds shot windows from shot anchors and overlapping cues", () => { + const shotWindows = buildShotWindowsForScenes( + [{ startTime: 5 }, { startTime: 10 }], + [ + { startTime: 0, endTime: 2, text: "Intro" }, + { startTime: 4, endTime: 6, text: "Welcome back" }, + { startTime: 6, endTime: 9, text: "Product overview" }, + { startTime: 10, endTime: 14, text: "Hands on demo" }, + ], + 15, + ); + + expect(shotWindows).toEqual([ + { + startTime: 0, + endTime: 5, + transcriptText: "Intro Welcome back", + cueCount: 2, + shotCount: 1, + }, + { + startTime: 5, + endTime: 10, + transcriptText: "Welcome back Product overview", + cueCount: 2, + shotCount: 1, + }, + { + startTime: 10, + endTime: 15, + transcriptText: "Hands on demo", + cueCount: 1, + shotCount: 1, + }, + ]); + }); + + it("merges low-signal adjacent shot windows before prompting", () => { + const merged = mergeSceneShotWindows([ + { + startTime: 0, + endTime: 1, + transcriptText: "", + cueCount: 0, + shotCount: 1, + }, + { + startTime: 1, + endTime: 6, + transcriptText: "Welcome back", + cueCount: 1, + shotCount: 1, + }, + { + startTime: 6, + endTime: 10, + transcriptText: "Here is the product", + cueCount: 1, + shotCount: 1, + }, + ], 2); + + expect(merged).toEqual([ + { + startTime: 0, + endTime: 6, + transcriptText: "Welcome back", + cueCount: 1, + shotCount: 2, + }, + { + startTime: 6, + endTime: 10, + transcriptText: "Here is the product", + cueCount: 1, + shotCount: 1, + }, + ]); + }); + + it("normalizes model output by snapping to candidates and computing stable end times", () => { + const normalized = normalizeScenesForAsset({ + scenes: [ + { startTime: 0.2, endTime: 3, title: "Opening" }, + { startTime: 4.9, endTime: 9, title: "Product Intro" }, + { startTime: 5.1, endTime: 9.5, title: "Duplicate Boundary" }, + { startTime: 9.8, endTime: 15, title: "Hands On Demo" }, + ], + sceneStartCandidates: [0, 5, 10], + assetDurationSeconds: 15, + }); + + expect(normalized).toEqual([ + { startTime: 0, endTime: 5, title: "Opening" }, + { startTime: 5, endTime: 10, title: "Product Intro" }, + { startTime: 10, endTime: 15, title: "Hands On Demo" }, + ]); + }); +}); + +describe("generateScenes", () => { + const mockAsset = { + tracks: [ + { + id: "track-123", + type: "text", + status: "ready", + text_type: "subtitles", + language_code: "en", + }, + ], + } as any; + + const rawVtt = dedent` + WEBVTT + + 00:00:00.000 --> 00:00:04.000 + Welcome back everyone. + + 00:00:04.000 --> 00:00:09.000 + Today we are unboxing the product. + + 00:00:09.000 --> 00:00:13.000 + Now let me show you how it works. + `; + + beforeEach(() => { + vi.clearAllMocks(); + + vi.mocked(resolveLanguageModelConfig).mockReturnValue({ + provider: "openai", + modelId: "gpt-5.1", + } as any); + vi.mocked(createLanguageModelFromConfig).mockResolvedValue({ id: "mock-model" } as any); + vi.mocked(getPlaybackIdForAsset).mockResolvedValue({ + asset: mockAsset, + playbackId: "playback-123", + policy: "public", + } as any); + vi.mocked(getAssetDurationSecondsFromAsset).mockReturnValue(13); + vi.mocked(isAudioOnlyAsset).mockReturnValue(false); + vi.mocked(resolveMuxSigningContext).mockResolvedValue(undefined); + vi.mocked(getReadyTextTracks).mockReturnValue(mockAsset.tracks); + vi.mocked(fetchTranscriptForAsset).mockResolvedValue({ + transcriptText: rawVtt, + track: mockAsset.tracks[0], + }); + vi.mocked(waitForShotsForAsset).mockResolvedValue({ + status: "completed", + createdAt: "1773108428", + shots: [ + { startTime: 4, imageUrl: "https://example.com/shot-1.webp" }, + { startTime: 9, imageUrl: "https://example.com/shot-2.webp" }, + ], + }); + vi.mocked(generateText).mockResolvedValue({ + output: { + scenes: [ + { startTime: 0.3, endTime: 4.1, title: "Warm Welcome" }, + { startTime: 4.2, endTime: 9.2, title: "Unboxing Begins" }, + { startTime: 9.1, endTime: 13, title: "Product Demo" }, + ], + }, + usage: { + inputTokens: 120, + outputTokens: 30, + totalTokens: 150, + }, + } as any); + }); + + it("builds a prompt from shot windows, honors prompt overrides, and normalizes output", async () => { + const result = await generateScenes("asset-123", "en", { + promptOverrides: { + boundaryGuidelines: "Prefer broader scenes unless the narrative clearly shifts.", + titleGuidelines: "Use short punchy titles under four words.", + }, + outputLanguageCode: "es", + }); + + expect(result).toEqual({ + assetId: "asset-123", + languageCode: "en", + scenes: [ + { startTime: 0, endTime: 4, title: "Warm Welcome" }, + { startTime: 4, endTime: 9, title: "Unboxing Begins" }, + { startTime: 9, endTime: 13, title: "Product Demo" }, + ], + usage: { + inputTokens: 120, + outputTokens: 30, + totalTokens: 150, + metadata: { + assetDurationSeconds: 13, + }, + }, + }); + + const generateTextCall = vi.mocked(generateText).mock.calls[0]?.[0]; + expect(generateTextCall).toBeDefined(); + const userMessageContent = generateTextCall!.messages?.[1]?.content; + expect(userMessageContent).toBeDefined(); + expect(userMessageContent).toContain( + "Prefer broader scenes unless the narrative clearly shifts.", + ); + expect(userMessageContent).toContain( + "Use short punchy titles under four words.", + ); + expect(userMessageContent).toContain( + "All output (title, description, keywords, chapter titles) MUST be written in Spanish.", + ); + }); + + it("rejects audio-only assets", async () => { + vi.mocked(isAudioOnlyAsset).mockReturnValue(true); + + await expect(generateScenes("asset-123", "en")).rejects.toThrow( + "Scene generation is only supported for video assets", + ); + expect(generateText).not.toHaveBeenCalled(); + }); +});