Runway + ElevenLabs + Nano Banana

"A nine-layer terrine, color-matched to Bruno's coat" — 41 seconds, 18 scenes, $7.20 in video credits.
▶ Watch it on YouTube · mp4

A pipeline for generating absurd transformation shorts with no manual editing — script by Claude, stills by Nano Banana, voice and music by ElevenLabs, video by Veo 3.1 via the Runway API, assembly by ffmpeg. The workflow's one rule: everything expensive happens only after a human approved the cheap draft.

Start with LEARNINGS.md — where each of these tools fails (drifting subjects, phantom hands, leaked phonemes) and what mitigates it.

That short was made in two passes, and the order is the whole trick:

1. First, make the video out of images. The preview stage builds a fully watchable draft — real narration, real music, Ken Burns camera moves — entirely from generated stills, for zero video credits:

python3 .claude/skills/absurd-transformation-shorts/scripts/validate_scenes.py \
  specs/bruno-terrine.json                     # spec invariants; exit 0 = clean
npm run preview -- specs/bruno-terrine.json    # stills + voice + music → preview.mp4

The same short as its $0 stills-only draft (▶ mp4).

Every scene exists as a single image you can inspect, reject, and re-roll for cents — this run rejected and regenerated 5 stills along the way. If the hook doesn't work as stills, it won't work animated.

2. Then, make it out of video. Once the stills are approved, each one becomes the literal first frame of an image-to-video clip — so what you approved is what moves — and ffmpeg assembles the final cut. This is where the demo's 18 clips × 4s × 10 credits/s = $7.20 went:

npm run spec-gen -- specs/bruno-terrine.json --animate   # approved stills → clips → final

Video generation is 8–80× more expensive per reject than stills, so all the spending is funneled behind that approval gate. There is also a one-shot generic pipeline (npm run shorts -- --topic "...") that goes straight from a topic line to text-to-video — here's a finished 31s short it made for ~$4.70 — but the stills-first pipeline is what the interesting parts of this repo are about. See How it works.

Setup

npm install
cp .env.example .env   # add your keys

Requires Node ≥ 22 and ffmpeg/ffprobe on PATH (with libass, the default in Homebrew builds). RUNWAYML_API_SECRET is the only required key; ANTHROPIC_API_KEY is needed for --topic (Claude writes the script), and GEMINI_API_KEY + ELEVENLABS_API_KEY only for the zero-Runway-credit preview stage.

Usage

The scene pipeline's three commands (validate → preview → spec-gen --animate) are shown above; what follows is the one-shot pipeline.

# one-shot: topic → final.mp4
npm run shorts -- --topic "why octopuses have nine brains"

# write + inspect the script first, generate nothing
npm run shorts -- --topic "why octopuses have nine brains" --dry-run
# ...edit out/<slug>/spec.json by hand, then:
npm run shorts -- --spec out/why-octopuses-have-nine-brains/spec.json

# options
npm run shorts -- --topic "..." --model gen4.5 --voice Eleanor --beats 5 \
  --style "grainy 35mm documentary, warm tones" --no-music

npm run credits   # remaining balance on the Runway account

Output: out/<slug>/final.mp4 (1080×1920, 30 fps, faststart) plus a title and hashtags printed for the upload.

The spec is the editable seed of the whole run. --dry-run writes it and stops, so you can rewrite narration or visuals before spending anything, then resume with --spec. It's just JSON:

{
  "title": "Octopuses Have Nine Brains",
  "slug": "octopus-test",
  "style": "cinematic underwater documentary, deep blue ocean light, volumetric rays, 35mm film grain",
  "voice": "Clint",
  "music": "mysterious ambient underwater drone, soft pulsing synth, seamless loop",
  "beats": [
    {
      "narration": "An octopus has nine brains. And that changes everything.",
      "visual": "Extreme close-up of an octopus eye opening suddenly in dark deep water, slow push-in, bioluminescent particles drifting, dramatic side lighting"
    },
    {
      "narration": "Each arm thinks for itself, even when severed.",
      "visual": "An octopus arm uncoiling and exploring a rocky reef crevice on its own, suckers gripping in rippling motion, camera tracking along the arm, god rays from above"
    }
  ],
  "hashtags": ["octopus", "oceanfacts", "nature", "mindblown"]
}

How it works

Two pipelines share the same Runway client (src/runway.ts), cost gate, and ffmpeg assembly. Everything lands in out/<slug>/ and every step is cached by file existence — delete a file to regenerate it, rerun to resume a failed run without re-paying for what succeeded.

  BEAT PIPELINE (npm run shorts)              SCENE PIPELINE (preview → spec-gen)
  general-purpose, text-to-video              consistency-first, image-to-video

  topic ──▶ Claude writes spec.json           specs/<slug>.json (skill-authored)
              │                                 │ validate_scenes.py (exit 0 = clean)
  textToSpeech▼ eleven_multilingual_v2          ▼
  narration_N.mp3 (sizes the clips)          Gemini stills + ElevenLabs voice
              │                              preview.mp4  ◀── approval gate,
              │ $ cost gate                     │             zero Runway credits
  textToVideo ▼ seedance2_mini 720:1280         ▼
  clip_N.mp4 (parallel)                      textToImage ref chain:
     +                                       plate ─▶ subject ─▶ mid-state
  soundEffect eleven_text_to_sound_v2           ▼
  music.mp3 (seamless loop)                  textToImage scene stills (refs+locks)
              │                                 │ $ confirm
              │                              imageToVideo ▼ veo3.1_fast,
              │                              approved still = first frame
              ▼                                 ▼
        ffmpeg: crop 1080×1920 · trim to narration · concat · mix · captions
              └────────────────────▶ final.mp4 ◀────────────────────┘

Beat pipeline, stage by stage

1. Script — Claude (Anthropic API) turns the topic into 4–6 beats, each a narration line plus a detailed visual prompt sharing one style string (src/script.ts).
2. Voiceover — client.textToSpeech.create({ model: 'eleven_multilingual_v2', voice: { type: 'runway-preset', presetId } }). Narration is generated first because it's cheap and each clip's duration is sized to its audio.
3. Cost gate — src/cost.ts prices the video stage from documented credit rates and asks y/N before proceeding (--yes skips it).
4. Clips — client.textToVideo.create({ model: 'seedance2_mini', ratio: '720:1280', duration }), one per beat, all in parallel. Music — client.soundEffect.create({ model: 'eleven_text_to_sound_v2', loop: true }) runs alongside.
5. Assembly — ffmpeg upscales/crops to 1080×1920, trims each clip to its narration, concats, ducks the music under the voice, and burns bold timed captions via libass (src/assemble.ts).

All generation is async task-based; the SDK's .waitForTaskOutput() handles polling. Output URLs expire in 24–48h, so every asset is downloaded into out/<slug>/ the moment its task completes.

Scene pipeline: consistency via reference chaining

Text-to-video drifts: ask for the same kitchen twice and you get two kitchens. The scene pipeline (src/spec-gen.ts) exists to hold one environment, one subject, and one lighting state across ~15 cuts:

1. Validate — validate_scenes.py enforces the spec's invariants (forward-only lighting, one transformation step per cut, shot-tier budget) before anything is generated.
2. Preview gate (src/preview.ts) — stills via the Gemini API directly (Nano Banana, no Runway credits) plus ElevenLabs narration, assembled with ffmpeg Ken Burns moves into a watchable preview.mp4. If the hook doesn't work as stills, it won't work animated. Stills land at the exact paths the next stage uses, so nothing approved here is regenerated.
3. Reference chain — client.textToImage.create({ referenceImages: [...] }) builds the master environment plate, then the "before" subject inside that plate, then a mid-transformation state — each generated with the previous ones as tagged references. Runway's output URLs are recorded in manifest.json and reused as referenceImages URIs while they're live (24–48h), falling back to data URIs from the local files.
4. Scene stills — every scene's prompt gets the same appended suffix stack: a byte-identical subject description ("subject lock"), a lighting lock, and a style suffix. Wide shots are never generated from text — they derive from the plate.
5. Animate — client.imageToVideo.create({ model: 'veo3.1_fast', promptImage: still, duration: 4, audio: false }). The approved still is the first frame, so what you approved is what moves. Scenes need ~2.4s; clips are generated at the 4s minimum and cut short in assembly.

Costs

Real numbers, because nobody publishes them. Runway credits are $0.01 each; rates from docs.dev.runwayml.com as of mid-2026 (src/cost.ts).

Call	Model	Credits
textToVideo	seedance2_mini	16/s, 64 minimum per clip
textToVideo	gemini_omni_flash	10/s
imageToVideo	veo3.1_fast	10/s silent, 15/s with audio
textToImage	gemini_2.5_flash (Nano Banana)	5 per image
textToImage	gen4_image	8 per image at 1080p
textToImage	gemini_image3_pro	20 per image (up to 14 refs)
soundEffect	eleven_text_to_sound_v2	~0.25/s, 5 minimum
textToSpeech	eleven_multilingual_v2	undocumented; small in practice

What the demo actually cost. The finished Bruno short at the top of this page spent $7.20 in Runway credits, all of it in the animate step: 18 clips × 4s × 10 credits/s on veo3.1_fast = 720 credits. Everything before that was free of Runway spend — scene stills (including 5 rejects and re-rolls) and five reference images went through the Gemini API directly, narration and the music loop through ElevenLabs (~700 character credits for the voice, ~100 per loop). That's the stills-first workflow paying off: the rejects all happened at the $0 stage.

What a finished text-to-video short cost. The pigeon short: six beats, clips of 6+4+6+4+4+5 seconds on seedance2_mini → 464 credits, plus 8 for the 30s music loop: **$4.70 for 31 seconds**, of which 98% is video. A single beat runs 64–96 credits ($0.64–0.96), which is why the cost gate prints an estimate and asks before the video stage — and why narration (cents) is generated before clips (dollars).

A 15-scene transformation short, budgeting the realistic 1.5–2 generations per usable scene: 25 stills at 5 credits ($1.25, or $0 via the Gemini API in the preview stage), 15 × 4s silent clips on veo3.1_fast at 40 credits each ($6.00), plus ~$0.40 per rejected clip re-roll → roughly $7–9 end to end, ~85% of it in image-to-video. ElevenLabs narration and the ambience loop bill separately (~1 character = 1 credit; the loop ~100 credits per call).

The economics drive the architecture: rejecting a bad still costs $0.05, rejecting a bad clip costs $0.40+, so everything funnels through stills-first approval, and video/voice generation always confirms interactively before spending (src/confirm.ts).

Notes on building with the Runway API

Honest field notes from building this (the cross-tool failure modes — image, video, voice, and music — live in LEARNINGS.md):

• The task model is the right shape. Everything is async create-then-poll, and the SDK's .waitForTaskOutput() makes a whole pipeline stage a one-liner. Fan-out (six clips in parallel) needed nothing beyond Promise.all.
• Download immediately — output URLs expire in 24–48h. Easy to learn the hard way. The pipeline saves every asset on task completion, and keeps the ephemeral URLs in manifest.json because passing a live URL as a referenceImages input is a much smaller request than re-uploading the image as a base64 data URI.
• Cross-clip consistency doesn't come from prompts — it comes from images. Repeating a description re-rolls the dice. What worked: generate reference stills once, chain them (plate → subject-in-plate → mid-state) as tagged referenceImages, keep the subject description byte-identical everywhere it appears, and append one shared lighting + style suffix to every prompt. Then use image-to-video so the approved still is literally frame one.
• Stills-first is the biggest cost lever. Image models are 8–80× cheaper per reject than video models. Approving stills and only then animating cut waste more than any prompt engineering did.
• Per-model quirks need a table somewhere. Duration windows differ (gen4.5 2–10s, gemini_omni_flash 3–10s, seedance 4–15s — clamped in src/runway.ts); Gemini image models have no 1080:1920 ratio (768:1344 is their 9:16 bucket); Veo at 1080:1920 locks clips to 8s while 720:1280 allows 4s. None of this is hard, but each one was discovered by a failed request.
• Content moderation fails per-clip, so make retries cheap. A flagged prompt rejects one beat, not the run. File-existence caching means the fix is: reword that beat's visual in spec.json, rerun with --spec, and only the failed clip regenerates.
• Some prices aren't documented (TTS, several video models). The cost estimator flags those lines as rough guesses rather than pretending. The organization.retrieve() endpoint exposes the credit balance, which made a preflight balance check (npm run credits) trivial — that endpoint deserves more visibility.
• "Generate long, cut short" beats fighting minimum durations. Retention-format cadence is ~2.4s per scene, below every model's floor. Generating 4s and trimming in ffmpeg is cheaper than any workaround and gives assembly slack to cut on the best frame.

The format: absurd transformation shorts

The whole pipeline serves one structure, derived from analyzing high-performing shorts (the research is in HANDOFF.md). Three mechanisms have to operate simultaneously — remove any one and the format collapses:

1. Universal relatability — the viral aspect. The topic must be known to literally everybody: dirty→clean, broken→whole, a dog getting a treat. No specialist knowledge, no explanation. If a stranger can't name the activity from a single frame, the topic is wrong.
2. Completion compulsion — why people watch to the end. A hyper-specific goal is declared up front ("a 9-layer terrine, matched to his coat") and visibly proven at the end against a token from the opening. The viewer stays because an open loop demands closing.
3. The hook — the first four seconds. The declaration lands inside 4s with motion in frame one (which doubles as the feed thumbnail). The absurdity — effort wildly disproportionate to the outcome — is what makes the promise worth staying for, and it's the share trigger.

On top of the mechanics sits a story: the demo video isn't a random cooking clip, it's episode one of a fictional business — a studio where people bring their dogs to have food made to match them, "Customer number 1: Bruno." The story is compressed into three words instead of explained, and it turns one video into a series engine (a different dog every episode) while giving the absurdity a deadpan premise: this isn't a hobby, it's someone's job.

The validator enforces the retention rules that follow from all this (declaration timing, ~2.4s cadence, forward-only lighting, hard cut on the reveal, structural variation between uploads), and the production playbook lives in the absurd-transformation-shorts skill — which improves with every video: nearly every rule in it (subject locks, the radio test, unmarked instruments, the discrete-subject criterion) was added after a concrete failure in production. The failure catalog is LEARNINGS.md; the skill is where those learnings become enforced defaults for the next run.

Where this can fail

Honest risk assessment, from the research in HANDOFF.md:

• Most of what captures people features people. Surveys in the research put consumer enthusiasm for AI-created work at 26% in 2025, down from 60% in 2023, and 68% say featuring real people is what supports authenticity. AI content is a large and growing minority of the feed — a mid-2026 Kapwing study found 278 exclusively-AI channels with 63B combined views, and ~21% of the first 500 Shorts served to a fresh account were pure AI content — but the top-performing shorts remain overwhelmingly human-fronted.
• The bet this repo makes is near-believability. The format's reference channel (250M+ views in 3 months) worked because nobody in the comments realized it was AI. These videos aim to sit just inside plausible — real-looking kitchen, real-sounding narrator, physics that holds — because attention collapses the moment a viewer clocks the content as synthetic and reclassifies it as slop. That makes this a window, not a permanent property: as detection literacy rises, the believability bar rises with it.
• Platform policy is the hard constraint. Synthetic content must be disclosed on upload, and platforms review AI channels channel-wide for template reuse — identical structure across uploads is the specific pattern that gets channels terminated. The skill enforces structural variation between videos for exactly this reason.
• Nothing here is empirically validated yet. Every retention rule is derived from other people's videos. The first uploads and their retention curves are the actual test.

What the whole journey actually cost

The cost tables above price a clean run. Producing the finished Bruno video was not a clean run — it was ~30 rounds of review and regeneration (documented in LEARNINGS.md), and the honest ledger looks like this:

	Clean-run cost	What we actually spent
Stills (Nano Banana)	~$1 (25 images)	~$3.50 (~90 generations: two full studio redesigns, card/physics/continuity re-rolls)
Video (Veo via Runway)	$7.20 (18 clips)	~$11 (28 clips: hook test, drift re-rolls, one reversed-physics unmold)
Voice + music (ElevenLabs)	~1k credits	~6k credits — incl. one full re-voice after switching narrators, a duplicated re-voice from a pipeline bug, and three discarded music directions (lo-fi → classical → marimba); covered by the $22/mo Creator plan
Scrapped exploration	—	~$2 — a full cable-untangling video specced and 30+ stills generated before the concept failed on a fundamental (amorphous subjects can't stay consistent); its lessons became permanent skill rules

Bottom line: roughly $15 of pure compute for the finished 42s video, about double that all-in once iteration, exploration, and the music subscription are counted. The gap between the two columns is the learning documented in this repo — and it's exactly the gap the skill closes over time, since every failure became an enforced rule that the next video inherits for free.

License

MIT