novel-to-movie

Upload a novel, get a two-minute movie made of generated video clips, with a single narrator voice over a music bed. Free to the visitor, no login. The operator funds and supplies all API keys, which live only on the server.

The defining design choice is aggressive condensation: a whole novel is compressed to ~15 story beats and told through a narrator speaking over the visuals. This deliberately avoids per-character voice casting and lip-sync — the hardest, most expensive parts of filmmaking — while still producing genuine video rather than a slideshow.

---

How it works

A single straight-through pipeline runs automatically once a visitor uploads a file and presses start:

Step	Stage	What happens
3	Text extraction	Read .txt / .docx / .pdf into clean plain text
4	Condensation	LLM → title, logline, 12–16 story beats
5	Shot list	LLM → validated shot-list JSON (the core contract)
6	Anchors	One style frame + one portrait per character
7	Clips	One image-to-video clip per shot (the real-video core)
8	Narration	One narrator line per shot (single voice)
9	Music	One background score for the whole film
10	Assembly	FFmpeg concat + crossfades + ducked music + loudnorm → MP4

Steps 11–12 (progress reporting and delivery/cleanup) wrap the run.

Every stage degrades gracefully so the job always ends with a finished movie or a clear terminal error, and never waits for a human (see Graceful degradation).

---

Quick start

1. Install

macOS / Linux

# System dependency: FFmpeg must be on PATH.
#   macOS:   brew install ffmpeg
#   Ubuntu:  sudo apt-get install -y ffmpeg

python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
cp .env.example .env

Windows (PowerShell)

# System dependency: FFmpeg must be on PATH. Install it with one of:
#   winget install --id=Gyan.FFmpeg -e
#   choco install ffmpeg
# (or download from https://www.gyan.dev/ffmpeg/builds/ and add the bin\
#  folder to your PATH). Verify with:  ffmpeg -version

py -m venv .venv
.\.venv\Scripts\Activate.ps1
pip install -r requirements.txt
copy .env.example .env

If PowerShell blocks the activation script with a "running scripts is disabled" error, run once: Set-ExecutionPolicy -Scope CurrentUser -ExecutionPolicy RemoteSigned

Windows (Command Prompt / cmd.exe)

:: Install FFmpeg as above (winget install --id=Gyan.FFmpeg -e), then:
py -m venv .venv
.venv\Scripts\activate.bat
pip install -r requirements.txt
copy .env.example .env

On Windows, set RUN_MODE per command instead of the inline RUN_MODE=mock prefix used in the examples below:

# PowerShell
$env:RUN_MODE = "mock"; python -m pytest tests/ -s
$env:RUN_MODE = "mock"; python main.py

:: Command Prompt
set RUN_MODE=mock && python -m pytest tests/ -s
set RUN_MODE=mock && python main.py

2. Run the free mock test first (no keys, no cost)

Mock mode replaces every provider with a local generator that produces real but trivial media, so the whole pipeline + assembly + UI run end to end in seconds:

RUN_MODE=mock python -m pytest tests/ -s

This asserts a valid, non-empty MP4 with a real duration is produced.

3. Start the service — the single command

RUN_MODE=mock python main.py

Open http://localhost:8000, upload tests/sample_novel.txt, press Make my movie, and watch live progress through to an inline player and a download button. (Drop the RUN_MODE=mock prefix once your real keys are set in .env — RUN_MODE defaults to real there.)

---

Mock vs. real mode

• RUN_MODE=mock — all providers are mocked. No keys required, no API calls, no cost. Use it to exercise orchestration, progress, assembly, and the UI. The included test runs in this mode.
• RUN_MODE=real — uses the configured providers. Required keys are checked at startup and the server fails fast with a readable error if any are missing.

Choosing the language model

The LLM passes (condensation + shot list) sit behind a swappable adapter. Set LLM_PROVIDER and the matching key + model id:

LLM_PROVIDER	Key	Example LLM_MODEL
anthropic (default)	ANTHROPIC_API_KEY	claude-sonnet-4-6
gemini	GEMINI_API_KEY	gemini-2.0-flash
openai	OPENAI_API_KEY	gpt-4o

Only the key for the selected provider is required. The OpenAI adapter also accepts OPENAI_BASE_URL for OpenAI-compatible gateways (Azure, OpenRouter, a local server). Adding another vendor is one new adapter implementing LLMProvider plus one line in pipeline/providers/__init__.py — no pipeline step changes.

Only switch to real mode after the mock test passes.

---

Environment variables

All configuration lives in .env (copy from .env.example). Keys are read only through config.py and are never sent to the browser.

Variable	Meaning
ANTHROPIC_API_KEY	Claude key for the LLM passes
AGGREGATOR_API_KEY	Key for the image/video/music aggregator (fal.ai or Replicate)
ELEVENLABS_API_KEY	ElevenLabs key for narration
LLM_PROVIDER / IMAGE_PROVIDER / VIDEO_PROVIDER / TTS_PROVIDER / MUSIC_PROVIDER	Adapter selection
AGGREGATOR	fal (default) or replicate
LLM_MODEL / IMAGE_MODEL / VIDEO_MODEL / MUSIC_MODEL / TTS_VOICE_ID	Model identifiers (configurable, not hardcoded)
TARGET_DURATION_SECONDS / SHOT_COUNT / SHOT_LENGTH_SECONDS	Movie shape
MAX_RETRIES_PER_SHOT	Retries per provider call before fallback
MAX_MOVIES_PER_SESSION	Per-session quota (default 1)
FAILED_SHOT_FALLBACK	hold_still (default) or drop_shot
SESSION_TTL_MINUTES	How long session files live before the sweep deletes them
MAX_UPLOAD_MB	Upload size cap
RUN_MODE	real or mock
PORT / HOST	Where the server listens

These are the operator-wide defaults. A visitor can override several of them per movie via the Advanced settings panel (below).

---

Per-job controls (Advanced settings)

Before pressing Make my movie, the visitor can open Advanced settings to tune one movie without touching .env:

• Movie length (30 s – 5 min) — a slider that sets how long the finished film runs. Leave Number of shots blank and the server auto-fits the shot count to this length at the chosen seconds-per-shot.
• Number of shots (6–20) and seconds per shot (4–10) — the movie shape. An explicit shot count overrides the length-based auto-fit.
• Visual style — free-text direction woven into the style + every shot prompt (e.g. "moody noir, neon-lit, watercolor").
• Background music volume (0–100%) — a slider for how loud the ducked score sits under the narration. 0% drops the music entirely.
• Narrator voice ID — override TTS_VOICE_ID for this movie.
• Music model and video model — swap models per movie.

Blanks fall back to the operator defaults, and the server bounds the numeric values. No keys are ever involved — the UI pre-fills only non-secret defaults from GET /api/config, and overrides ride along with POST /api/start.

Every run also writes a timestamped storage/sessions/<token>/log.txt with the chosen settings and every progress/degradation message, so you have a persistent record after the movie finishes (it's deleted with the session on the TTL sweep).

---

Graceful degradation

The job always reaches a finished movie or a clear terminal error:

• Every provider call retries a bounded number of times with short backoff.
• A shot that still fails → hold_still substitutes a held still from its anchor image (preserving narration alignment), or drop_shot removes the shot.
• A narration line that fails → that shot plays with no narration.
• The music track that fails → the film plays with no score.
• A malformed LLM plan → one repair attempt; if it still fails, the job stops with a terminal error before any paid video work, so money is never spent on a broken plan.

Every degradation is recorded in the session message log.

---

Security & cost guardrails

• No login, no account — each visit is an anonymous random token.
• API keys are server-side environment variables only; they never appear in client code or responses.
• A per-session quota caps each visit to one movie and bounds retries, so a public free page cannot run up an unbounded bill.
• Session working folders (uploads + intermediate assets) are deleted on a timer (SESSION_TTL_MINUTES), since there is no account to keep them in.

---

Project layout

novel-to-movie/
  main.py                 FastAPI app: routes, SSE progress, static, delivery, cleanup
  config.py               Single ingress for all keys + tunables
  pipeline/
    orchestrator.py       Runs steps 3–10, emits progress, owns fallbacks
    state.py              In-memory session store (+ optional SQLite mirror)
    ffmpeg_utils.py       FFmpeg/FFprobe wrappers
    steps/                step03_extract … step10_assemble
    providers/            base interfaces + anthropic / aggregator / elevenlabs / mock
  web/                    index.html, style.css, app.js (single light-mode page)
  storage/sessions/       per-session working folders (gitignored)
  tests/                  sample_novel.txt + end-to-end mock test

---

Operator: verify before going live

The real provider adapters are written against each service's documented interface, but the exact model ids and a few request/response field names shift over time. Each real adapter carries an OPERATOR VERIFY note describing what to confirm against current docs:

• providers/llm_anthropic.py — endpoint, anthropic-version, LLM_MODEL.
• providers/_aggregator_client.py — base URLs, auth header, result field names for your chosen fal.ai / Replicate models.
• providers/images_aggregator.py, video_aggregator.py, music_aggregator.py — each model's input schema (e.g. duration units, image_url for i2v).
• providers/tts_elevenlabs.py — replace TTS_VOICE_ID with a real voice id and confirm the TTS model_id.

Until those are confirmed, the entire system remains fully exercisable through mock mode.