AI Podcast Clipper

Long podcasts → short, vertical clips — transcription, moment detection, active-speaker framing, and burned-in subtitles, orchestrated end-to-end.

---

What it does

Upload a landscape podcast recording. The pipeline transcribes the audio (WhisperX), finds Q&A and story arcs worth clipping (Google Gemini), reframes around the active speaker for 9:16, renders subtitles, and writes finished clips to S3 while the web app tracks jobs, credits, and downloads.

flowchart LR
  A[Browser upload] --> B[S3]
  B --> C[Inngest job]
  C --> D[Modal GPU worker]
  D --> E[WhisperX + Gemini + CV]
  E --> F[Vertical clips + subs]
  F --> B
  C --> G[(PostgreSQL)]

Loading

---

Features

Capability	Detail
Smart segments	Gemini analyzes word-timed transcripts for ~30–60s clips (stories, questions → answers).
Vertical reframing	Active-speaker detection and crop to 1080×1920 for Shorts / Reels / TikTok.
Accurate captions	Word-level alignment; subtitles burned in with FFmpeg.
Production-shaped app	Google auth, dashboard, Stripe billing hooks, background jobs via Inngest.

---

Repository layout

ai-podcast-clipper/
├── ai-podcast-clipper-frontend/   # Next.js 15 · Prisma · NextAuth · Inngest · S3 · Stripe
└── ai-podcast-clipper-backend/    # Modal app: WhisperX, Gemini, OpenCV, FastAPI endpoint

---

Tech stack

Web app — Next.js (App Router), Tailwind CSS v4, shadcn/ui, Prisma + PostgreSQL, Auth.js (Google), Inngest, AWS S3, Stripe.

ML / video — Modal (GPU class workers), WhisperX, Google Gen AI (Gemini), FFmpeg / OpenCV, active-speaker pipeline under asd/.

---

Prerequisites

• Node.js 18+ and npm
• Python 3.12+ (for local Modal scripts)
• PostgreSQL
• AWS S3 bucket and IAM credentials
• Google OAuth (app) + Gemini API key (Modal secret)
• Modal account (modal setup)
• Inngest keys for durable jobs (see .env.example)
• Stripe (optional for full billing locally)

---

Quick start — frontend

cd ai-podcast-clipper-frontend
cp .env.example .env
# Fill DATABASE_URL, S3, Auth, Modal URL + token, Inngest, Stripe as needed

npm install
npm run db:push
npm run dev

App: http://localhost:3000

In a second terminal, run the Inngest dev server so background steps execute:

npm run inngest-dev

Dashboard: http://localhost:8288

Required env vars are documented in ai-podcast-clipper-frontend/.env.example. Deployment notes live in ai-podcast-clipper-frontend/DEPLOYMENT.md.

---

Quick start — Modal backend

cd ai-podcast-clipper-backend
python -m venv .venv && source .venv/bin/activate   # Windows: .venv\Scripts\activate
pip install modal
modal setup
pip install -r requirements.txt

Configure a Modal secret (e.g. ai-podcast-clipper-secret) with at least GEMINI_API_KEY, AUTH_TOKEN (must match PROCESS_VIDEO_ENDPOINT_AUTH in the frontend), and any AWS keys your worker uses for S3.

Deploy the HTTP worker:

modal deploy main.py

Copy the deployed process_video URL into the frontend PROCESS_VIDEO_ENDPOINT. For a one-off smoke test:

modal run main.py

---

NPM scripts (frontend)

Script	Purpose
npm run dev	Next.js dev server (Turbopack)
npm run inngest-dev	Local Inngest dev UI
npm run build / npm start	Production build & serve
npm run db:push	Sync Prisma schema to the DB
npm run db:studio	Prisma Studio
npm run lint / npm run typecheck	Quality gates

---

Design notes

• Credits + concurrency: Inngest functions gate processing on user credits and limit concurrency per user.
• Split architecture: The Next app never runs GPU work; it delegates to Modal and polls S3 for outputs.
• Auth on the pipeline: The Modal FastAPI endpoint expects a Bearer token checked against your configured secret.

---

Built for creators who want algorithm-ready clips without manual timeline surgery.