Video Clipper — long talks → vertical shorts

Turns selected moments of a long video of people talking (a lecture, podcast, sermon, interview or any show) into 9:16 shorts: pauses removed with optical-flow "morph" cuts, a crop that holds the main speaker on every camera (or wherever you place it), clean hard cuts on camera changes, styled word captions (right-to-left for Arabic) and an optional name plate. Arabic talks also get Quran and hadith quotations recognised and corrected.

Needs ffmpeg (NVENC is used when available), Pillow built with raqm and Node 18+ for the web app. ./install.sh installs the Python packages: with an NVIDIA GPU it adds requirements-gpu.txt; without one it installs the CPU build of PyTorch instead.

Without a GPU everything still works, only slower: Whisper transcribes on the CPU (about a third of the video's length on a 16-thread CPU, so an hour takes ~20 minutes; fewer cores take longer) and translation is much slower than on a GPU. Arabic is transcribed by Whisper alone, without the Cohere re-read and the doubtful-word marks, which need the GPU.

Models download on first use: about 11 GB for transcription (Whisper 2.9 GB, the two Cohere models 3.9 GB each, NVIDIA's checker 0.4 GB) and about 8 GB more if you use translation. Only one runs on the GPU at a time (peak about 4.5 GB). Cohere Transcribe Arabic is gated: accept its terms on its Hugging Face page and log in once with hf auth login; without it, transcription still works with the Tashkeel model alone. See Credits for every model.

Web app

./install.sh   # once: .venv, Python packages, web build (re-run after pulling updates)
./start.sh     # runs in the background, prints the address (8000, or the next free port)
./stop.sh      # stops it

Logs go to .run/server.log. Heavy work runs at low priority. Long shorts are cut in passes of at most 8 clips, so memory stays around 2 GB however long the short is. Encoding uses the GPU (NVENC) when available; otherwise a light CPU setting is used, and the render log says which. To run in the foreground instead: python -m server.

• Library (the home page) – paste a YouTube link or a file path already on this computer (linked, not copied) into one field, or drop a video anywhere on the page. Each video is analysed once: waveform, pauses, thumbnail. Each video lists its shorts; click one to open it.

  

• Editor – one screen, left to right: the transcript (search; click a sentence to select it, shift+click to extend, double-click to correct the text, and corrections are used in the captions), the video, then the preview and settings (clips, cuts, captions, framing). The waveform timeline runs under the transcript and video (I / O to mark, Enter to add, drag to select, drag along the time ruler to scrub, scroll to move, Ctrl+scroll to zoom). Shorts are tabs in the top bar, next to the Render button. Selecting in either place selects in both. Words the recognisers disagreed on are underlined; click one to pick one of the other readings (or keep it) in one click. The word keeps its timing. What's in the short is shown in red everywhere: transcript, 9:16 frame and timeline.

• Framing: each video is analysed once. Camera angles are recognised by matching the architecture in the picture, so a moving audience, a pan or a small zoom doesn't split one angle into several. The main speaker is found by face recognition, so they are framed even in wide shots where audience faces are bigger, and held in the same spot every time an angle comes back. Pause and drag the orange frame on the video (or a camera card in the Framing tab) to set an angle by hand. It's stored relative to the scene and mapped onto every shot of that angle, even if the camera moved. Each angle can also have its own zoom. The CLI does the same, cached in <video>.cameras.json/.npz; use --no-lock to just follow the main face. Angles seen only briefly (often cutaways) are framed automatically and hidden from the list; "Show them" lists them too.

  

• Settings (pause threshold, morph/fade/jump cuts, caption style, name plate and caption height, zoom in or out, grade, export speed) update a live 9:16 preview; Render runs in the background and the result plays and downloads in place.

• Accurate Arabic: Arabic lectures are transcribed with Whisper for the word timings, then the audio is re-read by Cohere Transcribe Arabic for the words (in up to 35 s chunks cut at pauses, as Cohere's own long-form mode does: it reads much better with that context) and its fine-tune Cohere-Speech-Tashkeel-2B for the harakat. Audio is prepared exactly as Cohere's reference code does (librosa-style downmix and soxr resampling), since the models are sensitive to it. Every word is then checked against the Tashkeel model, Whisper and NVIDIA's FastConformer (aligned word by word): where two agree on something else, the majority wins (a lone «و» of difference never replaces a word); where only the Tashkeel model, Cohere's own fine-tune, backs a word that Whisper and NVIDIA both dispute, it's flagged; where nobody backs a word it gets a red wavy underline (amber dotted = changed by majority), and hovering shows what the others heard. Captions can show harakat everywhere, only in Quran quotations, or not at all. Older transcripts get an Upgrade to accurate Arabic button.

• Quran and hadith: quotations are found by matching the transcript against the whole Quran and the six hadith books (helped by cues such as «قال تعالى» or «قال رسول الله ﷺ»). Without a cue a quotation needs 5 matching words; right after one, 2 are enough (e.g. «وَٱلرُّجْزَ فَٱهْجُرْ»), and a verse the speaker says again straight away counts too. Quran quotations are replaced by the exact Uthmani text and linked to quran.com. Words that fit several places in the Quran are marked as Quran without naming a surah; when the speaker discusses a reading («وفي قراءة», «قرأ يعقوب», «بالكسر») their own harakat are kept instead of Hafs. Hadith show their collection, number, grading and sunnah.com link; inside a hadith, a word one of the recognisers heard the collection's way (or a doubtful one a letter off) is corrected to its wording («الطاعون ريجز» → «الطَّاعُونُ رِجْزٌ»), and the speaker's own phrasing is otherwise kept. A word of the hadith that no recogniser heard between two that were (often a quick «لَهُ») is offered as a dashed «+ لَهُ» after its word: click it if it was said, × if not. It is never added on its own. Doubtful words one letter from a word of the lecture's title become that word («والرجل» → «والرِّجز» in a lecture on الرجز). Texts download once into models/texts/. In the video, a small gold line above the captions names the surah and ayah while a verse is quoted («سورة البقرة، الآية ٩٣», or "Quran · Al-Baqarah 2:93" with English captions); it can be turned off.

• Proofread with Gemini (optional, online): with GEMINI_API_KEY in .env (see .env.example; a free key from aistudio.google.com works), the transcript gets a Proofread doubtful words with Gemini button. The audio and transcript go to Google's Gemini, which hears the speech and knows Arabic, the Quran and hadith; only the underlined words can change (blue underline, its reason on hover), Quran text never; a word misheard as two is joined back («وسوسة هو» → «وسوسته»). One request per 30 minutes of audio. If the model is busy or out of free quota, the next newest Flash model is tried. The key can live in .env or .env.local (both git-ignored). On the free tier Google may use what you send to improve its products. On the Sa'd test lecture it took word errors from 6.6 % to 6.2 % (letters 4.1 % → 3.3 %).

• Suggested moments (optional, online, needs the Gemini key): Suggest moments above the transcript sends the transcript text (not the audio) to Gemini, which picks complete thoughts of 20–90 s that stand on their own (a question with its answer, a story with its point) and says why. Click one to select it, then add it to a short. Moments already in the short get a red bar.

• Who to frame: under Framing, The main speaker holds the person on screen most (lectures, sermons). Whoever is talking is for podcasts and interviews: in a shot with several people, each mouth is measured (anchored on the face's mouth corners, so near and far faces count the same) and matched against the sound, and the frame cuts to the person talking and holds still for their whole turn; it never pans from one person to another. It applies to the render; the quick preview keeps following the main speaker.

  

• Safe area: under the preview, show where TikTok, Reels or Shorts put their buttons and text, so faces and captions stay clear of them (approximate).

• Post text: under Export, a title, description and hashtags for the short (yours, or Write with Gemini from what the short says, in its language), with Copy all for pasting into any app.

  

• Post to YouTube (optional): upload the rendered short to your own channel (private, unlisted, public, or scheduled), with its post text and #Shorts. Setup, once: in Google Cloud create a project, enable the YouTube Data API v3, set up the OAuth consent screen (External, add yourself as a test user), create an OAuth client of type Desktop app, and put GOOGLE_CLIENT_ID= and GOOGLE_CLIENT_SECRET= in .env.local. Restart, then Connect YouTube channel under Export signs in on Google's own page. The token stays in data/accounts/ on this computer. Until Google verifies your app, uploads from an unverified app may be locked to private. TikTok and Instagram need an app reviewed by them (and Instagram a public video URL), so they aren't built in yet.

• Translation (optional): Translate to English (or to Arabic, for an English lecture) translates every sentence on the GPU with Qwen3-4B-Instruct. Whole Quran verses use the Saheeh International translation word for word; hadith use the collection's English as a reference. Show it under each sentence, double-click to correct it (your corrections survive re-translating), and pick it as the caption language.

• Speed: the preview can play at 1.25–2× on its own; Export speed speeds up the render with the voice pitch kept (ffmpeg atempo).

• Under the video and the preview: a volume control (slider and mute, one setting for the whole app, remembered).

• Check a cut exactly (under the preview): the instant preview only approximates morph cuts; pick one removed pause and render just that moment as the export will look.

App data lives in data/ (override with SHORTS_DATA=/path). For frontend development run npm run dev in web/ (proxies /api to port 8000).

Command line

# just give time ranges (from any video player) — no transcript needed;
# only the selected ranges are transcribed for captions
python -m shorts make lecture.mp4 --range 12:30-13:45 -o out/short1.mp4

# optional: full transcript to search/read when picking ranges
python -m shorts transcribe lecture.mp4    # -> lecture.transcript.txt

Ranges are START-END (75-90, 1:15-1:30.5, 1:01:15-1:02:00), repeatable; they are joined in order. --no-captions skips speech recognition entirely.

Transitions

--transition morph (default) warps the last frame before a pause into the first frame after it using optical flow; fade is an eased dissolve; cut leaves jump cuts. --transition-frames 4 sets the length (6 = softer). Joins that land on a camera change are always clean hard cuts, and the crop switches on that exact frame.

Caption styles

--style pill|box|outline|minimal (default pill), or a JSON file that starts from a preset and overrides any field — see styles/example.json:

key	meaning
font	fontconfig name ("Vazirmatn:style=Bold") or path to a .ttf
size, color, highlight	text size, colour, colour of the spoken word (null = off)
box, box_color, box_opacity, radius, pad_x, pad_y	background box
outline, outline_color, shadow	text stroke and soft shadow
y	vertical centre (0 = top, 1 = bottom)
max_words, max_chars, fade_ms	caption chunking and fade-in

captions styles the subtitles, title the name plate (--title "...").

Other flags: --min-gap 0.4 (cut pauses longer than this), --zoom 1.15 (tighter crop), --grade, --speed 1.25 (faster, same voice pitch), --keep-temp.

How it works

1. plan (shorts/plan.py) – Silero VAD finds pauses in the selected audio; speech is padded, every segment snapped to whole frames.
2. cut (shorts/cut.py) – one seeked input per segment, video hard-joined, audio crossfaded over the transition length (A/V stay frame-exact).
3. reframe (shorts/reframe.py) – YuNet face tracking, damped virtual camera, camera-change detection at the exact frame.
4. transitions (shorts/transitions.py) – morph/fade frames synthesized at same-camera joins; the crop glides across them.
5. render (shorts/render.py, shorts/captions.py) – crop, scale to 1080×1920, composite Pillow-rendered captions, encode with NVENC.

shorts/pipeline.py wraps all of this in one call (make_short) used by both the CLI and the web server (server/: FastAPI, a GPU job queue, JSON storage).

Where files go

• data/lectures/<id>/: one folder per lecture: the downloaded or uploaded video (source.mp4), rendered shorts (renders/), the transcript and translation, the analysis (waveform, cameras) and thumbnails. A video imported "from this computer" is only linked here, not copied.
• Deleting a lecture in the library removes its whole folder, renders included, and says how much space that frees. For a linked video you can also tick "Also delete the original video", which removes that file and the files the command-line tool keeps next to it. A lecture can't be deleted while a job (download, transcription, render...) is still running for it.
• data/styles/: saved caption styles.
• ~/.cache/huggingface/: the AI models (about 19 GB for everything the app uses; see Credits), shared with other apps on this computer.
• ./stop.sh won't stop the app while it's still working; ./stop.sh -f does.

Tests

python tests/test_sources.py   # Quran/hadith recognition, title vocabulary, word alignment (no GPU)
python tests/test_credits.py   # every model and dataset is credited below

Checking a render

python tests/verify_render.py <video> --range 322-445 renders a short and checks it is exactly what the editor planned. It compares the cut plan with the editor's estimate, checks the length to the frame, compares sampled frames with the planned crop of the source, checks manual framing and per-camera zoom, and checks audio sync.

Credits

Models and texts this app downloads and uses. Keep this table in step with the code: python tests/test_credits.py fails when a model or dataset referenced in shorts/ is missing here.

Used for	Model / data	Licence
Speech recognition, word timings	Whisper large-v3 (OpenAI) via faster-whisper (large-v3)	MIT
Accurate Arabic words	CohereLabs/cohere-transcribe-arabic-07-2026 (gated: accept its terms on Hugging Face once; without it the Tashkeel model is used alone)	Apache-2.0
Harakat (tashkeel)	NAMAA-Space/Cohere-Speech-Tashkeel-2B, fine-tuned from Cohere Transcribe Arabic	Apache-2.0
Second opinion on every word (doubtful-word marks)	NVIDIA FastConformer Arabic (pcd), ONNX export OpenVoiceOS/stt_ar_fastconformer_hybrid_large_pcd_v1.0_onnx, run with onnx-asr	CC-BY-4.0
Pause detection	Silero VAD (bundled with faster-whisper)	MIT
Proofreading doubtful words (optional, online)	Google Gemini API, newest Flash model your key can use (or GEMINI_MODEL)	Google's API terms
Translation	Qwen/Qwen3-4B-Instruct-2507	Apache-2.0
Face detection	YuNet (face_detection_yunet_2023mar.onnx, OpenCV Zoo)	MIT
Recognising the speaker	SFace (face_recognition_sface_2021dec.onnx, OpenCV Zoo)	Apache-2.0
Quran text (Uthmani)	arbml/quran_uthmani, which appears to be the Tanzil Uthmani text: used verbatim, as Tanzil's terms require	Tanzil terms
Quran in English	Saheeh International, from nassimjp/quran (en-saheeh)	© Saheeh International; downloaded at runtime, not redistributed
Hadith (six books, grading, English)	meeAtif/hadith_datasets, from sunnah.com	MIT