Skip to content

Add optional TwelveLabs Pegasus transcription backend - #22

Open
mohit-twelvelabs wants to merge 1 commit into
lewangdev:mainfrom
mohit-twelvelabs:feat/twelvelabs-integration
Open

Add optional TwelveLabs Pegasus transcription backend#22
mohit-twelvelabs wants to merge 1 commit into
lewangdev:mainfrom
mohit-twelvelabs:feat/twelvelabs-integration

Conversation

@mohit-twelvelabs

Copy link
Copy Markdown

Hi! I'm Mohit, I work at TwelveLabs (@mohit-twelvelabs).

This adds an opt-in transcription backend to the notebook using TwelveLabs Pegasus, a video-understanding model, as an alternative to Faster Whisper.

What it adds

  • A new markdown + code cell pair right after Run the model.
  • Tick use_twelvelabs to transcribe with Pegasus; leave it unchecked and the notebook behaves exactly as before (Whisper output is kept).
  • The Pegasus cell writes the same {start, end, text} fragments shape Whisper produces and sets language_detected, so the existing Merge and Translate cells run unchanged.

Why it helps this project

Pegasus watches the video server-side rather than only listening to the audio, so it tends to do better on on-screen text, speaker context and noisy audio — and it needs no local GPU, which is handy for users without an RTX card or who hit Colab GPU limits.

Opt-in / non-breaking

Defaults are untouched: use_twelvelabs = False. No existing cell, dependency or default behavior changes. The only new dependency (twelvelabs>=1.2.8) is installed lazily inside the opt-in cell.

How it was tested

  • Verified the client.analyze(model_name=, video=VideoContext_Url(url=), prompt=, max_tokens=) signature and response.data field against the official SDK twelvelabs==1.2.8.
  • Unit-checked the JSON-array extraction (handles prose/code-fence wrapping and drops empty segments).
  • Live-smoke-tested the SDK + auth path against the TwelveLabs API (a Marengo text embedding returned a 512-dim vector). A full Pegasus run over a public video URL works server-side but can be slow, so the request wiring is what's asserted here.

You can grab a free API key at https://twelvelabs.io — there's a generous free tier.

Adds an opt-in notebook cell that transcribes the video with TwelveLabs
Pegasus instead of Faster Whisper. Pegasus watches the video server-side
(no GPU) and emits the same {start, end, text} fragments, so the existing
merge and translate cells work unchanged. Defaults keep Whisper; enabled
only when use_twelvelabs is ticked.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant