While most AI agents only listen to YouTube videos, mine watches, capturing the complete picture to deliver unparalleled understanding and truly comprehensive notes.
U-Tube AI is a multi-modal AI agent that transforms YouTube videos into rich, actionable knowledge assets. Unlike traditional tools that rely solely on a video's transcript, U-Tube AI analyzes both the audio and the visual stream. It reads slides, understands diagrams, and captures code on screen, fusing what's said with what's shown to create notes that are incredibly detailed, accurate, and truly comprehensive.
This system is perfect for students, researchers, and professionals who need to efficiently digest complex tutorials, technical presentations, or academic lectures without losing critical visual context.
The Problem with Existing Tools:
- NoteGPT, Notta, MyMap.AI: Extract YouTube transcripts and summarize text only—completely blind to slides, diagrams, and code shown on screen
- Direct VLM Processing: Feeding entire videos to Gemini costs ~$139/hour at 258 tokens/sec, making it impractical for long educational content
U-Tube AI bridges this gap by combining adaptive frame sampling with intelligent dual-model synthesis, reducing costs by 70-90% while capturing visual context that transcript-only tools miss entirely.
Hence, making it perfect for students, researchers, and professionals digesting technical tutorials, academic lectures, or slide-heavy presentations where visual content is critical.
Unlike mainstream tools that rely solely on YouTube's transcript API, U-Tube AI analyzes both audio and visual streams. Using adaptive frame sampling, it intelligently extracts key visual moments (slides, diagrams, code snippets) and employs OCR to capture text that would otherwise be lost. This approach addresses what research (CVPR 2025 AKS, BOLT) identifies as a critical gap: uniform processing performs poorly for long videos, but selective frame analysis improves accuracy while maintaining efficiency.
What You Get:
- Slide content extracted and integrated into notes
- Code snippets from screen captures preserved
- Diagram descriptions fused with spoken explanations
- No more missing context from visual-only information
Instead of brute-forcing entire videos through expensive models, U-Tube AI uses a two-stage approach optimized for both cost and quality:
Stage 1 - Structure Mapping (Gemini Flash):
- Rapidly analyzes the full transcript to extract topic hierarchy
- Identifies which sections likely contain visual content requiring OCR
- Creates comprehensive outline in seconds
Stage 2 - Deep Synthesis (Gemini Pro):
- Performs targeted "deep dives" on each section
- Fuses transcript + extracted visual data into coherent narratives
- Generates production-quality notes with rich context
Cost Impact: Processing full hour-long videos with Gemini Pro alone costs ~$139. This dual-model approach reduces processing cost significantly while maintaining comprehensive understanding.
Not every video needs frame-by-frame analysis. U-Tube AI includes a confidence-based decision engine that:
- Evaluates whether visual content would enhance note quality
- Triggers OCR processing only for visually dense sections
- Implements iterative refinement loops for low-confidence outputs
- Avoids unnecessary processing for talking-head videos
This intelligent routing ensures you're not paying for OCR when transcripts alone suffice, while guaranteeing visual extraction when slides or diagrams appear.
Beyond note generation, U-Tube AI implements a vector-based RAG pipeline (ChromaDB) that enables:
- Semantic search across video content with timestamp retrieval (Cosine Similarity)
- Q&A with context-aware responses drawing from both transcript and visual data
- Multi-round conversations with confidence tracking
- Automatic OCR integration when queries require visual context
Final notes aren't locked in a proprietary format. Download as:
- Markdown (.md) for documentation systems
- PDF for presentations and archival
- Word (.docx) for collaborative editing
| Feature | NoteGPT/Notta/MyMap.AI | Direct VLM Processing | U-Tube AI |
|---|---|---|---|
| Visual Content | ❌ Transcript only | ✅ Full video | ✅ Adaptive sampling |
| Cost (1hr video) | ~$5 | ~$139 | ~$15-40* |
| Processing Speed | Fast | Slow | Moderate |
| Slide Extraction | ❌ | ✅ | ✅ |
| Handles Talking-Head | ✅ | ✅ (expensive) | ✅ (optimized) |
*Estimated based on dual-model architecture and selective frame processing
Frame Sampling Strategy: Built on principles from recent computer vision research (AdaFrame CVPR 2019, AKS CVPR 2025, BOLT 2025) showing that intelligent frame selection outperforms uniform sampling by5-6% while processing 70-85% fewer frames.
Multimodal Fusion: Implements hierarchical synthesis where structural analysis (topics/subtopics) guides targeted visual extraction, ensuring OCR effort focuses on information-dense segments rather than redundant frames.
Cost-Quality Tradeoff: Gemini's video processing capability (documented at 258 tokens/sec @ 1 FPS) makes full-video analysis prohibitively expensive for educational content. U-Tube AI's selective approach maintains >90% information capture at fraction of the cost.
Educational YouTube videos (Strivers, Programming with Most, CampusX, etc..) increasingly rely on visual aids—code editors, slides, architectural diagrams—that don't appear in auto-generated transcripts. As of 2025, major AI note-taking tools still process only audio/text, leaving students manually pausing videos to copy slide content.
U-Tube AI is my try of solving this by treating visual information as first-class data, not an afterthought, while maintaining practical cost efficiency for real-world use.
[START: User provides YouTube URL, Task, & Language]
|
V
[1. Transcript Generation (via Flash Model)]
|
V
(Raw Transcript)
|
V
[2. Language Processing & Translation]
|
|
+------------------------------------------+------------------------------------------+
| | |
V V V
[BRANCH 1: INTERACTIVE TASKS] (Processed Transcript) [BRANCH 2: GENERATIVE TASKS]
| |
| |
+----------------+-----------------+ +-------------------+-----------------+
| | | |
V V V V
[Task A: Translate Video] [Task B: Chat with Video] [Task C: Important Points] [Task D: Note Generation]
| | | |
L--> (Output: Final +-->[B1. Vectorize & Create ChromaDB] L-->[C1. Topic Extraction (Flash)] (Uses "Imp. Points" as base)
Processed Transcript) | | |
+-->[B3. Find Chunks & Time Frames (Flash)] L-->(Output: Detailed +-->[D1. Initial Note Gen (Pro)]
| Topics & Subtopics) | (Transcript-only notes)
+-->[B5. Initial Answer Gen (Pro)] |
| | +-->[D2. Visuals Decision]
| L-->(Output: Answer, is_ocr_needed, confidence) | |
| | +-->(If Visuals = NO)-->[D6]
+-->[B6. OCR Decision] |
| | L-->(If Visuals = YES)
| +-->(If YES)-->[B7. Refine with OCR (Pro)] |
| | +-->[D3. Visual Extraction]
+-->[B8. Confidence Check & Loop] | | (Adaptive Sampling/OCR)
| (If low, repeat B5-B8 up to 3x) | |
| | +-->[D4. Iterative Loop (Flash)]
L-->(Output: Final, high-confidence answer) | | (Enhances 2 sub-topics
| | at a time with OCR)
| |
| L-->[D5. Final Assembly]
| |
| V
L-->[D6. Download Options]
|
+-->(.md File)
+-->(.pdf Document)
L-->(.docx Document)
- Language: Python
- UI Framework: Streamlit
- LLM Models: Google Gemini Pro & Gemini Flash
- LLM SDK:
google-generativeai(for interacting with the Gemini API) - Vector Database (RAG): ChromaDB
- Video/Image Processing: OpenCV
- Transcript Fetching:
youtube-transcript-api
- Langchain
- Role based Prompting
- Few Shot Prompting
- Case based Prompting
- Environment Management:
python-dotenv(for loading API keys from the.envfile)
Follow these steps to get a local copy up and running.
You will need a Google AI Studio API key to use the Gemini models.
- Get your API key from Google AI Studio.
git clone https://github.com/Rishab27279/U-Tube-AI.git
cd U-Tube-AI
python -m venv venv
.\venv\Scripts\activate
pip install -r requirements.txt
copy example.env .env
Add your API Key: Open the newly created .env file and add your Google AI Studio API key.
env GOOGLE_API_KEY=YOUR_API_KEY_HERE
git clone https://github.com/Rishab27279/U-Tube-AI.git
cd U-Tube-AI
python -m venv venv
source venv/bin/activate
pip install -r requirements.txt
cp example.env .env
Add your API Key: Open the newly created .env file and add your Google AI Studio API key.
env GOOGLE_API_KEY=YOUR_API_KEY_HERE
To start generating notes, run the below code in terminal where you followed the above mentioned process
streamlit run app.pyI warmly welcome contributions! Whether it's bug fixes, new features, or documentation improvements, please feel free to open issues and submit pull requests.
- 🐛 Bug fixes
- ✨ New features or feature ideas
- 📖 Documentation improvements
- 🎨 UI/UX enhancements
If you find this project helpful, your support would mean the world:
- Give it a star ⭐
- Share your feedback
- Try it out and report any issues
---
Made by Rishab with ❤️ & 🔥 for the community