FrameWise AI is an AI-powered video learning platform that transforms video content into structured and interactive learning resources. Users can upload local videos or process YouTube videos and automatically generate transcripts, summaries, quizzes, and context-aware AI conversations.
The project was built to solve a common problem: learning from video content is often passive, time-consuming, and difficult to revisit efficiently. While videos are excellent for consuming information, they are poor knowledge repositories because users cannot easily search, query, or interact with the content.
FrameWise AI converts videos into searchable knowledge assets.
Modern learning increasingly happens through videos:
- YouTube tutorials
- Technical conference talks
- University lectures
- Online courses
- Podcasts
However, videos present several challenges:
- Difficult to search for specific concepts
- Requires repeatedly rewatching content
- Notes are often disconnected from the source material
- No efficient way to ask questions about video content
- Learning retention is low without active engagement
The goal was to build a system that allows users to:
- Upload a video
- Generate a transcript automatically
- Produce concise study notes
- Ask questions about the content
- Generate quizzes for knowledge retention
The platform consists of two major components:
Technology:
- Flask
- HTML
- CSS
- JavaScript
Responsibilities:
- User interface
- Video uploads
- YouTube URL submission
- Chat interface
- Quiz rendering
- API proxy layer
Technology:
- FastAPI
- LangChain
- Gemini
- Faster-Whisper
- ChromaDB
Responsibilities:
- Video processing
- Audio extraction
- Speech transcription
- Summary generation
- Vector storage
- Retrieval-Augmented Generation
- Quiz generation
The user uploads a local video or provides a YouTube URL.
The system stores the video using a unique identifier.
Output:
video_id.mp4
FFmpeg extracts audio from the video.
Input:
video.mp4
Output:
video.mp3
Reason:
Speech recognition performs significantly better on isolated audio streams than directly processing video files.
The extracted audio is processed using Faster-Whisper.
Responsibilities:
- Speech-to-text conversion
- Transcript segmentation
- Timestamp generation
Output:
{
"text": "...",
"segments": [...]
}The transcript is passed to Gemini.
The summarization pipeline generates:
- Key ideas
- Important concepts
- Structured notes
Output:
Summary
Key Takeaways
Important Concepts
Long transcripts cannot be sent directly to an LLM.
The transcript is split into semantic chunks.
Purpose:
- Efficient retrieval
- Reduced token usage
- Improved answer relevance
Implementation:
Custom transcript chunking strategy.
Each chunk is converted into embeddings.
Storage:
ChromaDB
Purpose:
- Semantic similarity search
- Context retrieval
When a user asks a question:
- Query embedding generated
- Similar transcript chunks retrieved
- Context assembled
- Gemini generates grounded response
This ensures answers are based on actual video content rather than model assumptions.
Quiz generation uses Gemini.
Input:
- Transcript
- Summary
Output:
- Multiple-choice questions
- Knowledge checks
- Concept reinforcement
Purpose:
Transform passive viewing into active learning.
Large videos generate massive transcripts.
Problem:
Sending entire transcripts to an LLM is expensive and inefficient.
Solution:
Implemented transcript chunking and semantic retrieval.
Result:
Lower token usage and improved response quality.
Direct LLM conversations frequently produced inaccurate answers.
Solution:
Implemented Retrieval-Augmented Generation using:
- LangChain
- ChromaDB
- Gemini
Result:
Responses remain grounded in transcript content.
Video processing involves multiple expensive operations:
- File upload
- Audio extraction
- Transcription
- Summarization
- Embedding generation
Solution:
Designed a modular pipeline architecture.
Benefits:
- Easier debugging
- Better maintainability
- Independent component testing
The frontend should not directly expose backend infrastructure details.
Solution:
Implemented a Flask proxy layer.
Benefits:
- Backend URL abstraction
- Cleaner frontend architecture
- Easier deployment flexibility
Reasons:
- High performance
- Async support
- Automatic API documentation
- Excellent developer experience
Reasons:
- Strong summarization quality
- Cost-effective API access
- Good reasoning capabilities
Reasons:
- Lightweight setup
- Local vector storage
- Easy integration with LangChain
Reasons:
- Simplified RAG development
- Retrieval orchestration
- Prompt management
- Future extensibility
Key modules:
Handles:
- Audio extraction
- Speech transcription
Handles:
- Gemini note generation
- Summary creation
Handles:
- ChromaDB integration
- Embedding storage
Handles:
- Retrieval pipeline
- Context generation
- Conversational workflows
Handles:
- Quiz generation
Current processing is synchronous.
Future:
- Celery
- Redis queues
- Background workers
Current:
- Semantic retrieval
Future:
- Hybrid search
- Reranking
- Metadata filtering
Planned features:
- User accounts
- Saved notes
- Multi-video collections
- Export to PDF
- Flashcards
- Spaced repetition
This project provided practical experience with:
- Retrieval-Augmented Generation
- LangChain architecture
- Vector databases
- Speech-to-text pipelines
- LLM integration
- API design
- Modular backend development
- End-to-end AI application engineering
The most important lesson was that building production-grade AI systems is less about the LLM itself and more about designing reliable pipelines, retrieval systems, and data workflows around the model.