ClipAI

AI-powered video clipping and transcription tool.
Transform long videos into viral clips, generate intelligent summaries, and transcribe content from 30+ platforms.

🚀 Features

🎬 Video Clipping (Core)

• AI Clip Detection - Automatically finds engaging viral moments using semantic analysis.
• Smart Resizing - Converts landscape (16:9) video to portrait (9:16) with active speaker tracking.
• Facecam Detection - Intelligent layout handling for gaming/streaming videos.
• Viral Captions - Generates social media descriptions and hashtags using AI (Gemini/OpenAI/Anthropic).
• Video Editor - Trimming, resizing, and burning subtitles.
• Real-time Export Progress - Live updates during clip export with automatic download on completion.

🛠️ Recent Improvements

• ✅ Fixed Torch Import Errors - Graceful fallback when torch is not available
• ✅ Real Export Functionality - Frontend now uses actual API calls (no more simulations)
• ✅ Environment Configuration - Proper API URL configuration for frontend
• ✅ Improved Error Handling - Better user feedback for connection issues
• ✅ All Dependencies Installed - Critical packages verified and working

📝 AI Transcriber (New in v1.1)

• Multi-Platform Download - Supports 30+ platforms including YouTube, TikTok, Instagram, Twitter/X, Bilibili, Vimeo, and more.
• High-Accuracy Transcription - Uses Faster-Whisper with VAD filtering for precise word-level timestamps.
• AI Text Optimization - Automatically fixes typos, punctuation, and grammar using LLMs.
• Intelligent Summarization - Generates comprehensive summaries, key points, and topic lists in 20+ languages.
• Translation Service - Translates transcripts between languages with auto-detection.
• Real-Time Progress - Live progress tracking via Server-Sent Events (SSE).
• Export Formats - Download results in Markdown, SRT, VTT, and JSON.

---

🛠️ Tech Stack

• Backend: FastAPI, Python 3.11+
• Frontend: Next.js 15, React, TailwindCSS, shadcn/ui
• AI/ML:
  • Transcription: Faster-Whisper
  • LLMs: Google Gemini 2.0, OpenAI GPT-4o, Anthropic Claude
  • Embeddings: Sentence-Transformers
  • Vision: OpenCV, MediaPipe (Face detection)
• Processing: FFmpeg, MoviePy, yt-dlp

---

⚡ Quick Start

Prerequisites

• Python 3.11+
• Node.js 20+
• FFmpeg (must be installed on your system)
• libmagic

Installation

Quick Start: See QUICKSTART.md for detailed step-by-step instructions.

1. Clone the repository

   git clone https://github.com/yourusername/clipai.git
   cd clipai

2. Setup Backend

   cd backend
   
   # Install Python dependencies (includes torch, faster-whisper, etc.)
   pip install -r requirements.txt
   
   # If torch installation fails, install CPU version:
   pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cpu
   
   # Setup environment variables
   cp .env.example .env
   # Edit .env with your API keys (GOOGLE_API_KEY, HUGGINGFACE_TOKEN, etc.)
   
   # Run the server (or use start.bat/start.sh scripts)
   python main.py

3. Setup Frontend

   cd frontend
   
   # Install dependencies
   npm install
   
   # Create environment file (if not exists)
   echo "NEXT_PUBLIC_API_URL=http://localhost:8000" > .env.local
   
   # Run development server
   npm run dev

4. Access the Application

   • Frontend: http://localhost:3000
   • Backend API: http://localhost:8000
   • API Docs: http://localhost:8000/docs

Startup Scripts:

• Windows: Double-click start.bat
• Linux/macOS: Run ./start.sh

🐳 Using Docker

# Copy env file
cp backend/.env.example backend/.env

# Start services
docker-compose up -d

# View logs
docker-compose logs -f

---

📖 API Documentation

1. Transcription & Summary API (Standalone)

Transcribe from URL (30+ platforms supported):

POST /api/transcribe/url
Content-Type: application/json

{
  "url": "https://www.youtube.com/watch?v=...",
  "language": "en",           // Optional: Source language (auto-detect if null)
  "generate_summary": true,   // Generate AI summary
  "summary_language": "es",   // Optional: Output summary in Spanish
  "generate_translation": true,
  "translation_language": "es"
}

Transcribe from File:

POST /api/transcribe/file
Content-Type: multipart/form-data

file: (binary video/audio file)
generate_summary: true

Stream Progress (SSE):

GET /api/transcribe/stream/{job_id}

Get Results:

GET /api/transcribe/result/{job_id}

Export:

GET /api/transcribe/export/{job_id}  // Download Markdown

2. Video Clipping API

Upload & Process:

POST /api/upload/url
Content-Type: application/json

{
  "url": "https://www.tiktok.com/@user/video/...",
  "aspect_ratio": [9, 16],
  "generate_description": true,
  "generate_summary": true
}

Get Detected Clips:

GET /api/clips/{job_id}

---

🌍 Supported Platforms & Languages

Video Platforms

Supports downloading from 30+ sites including:

• YouTube, TikTok, Instagram, Facebook, Twitter/X
• Bilibili, Vimeo, Twitch, Reddit, LinkedIn
• Pinterest, Tumblr, SoundCloud, and many more.

Summary Languages

English, Chinese (Simplified/Traditional), Japanese, Korean, Spanish, French, German, Portuguese, Russian, Arabic, Hindi, Italian, Dutch, Polish, Turkish, Vietnamese, Thai, Indonesian, Malay, Ukrainian.

---

🤝 Contributing

Contributions are welcome! Please read our contributing guidelines before submitting PRs.

📄 License

MIT License - see LICENSE for details.

---

Built with ❤️ by the ClipAI Team.