Status: 🚀 Feature-Complete All 11 stages implemented, fully isolated, and tested.
BriefAI converts live speech or pasted meeting notes into structured summaries, translations, action items, academic notes, decisions logs, and custom-defined outputs. It includes a unified Workspace, a "Ask BriefAI" RAG-powered chat, a Custom Template Builder, and Speaker Diarization all running locally via open-weight LLMs, strictly isolated per user with JWT authentication.
📹 Watch the demo: BriefAI full walkthrough (YouTube)
BriefAI's pipeline: from raw audio/text input through transcription, speaker diarization, LLM processing, and an isolated, grounding-checked RAG chat layer all running locally.
BriefAI is designed to run completely locally, meaning transcription and LLM inference happen on your machine:
- GPU (Recommended): A dedicated graphics card with at least 6 GB of VRAM (e.g. Nvidia RTX 3060/4060 series, or Apple Silicon Mac M1/M2/M3 with unified memory) is highly recommended. This delivers sub-second to single-digit-second latencies (30–60+ tokens/sec) for a real-time experience.
- CPU Fallback: If no dedicated GPU is found, Ollama and Whisper will run on your CPU. While fully functional, latencies can be significantly higher (e.g., 25–45 seconds per LLM request depending on your CPU, which translates to 8–15 tokens/sec).
- Perceived Latency Mitigation: To offset CPU latencies, all LLM-based API routes support chunk-by-chunk token streaming (
stream=True). The frontend can read and display progressive tokens in real time as they are generated by Ollama.
The easiest way to stand up the BriefAI web architecture is using Docker Compose. This instantly provisions the frontend, backend, and database without needing Node.js or Python environments.
Tip
End-to-End Verified: The Docker configuration has been fully tested and verified end-to-end. Both frontend (Nginx) and backend (FastAPI/Alembic) containers build and communicate seamlessly in isolated environments.
Caution
AI Models Disclaimer: While Docker brings the web app up instantly, the AI models (Ollama Qwen/Llama and Whisper/Diarization models) must still be pulled manually and downloaded on first run. Without GPU passthrough (which is complex across OS boundaries), the backend defaults to CPU-only inference. Expect significant latency (often 60s+) for transcription and summarization tasks.
- Install Docker Desktop (or Docker + docker-compose on Linux).
- Install Ollama natively on your host machine.
- Pull the required models in Ollama:
ollama pull qwen3:1.7b ollama pull llama3.2:1b ollama pull nomic-embed-text
- Start the BriefAI application:
docker-compose up --build -d
- Navigate to
http://localhostin your browser.
We implemented an automated benchmarking module to compare the speed, throughput, context loading, memory utilization, and quality of Qwen3-1.7B (qwen3:1.7b) vs. Llama 3.2-1B (llama3.2:1b) on identical meeting transcripts.
The following head-to-head comparison was evaluated on identical inputs:
| Model | Workload | Status | TTFT (ms) | Latency (s) | Throughput (tok/s) | Context Speed (tok/s) | Memory RSS (MB) | Peak RAM (MB) | Quality (1-5) |
|---|---|---|---|---|---|---|---|---|---|
qwen3:1.7b |
Small | Empty (Token Cap) | 14321.5 | 14.32 | 11.7 | 1515.6 | 3428.3 | 3448.9 | N/A |
llama3.2:1b |
Small | Success | 1603.5 | 10.63 | 11.6 | 1556.8 | 3467.0 | 3467.0 | 5.0 |
qwen3:1.7b |
Medium | Empty (Token Cap) | 14851.9 | 14.85 | 11.2 | 3983.2 | 3477.0 | 3477.0 | N/A |
llama3.2:1b |
Medium | Success | 1657.1 | 15.49 | 11.1 | 3649.1 | 3508.8 | 3508.8 | 4.5 |
qwen3:1.7b |
Large | Success | 29870.7 | 40.39 | 9.0 | 7793.0 | 3584.7 | 3584.7 | 3.5 |
llama3.2:1b |
Large | Success | 1411.8 | 25.64 | 10.2 | 8053.1 | 3492.6 | 3492.7 | 5.0 |
Metrics average of 3 trials after initial warmup. RSS/Peak memory tracks aggregate host RAM of active llama-server.exe processes.
A significant finding during Stage 5 benchmarking was the Reasoning Token Cap Failure Mode in qwen3:1.7b (which behaves as a distilled reasoning model):
- The Issue: Reasoning models allocate a significant portion of their token budget (typically 150–200 tokens) to the inner
<thinking>loop before producing the first response token. When enforcing a token generation limit (num_predict=150), the model exhausts its entire budget on thinking and returns a silent empty text response to the user. - The Solution (Safeguard): In the production route (
/api/v1/summarization/process), we implemented a safeguard:- The backend automatically detects empty/whitespace text returned from Qwen3.
- For non-streaming requests, it triggers an immediate retry with an expanded budget (
num_predict=1024). If it still returns empty, it fails loudly. - For streaming requests, it monitors the generated tokens and appends a loud error notification block in the stream if reasoning budget exhaustion is detected, ensuring the failure mode is never silent.
BriefAI uses a fully open-source, token-free diarization pipeline (SpeechBrain ECAPA-TDNN + Scikit-Learn AgglomerativeClustering) instead of gated models like Pyannote. While this allows out-of-the-box usage without HuggingFace authentication, it comes with strict, scientifically-understood accuracy tradeoffs:
Warning
Limitation 1: Overlapping Speech: The custom clustering approach assigns a single speaker label to each discrete Whisper segment. It cannot detect when two people talk at exactly the same time. If two speakers overlap, the segment is assigned to whichever speaker's voice fingerprint (embedding) is dominant.
Limitation 2: Straddling Segments (Quick Turn-Taking): Even without simultaneous overlap, rapid turn-taking can cause issues. If Whisper generates a single text segment that straddles the boundary between two speakers (e.g., contains the end of Speaker 1's sentence and the beginning of Speaker 2's sentence), the resulting audio chunk contains both voices. This produces a "mixed" embedding, which the clustering algorithm will often identify as a hallucinated third speaker.
Limitation 3: Short Interjections: The ECAPA-TDNN embedding model requires a sufficient length of audio (typically 1.5+ seconds) to generate a reliable voice fingerprint. Very short interjections ("yeah", "mhmm") may be misclassified because there isn't enough acoustic data to identify the speaker.
BriefAI uses a minimalist, local vector search approach (Nomic embeddings + exact cosine similarity in Python/SQLite) to provide "Ask BriefAI" chat functionality grounded strictly in your meeting history.
Warning
Grounding Threshold Tuning: The system enforces a strict similarity threshold to prevent the LLM from hallucinating answers when no relevant meeting chunks exist. This threshold was tuned via best-effort empirical testing on a limited set of queries. It may not perfectly handle every edge case (e.g., an unrelated query that happens to share rare vocabulary with the transcript might still surpass the threshold). Additionally, short transcripts naturally produce weaker, less separable embeddings than rich, multi-sentence transcripts, meaning the threshold's reliability scales with content length.
flowchart TB
subgraph Client [Client Environment]
UI[React + Vite UI]
end
subgraph Docker [Docker Compose Boundary]
subgraph Frontend [Nginx Container]
RP[Reverse Proxy & Static Serving]
end
subgraph Backend [FastAPI Container]
API[FastAPI Router<br/>JWT Auth]
Q[Concurrency Semaphores]
subgraph Services [Core Services]
TS[Transcription & Diarization<br/>faster-whisper + ECAPA-TDNN]
LS[LLM & RAG Engine<br/>Nomic Embeddings]
end
end
DB[(SQLite Database<br/>Alembic Migrations)]
end
subgraph Host [Host Machine]
OLLAMA[Ollama Engine<br/>Qwen3:1.7B / Llama 3.2:1B]
end
%% Connections
UI -- "REST (HTTP)" --> RP
UI -- "Live Audio (WebSocket)" --> RP
RP -- "Proxy pass" --> API
API --> Q
Q --> TS
Q --> LS
API -- "CRUD" --> DB
LS -- "REST API" --> OLLAMA
| Layer | Technology |
|---|---|
| Speech-to-Text | faster-whisper |
| Backend API | FastAPI |
| LLM Runtime | Ollama |
| LLM Models | Qwen3-1.7B, Llama 3.2-1B |
| Frontend | React + Vite |
| Python | 3.12 |
- Python 3.12 (via
pylauncher or Anaconda) - Ollama installed and running
- Node.js 18+ (for frontend)
- Git
git clone https://github.com/talhasaleemm/briefai.git
cd briefai# Using venv (recommended)
# venv directory is already initialized
.\.venv\Scripts\Activate.ps1
# Install backend dependencies
pip install -r backend/requirements.txtCopy-Item backend\.env.example backend\.env
# Edit backend\.env with your settings.\scripts\pull_models.ps1cd backend
uvicorn briefai.main:app --reload --host 0.0.0.0 --port 8000cd frontend
npm install
npm run devTo ensure high reliability on resource-constrained environments (especially CPU-fallback hardware), BriefAI implements backend concurrency semaphores to regulate system load:
- Whisper Concurrency Limit (
WHISPER_CONCURRENCY_LIMIT): Defaults to2concurrent audio transcription streams/uploads. Excess requests are queued and processed as slots become available. - Ollama Concurrency Limit (
OLLAMA_CONCURRENCY_LIMIT): Defaults to1concurrent LLM summarization/translation request on CPU-fallback host machines.
Important
Queued Execution vs. Parallel Processing: On CPU-only hardware, setting OLLAMA_CONCURRENCY_LIMIT=1 enforces strict request serialization rather than true parallel execution. While this introduces queuing delays under concurrent loads, it serves as a critical safeguard that prevents CPU cache thrashing, system slowdowns, and host out-of-memory (OOM) failures.
briefai/
├── backend/
│ ├── briefai/ # FastAPI application package
│ │ ├── main.py # App entrypoint, router includes, CORS, health check
│ │ ├── config.py # Pydantic-settings config (env-driven)
│ │ ├── constants.py # TaskType / ModelName enums
│ │ ├── routers/ # API route handlers (auth, transcription, summarization, chat, templates)
│ │ ├── services/ # Business logic (whisper, ollama, diarization)
│ │ ├── models/ # SQLAlchemy ORM models (users, transcripts, summaries, templates)
│ │ ├── schemas/ # Pydantic request/response models
│ │ ├── retrieval/ # RAG: chunking, embedding, cosine search (rag_service)
│ │ ├── prompts/ # LLM prompt templates
│ │ ├── utils/ # security (JWT), deps (get_current_user), limiter
│ │ └── internal/ # DB engine / session
│ ├── tests/ # Pytest test suite
│ └── requirements.txt
├── frontend/ # React + Vite app (Stage 6)
├── benchmarks/ # Latency/quality benchmarking (Stage 5)
├── scripts/
│ ├── pull_models.ps1 # Downloads Ollama models
│ └── setup.ps1 # Full environment setup
├── sample_audio/ # Test audio files (not committed)
├── .env.example
│ └── README.md
└── README.md
| Stage | Description | Status |
|---|---|---|
| 1 | Scaffold folder structure, config, deps | ✅ Complete |
| 2 | Transcription pipeline faster-whisper + WebSocket | ✅ Complete |
| 3 | Ollama integration Qwen3 + Llama wired in | ✅ Complete |
| 4 | Prompt templates summary, translate, actions, notes, decisions, terminology | ✅ Complete |
| 5 | Benchmarking module | ✅ Complete |
| 6 | Frontend React/Vite UI | ✅ Complete |
| 7 | User Authentication & Data Isolation | ✅ Complete |
| 8 | Speaker Diarization ECAPA-TDNN clustering | ✅ Complete |
| 9 | Ask BriefAI RAG grounded chat | ✅ Complete |
| 10 | Custom Template Builder | ✅ Complete |
| 11 | Docker Containerization | ✅ Complete |
This project is built stage-by-stage with explicit approval gates. See the Stage Report at the end of each stage for details on what was built and what comes next.
MIT