A real-time, semantic search engine that generates a dynamic visual collage based on streaming text input. As you type or speak, the system analyzes the context and retrieves relevant images using CLIP embeddings and a vector database, creating an evolving visual narrative.
The Dynamic Collage project introduces a powerful new paradigm in information retrieval and visualization. Unlike traditional search engines that require explicit, static queries (e.g., typing "dog" and hitting enter), this system operates on a continuous stream of thought.
As the user provides input—whether through a conversational interface, a speech-to-text stream, or by pasting a paragraph—the system actively "listens" and understands the evolving context. It dynamically constructs a visual representation (a collage) that mirrors the semantic content of the discussion in real-time.
This approach addresses the Semantic Gap by moving beyond keyword matching to deep semantic alignment, ensuring that the visual feedback loop is as fluid and expressive as the textual input.
Traditional search relies on keyword matching (lexical search). If you search for "canine", a keyword system might miss an image tagged "dog". Semantic Search solves this by understanding meaning. We use Vector Embeddings—numerical representations of data in a high-dimensional space—where similar concepts are located close together.
To bridge the gap between text and images, we utilize OpenAI's CLIP model. CLIP is trained on massive datasets to understand images and text in the same "embedding space". This allows us to convert a text description (e.g., "a calm forest at sunset") into a vector and find image vectors that are mathematically similar, even if they don't share the exact same metadata tags.
Standard retrieval is static: Query
-
Context: "I want to go on vacation..."
$\rightarrow$ Shows generic travel images. -
Refinement: "...to a tropical beach..."
$\rightarrow$ Updates to oceans and palm trees. -
Shift: "...but maybe the mountains are better."
$\rightarrow$ Smoothly transitions to snowy peaks and cabins.
The system follows a real-time processing pipeline designed for low latency and high relevance.
graph LR
User[User Input] -->|WebSocket Stream| API[FastAPI Backend]
API -->|Raw Text| Chunker[Semantic Chunker]
Chunker -->|Context Window| Encoder[CLIP Text Encoder]
Encoder -->|Query Vector| VectorDB[(ChromaDB)]
VectorDB -->|Top-K Image URLs| API
API -->|JSON Update| Frontend[React UI]
Frontend -->|Render| Collage[Dynamic Masonry Grid]
- Text Stream Ingestion: The Frontend sends keystrokes or text chunks via a persistent WebSocket connection to the Backend.
- Semantic Processing:
- Chunking: The text is broken down into meaningful semantic units (sentences or phrases).
- Context Management: A sliding window keeps track of the current topic to maintain coherence.
- Vector Encoding: The
SearchEngineuses the CLIP model to convert the current text context into a 512-dimensional embedding vector. - Similarity Search: This vector is queried against ChromaDB, which holds pre-computed embeddings for thousands of images. It uses HNSW (Hierarchical Navigable Small World) indexing for millisecond-level retrieval.
- Dynamic Presentation: The top matching images are sent back to the Frontend, which updates the Masonry Collage layout, smoothly animating new images in and old ones out.
- Real-Time Semantic Search: Instantly retrieves images matching the meaning of your text, not just keywords.
- Dynamic Collage: A masonry-style grid that updates live as the context changes.
- Multimodal Understanding: Uses OpenAI's CLIP model to bridge the gap between text and images.
- Vector Database: Powered by ChromaDB for efficient similarity search.
- Modern UI: A sleek, dark-themed React application for an immersive experience.
- Backend: Python, FastAPI, WebSockets, Sentence-Transformers (CLIP), ChromaDB.
- Frontend: React, Vite, CSS Modules.
- Python: 3.8 or higher
- Node.js: 18 or higher
- npm: Installed with Node.js
-
Clone the repository (if applicable) or navigate to the project root.
-
Backend Setup:
cd backend python3 -m venv venv source venv/bin/activate pip install -r requirements.txt
-
Frontend Setup:
cd frontend npm install -
Download Sample Data: The system needs images to search through. We provide a script to download sample images from Unsplash.
cd backend source venv/bin/activate python download_images.py python index_data.py
The easiest way to run the application is using the provided helper script:
./run_all.shThis script will:
- Start the FastAPI backend server on
http://localhost:8000. - Start the Vite frontend development server on
http://localhost:5173.
Open your browser to http://localhost:5173 to start using the application.
visual-search/
├── backend/ # Python FastAPI Backend
│ ├── chroma_db/ # Vector Database storage
│ ├── images/ # Downloaded image assets
│ ├── venv/ # Python virtual environment
│ ├── engine.py # Core search engine logic (CLIP + ChromaDB)
│ ├── main.py # FastAPI app and WebSocket endpoint
│ ├── processing.py # Text chunking and context management
│ ├── download_images.py # Script to fetch sample data
│ ├── index_data.py # Script to index images
│ └── requirements.txt # Python dependencies
├── frontend/ # React Frontend
│ ├── src/
│ │ ├── components/ # React components (Collage, TextInput)
│ │ ├── App.jsx # Main application layout
│ │ └── index.css # Global styles
│ └── package.json # Frontend dependencies
└── run_all.sh # Startup script