Search a photo collection by typing a description or by handing it an example image — same index, same vector space, powered by CLIP + Qdrant.
Most "vector search" portfolio projects are text-only RAG. This one uses CLIP's defining property — that an image and a text description of it are trained to land close together in the same embedding space — to support two query modes against one Qdrant collection: text → image and image → image. A small FastAPI app makes both interactively testable, and a text-to-image retrieval benchmark (Recall@k, MRR) turns "it works" into a measured number.
- True cross-modal retrieval — one CLIP model, one vector space, one Qdrant collection; the query modality (text or image) is decided per request, not per index.
- Retrieval evaluation, not just a demo — a labelled benchmark dataset drives Recall@1/5/10 and MRR for text→image search, the standard metrics in the CLIP/cross-modal-retrieval literature.
- A usable interface — FastAPI backend + a small vanilla-JS frontend (text box and drag-and-drop image upload), not just a Jupyter notebook.
- Clean dependency isolation — the search service depends on embedder
and store interfaces (
typing.Protocol), so its query-routing logic is unit-tested with fakes, without ever loading CLIP or running Qdrant. - Offline-reproducible by default — a synthetic image+caption generator (colored shapes, PIL) avoids licensing/network concerns for anyone cloning the repo, while staying a drop-in replacement for a real photo folder.
┌─────────────────────────────┐
│ data/images/*.jpg,.png │
│ (demo: synthetic shapes, │
│ or your own photos) │
└──────────────┬───────────────┘
▼
Ingestion (thumbnails)
▼
┌─────────────────────────────┐
│ ClipEmbedder.embed_images │
│ (CLIP ViT-B/32) │
└──────────────┬───────────────┘
▼
Qdrant collection "images"
(cosine similarity, one vector
space for text AND images)
▲
┌───────────────────┴───────────────────┐
│ │
ClipEmbedder.embed_texts ClipEmbedder.embed_image
"a red bicycle" example.jpg
│ │
▼ ▼
search_by_text() search_by_image()
│ │
└───────────────────┬───────────────────┘
▼
FastAPI (/api/search/*) ──► static/index.html
│
▼
Benchmark: captions.json -> Recall@k, MRR
results/detector... (per_query.json + printed summary)
Requirements: Python 3.10+, Docker. No API keys — CLIP and Qdrant both run fully locally (CPU is fine for the demo dataset size).
# 1. Install
pip install -r requirements.txt && pip install -e .
# 2. Unit tests (run without Qdrant or CLIP)
pytest -v
# 3. Start Qdrant
docker compose up -d # dashboard: http://localhost:6333/dashboard
# 4. Generate the synthetic demo dataset (36 labelled images)
python scripts/generate_demo_dataset.py 36 42
# 5. Embed + index every image
python scripts/ingest_images.py
# 6. Launch the web app
uvicorn imagesearch.app:app --reload --app-dir src
# -> open http://localhost:8000, type a query or upload an image
# 7. Run the text -> image retrieval benchmark
python scripts/run_benchmark.pyAd-hoc search from the command line:
python scripts/search_cli.py text "a red circle on a white background"
python scripts/search_cli.py image data/images/demo-000.jpgrun_benchmark.py uses each image's ground-truth caption as a text query
and records the rank of the correct image in the results:
Text -> image retrieval benchmark:
recall@1: ...
recall@5: ...
recall@10: ...
mrr: ...
(Run it to get real numbers.) On the synthetic shapes dataset this mostly demonstrates that the pipeline is correct end-to-end; CLIP's retrieval quality becomes genuinely interesting on real photos with rich visual content — see data/README.md for swapping in your own captioned image folder.
sentence-transformers already ships a CLIP checkpoint (clip-ViT-B-32)
with the exact same .encode() API used for text embeddings elsewhere —
one dependency, one mental model, and it keeps this project's stack
consistent with the other two in this series (RAG Evaluation Lab, Log
Anomaly Lab).
multimodal-image-search/
├── src/imagesearch/
│ ├── config.py # env-based settings
│ ├── models.py # ImageRecord, SearchResult, captions I/O
│ ├── pipeline.py # factory: settings -> service
│ ├── app.py # FastAPI backend
│ ├── embedding/clip_embedder.py # CLIP wrapper (text + image encoding)
│ ├── storage/vector_store.py # Qdrant: one collection, both modalities
│ ├── ingestion/loader.py # scan images dir, generate thumbnails
│ ├── search/service.py # query routing (Protocol-based, unit-testable)
│ ├── evaluation/
│ │ ├── metrics.py # Recall@k, MRR
│ │ └── runner.py # text->image benchmark orchestration
│ └── simulation/dataset_generator.py # synthetic demo images + captions
├── static/index.html # search UI (text box + image upload)
├── scripts/ # generate_demo_dataset / ingest / search_cli / run_benchmark
├── tests/ # unit tests (no CLIP or Qdrant needed)
├── data/README.md # demo vs. bring-your-own-photos
├── docker-compose.yml # local Qdrant
└── docs/DESIGN_DECISIONS.md # rationale for the non-obvious choices
See docs/DESIGN_DECISIONS.md — why a synthetic
dataset instead of bundled/downloaded photos, why the store doesn't care
which modality produced a query vector, self-exclusion in image→image
search, and the Protocol-based service design.
- Synthetic demo data — swap in a real, licensed, captioned dataset (e.g. a small Unsplash Lite or COCO subset) for a meaningful benchmark; only the loader/captions file changes.
- No re-ranking stage — a cross-encoder-style joint reranker for text-image pairs (e.g. BLIP ITM) would be a natural precision-boosting addition on top of CLIP's bi-encoder retrieval, mirroring the re-ranker in the RAG Evaluation Lab project.
- No batching UI feedback — the app embeds one query at a time; a
drag-and-drop of multiple images (find images similar to any of these)
is a small
search_by_imageextension. - CPU only — fine for a few thousand images; a GPU would matter for large-scale ingestion.
MIT