Local Gallery Semantic Search

Local semantic photo search engine using multimodal embeddings, Qdrant vector search, Flask, and ONNX Runtime inference optimization.

Repository description

Local semantic photo search with JinaCLIP, Qdrant, Flask and ONNX Runtime, optimized for low-latency text-query inference on Apple Silicon.

Overview

This project lets users search a local photo gallery using natural language.

Example queries:

dog on a walk 
sunset in Fortaleza
steak in New York
family photo with grandpa
city at night
beach

The system converts images and text into the same embedding space. Image vectors are stored in Qdrant, and text queries are embedded at search time to retrieve the most similar photos.

The main engineering focus of this project is not model training. The focus is local inference optimization and retrieval system design.

Motivation

Most photo search systems depend on cloud services, filename metadata, or manual tags. With the release of Siri AI on iOS 27, I was wondering how Apple would optimize inference time to search images locally in the user's gallery.

This project explores a local-first alternative:

local photos
→ image embeddings
→ local Qdrant vector database
→ local text-query embedding
→ semantic search results

The goal is to build a fast local semantic search system and benchmark the inference path.

Tech stack

• Python
• Flask
• JinaCLIP v2
• SentenceTransformers
• ONNX Runtime
• Qdrant
• Hugging Face Transformers
• Apple Silicon / macOS with M5

Architecture

Photo cache
   ↓
Image encoder
   ↓
Image embeddings
   ↓
Qdrant vector database

User text query
   ↓
Text encoder
   ↓
Query embedding
   ↓
Qdrant similarity search
   ↓
Top-k matching photos
   ↓
Flask web UI

Current features

• Local web app for natural-language photo search
• Multimodal image/text embeddings using JinaCLIP v2
• Qdrant vector database for image retrieval
• SentenceTransformer baseline backend
• ONNX Runtime CPU backend
• Query-time latency measurement
• Qdrant search latency measurement
• Benchmark script for text encoder latency
• CoreMLExecutionProvider investigation on Apple Silicon

Backends tested

Backend	Status	Notes
SentenceTransformer	Working	Baseline text-query encoder
ONNX Runtime CPU, FP32	Working	Best backend so far
ONNX Runtime CPU, FP32 + ORT_ENABLE_ALL	Working	Similar p50, slightly worse tail latency
ONNX Runtime CPU, FP16	Working	Faster than baseline, but worse than FP32 in this setup
ONNX Runtime CoreMLExecutionProvider	Failed	Dynamic/unbounded shape issues
MLX	Not implemented	Future work
TensorRT	Not applicable on Mac	Planned for NVIDIA RTX backend

Main benchmark result

Text-query embedding latency was reduced from approximately:

214.29 ms p50 → 16.69 ms p50

This is approximately:

12.8x median speedup

using ONNX Runtime CPU with the FP32 ONNX export.

Benchmark results

Benchmarks were run locally on macOS using an Apple Silicon MacBook. I uploaded all my 7789 photos into the vector storage to see i large sacle how would ONNX Runtime perfom.

The benchmark measures text-query embedding latency only. It does not include image indexing.

SentenceTransformer baseline vs ONNX Runtime CPU

Backend	Mean	p50	p95	p99
SentenceTransformer	217.43 ms	214.29 ms	227.23 ms	277.71 ms
ONNX Runtime CPU, FP32	16.80 ms	16.69 ms	18.33 ms	19.33 ms

ONNX Runtime CPU with ORT_ENABLE_ALL

Backend	Mean	p50	p95	p99
SentenceTransformer	224.14 ms	214.29 ms	277.41 ms	375.22 ms
ONNX Runtime CPU, FP32 + ORT_ENABLE_ALL	16.61 ms	16.18 ms	19.48 ms	25.11 ms

Speedup

Using the more stable ONNX Runtime CPU FP32 result:

Metric	Speedup
Mean	~12.9x
p50	~12.8x
p95	~12.4x
p99	~14.4x

Interpretation

The main bottleneck was the text encoder, not Qdrant.

Qdrant search was consistently much smaller than text embedding latency, so the most effective optimization was replacing the SentenceTransformer query path with ONNX Runtime.

CoreMLExecutionProvider investigation

I also tested ONNX Runtime with CoreMLExecutionProvider to evaluate Apple Silicon acceleration.

This did not work reliably with the exported jina-clip-v2 ONNX graph.

The errors were related to:

dynamic shapes
zero-sized dimensions
unbounded dimensions
CoreML subgraph execution failure

Current conclusion:

CoreMLExecutionProvider is not a drop-in optimization path for this exported multimodal ONNX model.

A better future direction would be exporting a text-only ONNX/CoreML encoder with fixed input shapes.

ONNX model files

The best result so far used the FP32 ONNX export:

models/jina-clip-v2/onnx/model.onnx
models/jina-clip-v2/onnx/model.onnx_data

Important:

model.onnx is the model file loaded by ONNX Runtime.
model.onnx_data contains the external weights.
Do not load model.onnx_data directly.

Why FP32 was faster than FP16 here

The initial assumption was that model_fp16.onnx would be faster because it is smaller.

In practice, on this Mac and with this ONNX Runtime setup, the FP32 export with external weights was much faster for text-query inference.

This is why the project benchmarks multiple backends instead of assuming the smallest model file is the fastest.

Running locally

Install dependencies:

pip install -r requirements.txt

Download the ONNX model:

huggingface-cli download jinaai/jina-clip-v2 \
  --include "onnx/model.onnx" \
  --include "onnx/model.onnx_data" \
  --local-dir models/jina-clip-v2

Start the Flask app:

python -m src.flask_app

Open:

http://127.0.0.1:5000

Running benchmarks

Run the text encoder benchmark:

python -m benchmarks.benchmark_text_encoder

The benchmark reports:

mean
p50
p95
p99
min
max

Current limitations

• The current ONNX query path uses the exported multimodal ONNX model in text-only mode.
• CoreMLExecutionProvider does not currently work with this ONNX graph.
• Image vectors should be regenerated if the embedding backend changes.
• The UI is intentionally simple and focused on the search pipeline.

Future work

• Add retrieval quality evaluation with Recall@1, Recall@5 and Recall@10
• Add top-k overlap comparison between SentenceTransformer and ONNX results
• Test ONNX INT8 and quantized exports
• Export a text-only ONNX encoder with fixed input shapes
• Re-test CoreMLExecutionProvider with static-shape ONNX
• Convert a text-only encoder to Core ML directly
• Add a FastAPI backend
• Improve frontend UI
• Add NVIDIA RTX backend with ONNX Runtime CUDA
• Add TensorRT backend for NVIDIA GPU inference

TensorRT plan

TensorRT is not available on Apple Silicon. It is planned as a second phase on an NVIDIA RTX machine.

Planned NVIDIA backend comparison:

SentenceTransformer GPU
ONNX Runtime CUDA
ONNX Runtime TensorRT
TensorRT FP16 engine

Target metrics:

embedding latency
throughput
VRAM usage
model loading time
p50 / p95 / p99 latency
retrieval quality

Key lessons

• The vector database was not the bottleneck.
• Query-time text embedding dominated latency.
• ONNX Runtime CPU significantly outperformed the SentenceTransformer baseline.
• Smaller model files are not always faster.
• CoreMLExecutionProvider is sensitive to dynamic and unbounded shapes.
• Benchmarking p50, p95 and p99 is more useful than reporting a single inference time.

Resume summary

Built a local semantic photo search engine using JinaCLIP, Qdrant, Flask and ONNX Runtime. Optimized text-query embedding latency from ~214 ms p50 with SentenceTransformer to ~16.7 ms p50 with ONNX Runtime CPU, achieving ~12.8x median speedup while investigating CoreMLExecutionProvider limitations with dynamic ONNX shapes on Apple Silicon.

Backend	Mean	p50	p95	p99	Status
SentenceTransformer	217 ms	214 ms	227 ms	278 ms	baseline
ONNX FP16 CPU	92 ms	79 ms	154 ms	310 ms	faster, unstable
ONNX FP32 CPU	16.8 ms	16.7 ms	18.3 ms	19.3 ms	best
ONNX FP32 CPU + ORT_ALL	16.6 ms	16.2 ms	19.5 ms	25.1 ms	fastest median, worse tail
ONNX CoreML EP	failed	-	-	-	dynamic/unbounded shapes