Skip to content

amirhf/imageSearch

BranchesTags

Repository files navigation

AI Feature Router for Image Metadata

Local‑first image captioning + Hybrid Search (Vector + Keyword) with a scalable Go microservice for high-throughput retrieval. Built to demonstrate AI integration, polyglot architecture, and production‑grade engineering.

Why this project

  • Scalable Search: Dedicated Go microservice for high-performance hybrid search (pgvector + FTS).
  • Smart Routing: Local vs cloud policy routing for captioning (confidence & SLO‑based).
  • Production Engineering: Shadow Mode deployment, Load Testing (Locust), and ADRs.
  • Observability: Prometheus, Grafana, and Jaeger tracing across Python and Go services.
  • Multi-tenant: Supabase authentication with user-owned private/public images.

Quick Start

Option 1: Run API Locally

# Start infrastructure (Postgres, Qdrant, etc.)
docker compose -f infra/docker-compose.yml up -d

# Run API with uvicorn
uvicorn apps.api.main:app --host 0.0.0.0 --port 8000
# open http://localhost:8000/docs

Option 2: Run API in Docker

# 1. Create .env.docker file (see Configuration section below)
cp .env.example .env.docker
# Edit .env.docker with your values

# 2. Build API image with embedder support
docker build -f apps/api/Dockerfile --build-arg INSTALL_EMBEDDER=true -t imagesearch-api:embedder .

# 3. Start infrastructure
cd infra && docker-compose up -d

# 4. Run API container
docker run -d --name api \
  --network infra_default \
  -p 8000:8000 \
  --env-file .env.docker \
  imagesearch-api:embedder

# open http://localhost:8000/docs

Run Frontend (Next.js UI)

The project includes a full-featured web UI in apps/ui/:

cd apps/ui
cp .env.local.example .env.local
# Edit .env.local with your Supabase credentials and API URL
npm install
npm run dev
# open http://localhost:3100

See the Frontend section below for detailed setup and features.

Storage Options

  • Local - Files stored on disk (default, zero setup)
  • MinIO - S3-compatible, runs in Docker (dev/self-hosted)
  • Cloudflare R2 - Zero egress fees, $1.50/month for 100GB (production)
  • AWS S3 - Industry standard with CloudFront CDN

See S3_STORAGE_SETUP.md for complete setup guide.

Architecture

  • FastAPI gateway: Handles writes, auth, and orchestrates requests.
  • Go Search Service: Dedicated microservice for high-performance read-only search.
  • Authentication: Supabase JWT-based auth with user profiles.
  • Multi-tenant: User-owned images with privacy controls (private/public).
  • Captioner: local BLIP first; cloud fallback (OpenAI/Gemini/Anthropic) via adapter.
  • Embedder: OpenCLIP (or SigLIP) image+text encoders → vectors.
  • Vector store: pgvector (HNSW) or Qdrant, selectable via env.
[Client/UI] → Next.js (Supabase Auth) → FastAPI (Gateway)
                                      → (caption: BLIP → cloud?)
                                      → (embed: OpenCLIP)
                                      → Go Search Service (Read Path)
                                      → Vector Store (pgvector/Qdrant)
                                      → Prometheus/Jaeger → Grafana

## Go Search Service (New)
To improve scalability and performance for read-heavy search traffic, we have introduced a dedicated **Go Search Service**.
- **Role**: Handles `POST /search` requests (hybrid vector + keyword search).
- **Stack**: Go 1.23, `pgx`, `pgvector`.
- **Performance**: Lower tail latency (P99) and better resource efficiency than the Python baseline.
- **Integration**: The Python API proxies search requests to this service when `SEARCH_BACKEND=go` is set.

See [Scaling Go Search Service](docs/scaling_go_search.md) for detailed benchmarks.

Endpoints

Public Endpoints

  • GET /search?q=... – semantic text→image search; query params: k, scope (all/mine/public)
  • GET /images – list images with filtering; query params: limit, offset, visibility
  • GET /images/{id} – metadata including download URLs (respects privacy)
  • GET /images/{id}/download – download original image (respects privacy)
  • GET /images/{id}/thumbnail – download 256x256 thumbnail (respects privacy)
  • GET /metrics – Prometheus exposition
  • GET /health – health check

Authenticated Endpoints (requires Bearer token)

  • POST /images – upload image (URL or file); returns {id, caption, download_url, thumbnail_url, ...}
  • PATCH /images/{id} – update image metadata (caption, visibility)
  • DELETE /images/{id} – soft delete image (owner only)

Configuration

Local Development (.env)

For running the API with uvicorn locally:

# Vector Store
VECTOR_BACKEND=pgvector
DATABASE_URL=postgresql+psycopg://postgres:postgres@localhost:5432/ai_router
QDRANT_URL=http://localhost:6333

# Supabase Authentication
SUPABASE_URL=https://your-project.supabase.co
SUPABASE_JWT_SECRET=your-jwt-secret-from-supabase-settings
SUPABASE_SERVICE_KEY=your-service-role-key
SUPABASE_ANON_KEY=your-anon-key
ADMIN_USER_ID=your-admin-user-uuid

# Cloud providers
CLOUD_PROVIDER=openrouter
OPENROUTER_API_KEY=sk-or-v1-...
OPENROUTER_MODEL=openai/gpt-4o-mini

# Routing policy
CAPTION_CONFIDENCE_THRESHOLD=0.55
CAPTION_LATENCY_BUDGET_MS=600

# Model Configuration
USE_MOCK_MODELS=false

# Image storage
IMAGE_STORAGE_BACKEND=local
IMAGE_STORAGE_PATH=./storage/images
BASE_URL=http://localhost:8000

Docker Deployment (.env.docker)

For running the API in Docker, create .env.docker with Docker network hostnames:

# Vector Store Configuration (Docker network hostnames)
VECTOR_BACKEND=pgvector
DATABASE_URL=postgresql+psycopg://postgres:postgres@infra-postgres-1:5432/ai_router
QDRANT_URL=http://infra-qdrant-1:6333

# Model Configuration
# Set to false to use real models (requires embedder image)
USE_MOCK_MODELS=false

# Cloud Provider Configuration
CLOUD_PROVIDER=openrouter
OPENROUTER_API_KEY=sk-or-v1-your-key-here
OPENROUTER_MODEL=openai/gpt-4o-mini

# Routing Policy
CAPTION_CONFIDENCE_THRESHOLD=0.55  # Use 0.99 to force cloud usage
CAPTION_LATENCY_BUDGET_MS=600

# Cloud Rate Limiting
CLOUD_MAX_REQUESTS_PER_MINUTE=60
CLOUD_MAX_REQUESTS_PER_DAY=10000
CLOUD_DAILY_BUDGET_USD=10.00

# Supabase Configuration (Multi-tenant Authentication)
# Get these from: https://supabase.com/dashboard/project/YOUR_PROJECT/settings/api
SUPABASE_URL=https://your-project.supabase.co
SUPABASE_JWT_SECRET=your-jwt-secret-from-api-settings
SUPABASE_SERVICE_KEY=eyJhbGc...your-service-role-key
SUPABASE_ANON_KEY=eyJhbGc...your-anon-key
ADMIN_USER_ID=your-admin-user-uuid

# Image Storage (Docker)
IMAGE_STORAGE_BACKEND=minio  # or local, s3
IMAGE_STORAGE_PATH=/app/storage/images
BASE_URL=http://localhost:8000

# MinIO Configuration (if using minio backend)
S3_BUCKET_NAME=imagesearch
S3_ENDPOINT_URL=http://infra-minio-1:9000
S3_ACCESS_KEY_ID=minioadmin
S3_SECRET_ACCESS_KEY=minioadmin
S3_REGION=us-east-1
S3_USE_PRESIGNED_URLS=true
S3_PRESIGNED_URL_EXPIRY=3600

Key differences for Docker:

  • Use Docker service names (e.g., infra-postgres-1 instead of localhost)
  • Set USE_MOCK_MODELS=false if you built with INSTALL_EMBEDDER=true
  • Use minio storage backend with Docker service name infra-minio-1
  • Get Supabase credentials from your project's API settings page

Multi-Tenant Setup

1. Create Supabase Project

  1. Go to supabase.com and create a new project
  2. Get your credentials from Settings → API:
    • Project URL
    • Anon/Public key
    • Service role key (keep secret!)
    • JWT Secret (Settings → API → JWT Settings)

2. Run Database Migrations

# Connect to your database and run migrations
psql $DATABASE_URL -f apps/api/storage/migrations/001_add_profiles.sql
psql $DATABASE_URL -f apps/api/storage/migrations/002_add_multi_tenant_fields.sql

3. Migrate Existing Data (Optional)

If you have existing images, migrate them to the admin user:

# Set environment variables
export DATABASE_URL="your-database-url"
export ADMIN_USER_ID="your-admin-uuid"

# Run migration
python scripts/migrate_to_multitenant.py --yes

4. Configure Environment

Update your .env file with Supabase credentials (see Config section above).

5. Configure Supabase

In your Supabase dashboard:

  • Authentication → URL Configuration: Add your frontend URL to redirect URLs
  • Authentication → Providers: Enable Email provider
  • Authentication → Email Templates: Customize if needed

See docs/multi_tenant/DEPLOYMENT_CHECKLIST.md for complete deployment guide.

Run tests

# Run all CI/CD safe tests
python tests/test_infrastructure.py
python tests/test_load.py
python tests/test_image_storage.py

# Multi-tenant E2E tests
pytest tests/test_e2e_multitenant.py -v

# Or use pytest for all tests
pytest tests/ -v

# See tests/README.md for more options

Dataset Seeding

Seed the database with real image datasets. Requires authentication since the /images endpoint is protected.

Get Authentication Token

  1. Log in to your frontend (http://localhost:3100)
  2. Open browser DevTools (F12) → Console
  3. Run: JSON.parse(localStorage.getItem('sb-YOUR_PROJECT_ID-auth-token')).access_token
  4. Copy the JWT token

Seed Images

# COCO validation set (automatic download)
python scripts/seed_datasets.py \
  --dataset coco \
  --count 100 \
  --auth-token "YOUR_JWT_TOKEN" \
  --visibility public

# Unsplash (requires API key)
python scripts/seed_datasets.py \
  --dataset unsplash \
  --count 50 \
  --api-key YOUR_UNSPLASH_KEY \
  --auth-token "YOUR_JWT_TOKEN" \
  --visibility public

# List available datasets
python scripts/seed_datasets.py --list

Note: Images are processed sequentially to avoid overwhelming cloud APIs. Expect ~10-15 seconds per image with cloud captioning.

Benchmarking

Run comprehensive performance benchmarks:

cd notebooks
python benchmark.py --test-image ../test_image.jpg --sample-size 100

Generates:

  • Latency statistics (mean, P95, P99)
  • Cost projections
  • Quality metrics (BLEU, METEOR)
  • Visualization plots

Observability

ADRs

Frontend (Next.js UI)

Path: apps/ui/

  • Stack: Next.js 16 (App Router), React 18, TypeScript, Tailwind CSS, TanStack Query, Supabase Auth
  • Features: User authentication, Search gallery, Upload (URL or file) with progress, Image library, Privacy controls, Explore public images
  • Default port: 3100

Prerequisites

  • Node.js 18+ (recommended 20+)
  • Backend API running (default http://localhost:8000)
  • Supabase project (for authentication)

Setup & Run (Dev)

cd apps/ui
cp .env.local.example .env.local
# Edit .env.local with your values:
# NEXT_PUBLIC_API_BASE=http://localhost:8000
# NEXT_PUBLIC_SUPABASE_URL=https://your-project.supabase.co
# NEXT_PUBLIC_SUPABASE_ANON_KEY=your-anon-key
npm install
npm run dev   # http://localhost:3100

Build & Start (Prod)

cd apps/ui
npm run build
npm run start   # http://localhost:3100

Important environment

  • NEXT_PUBLIC_API_BASE – Backend API URL (e.g., http://localhost:8000)
  • NEXT_PUBLIC_SUPABASE_URL – Your Supabase project URL
  • NEXT_PUBLIC_SUPABASE_ANON_KEY – Supabase anonymous/public key

UI routes

  • / – Search page (query by text, shows local/cloud origin badges)
  • /login – User login with Supabase
  • /signup – User registration
  • /upload – Upload images (authenticated); set privacy (private/public)
  • /library – User's uploaded images with privacy controls
  • /explore – Browse public images from all users
  • /image/[id] – Image detail (caption, origin, metadata, "Find similar")
  • /metrics – Metrics dashboard (p50/p95, local vs cloud split, cost, cache hits)

API proxy routes (UI → Backend)

All proxy routes automatically forward the Authorization header from Supabase:

  • GET /api/search → ${NEXT_PUBLIC_API_BASE}/search (with auth token)
  • GET /api/images → ${NEXT_PUBLIC_API_BASE}/images (with auth token)
  • GET /api/images/[id] → ${NEXT_PUBLIC_API_BASE}/images/{id} (with auth token)
  • POST /api/images → ${NEXT_PUBLIC_API_BASE}/images (authenticated, file or url)
  • PATCH /api/images/[id] → ${NEXT_PUBLIC_API_BASE}/images/{id} (authenticated)
  • DELETE /api/images/[id] → ${NEXT_PUBLIC_API_BASE}/images/{id} (authenticated)
  • GET /api/metrics/summary → parses ${NEXT_PUBLIC_API_BASE}/metrics into a compact JSON summary

Troubleshooting

  • Hydration warnings from extensions: we set suppressHydrationWarning on <body>.
  • Dev over LAN (192.168.x.x) may show an "allowedDevOrigins" warning; add to apps/ui/next.config.mjs if needed: 
    const nextConfig = {
  images: { unoptimized: true },
  allowedDevOrigins: ['http://192.168.70.10:3100']
}
export default nextConfig
  • Tailwind warning about @tailwindcss/line-clamp: remove the plugin from tailwind.config.ts (included by default in v3.3+).

Production Deployment (Hybrid Architecture)

We use a Hybrid Deployment strategy to balance scalability and cost:

  • API & Search Service: Google Cloud Run (Serverless, scales to zero).
  • Worker & Redis: Google Compute Engine (VM, cost-effective for always-on background tasks).

See docs/architecture.md for the detailed architecture diagram.

1. Prerequisites

  • Google Cloud Project with Billing enabled.
  • gcloud CLI installed and authenticated.
  • .env.production file with production secrets.

2. Deploy Worker VM (Redis + Ingestion Worker)

This script provisions a small VM (e2-small), installs Docker, and deploys Redis and the Ingestion Worker.

./scripts/deploy_worker.sh

Note: This script outputs the VM's public IP and Redis password. Ensure these are updated in your .env.production.

3. Deploy Cloud Run Services (API + Go Search)

This script builds and deploys the stateless services to Cloud Run, connecting them to the Worker VM's Redis.

./deploy.sh

4. Frontend (Vercel)

Deploy the Next.js frontend to Vercel:

cd apps/ui
vercel --prod

Ensure you set the NEXT_PUBLIC_API_BASE environment variable in Vercel to your Cloud Run API URL.

See docs/multi_tenant/DEPLOYMENT_CHECKLIST.md for a complete checklist.

License

MIT (sample) – replace or update as you prefer.

About

Image captioning and search using local models with fallback to cloud models

image-search-two-lovat.vercel.app

Resources

Readme
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

No releases published

Packages

 
 
 

Contributors

Languages