Vibe Search — Multimodal RAG Place Recommender

A place-recommendation system built for a datathon. It combines text and image understanding (RAG over reviews + CLIP-analyzed photos) to match natural-language "vibe" queries like "cafes to cowork from" or "romantic restaurants with dim lighting" against a dataset of NYC places.

Stack

• FastAPI + Uvicorn — web app / API (rag_app.py, run_rag.py)
• Sentence Transformers (MiniLM / BGE / MPNet) — text embeddings
• CLIP (open-clip-torch) — visual attribute extraction from place photos
• FAISS — vector similarity search
• Ollama (local LLM, default llama3) — query expansion & result explanations, via ollama_wrapper.py. Falls back gracefully if Ollama isn't running.

Data

Requires places.csv, media.csv, and reviews.csv in the project root (included in this repo). These are the datathon-provided place/review/media records the pipeline indexes.

Setup

git clone https://github.com/harshagarwalnyu/Vibe-Search-Corner.inc.git
cd Vibe-Search-Corner.inc
pip install -r requirements.txt

Optional: install and run Ollama locally for query expansion and explanations (ollama pull llama3). Without it, the app still runs using fallback logic.

Pipeline (first-time setup)

Run in order:

# 1. Analyze place images with CLIP, produce visual attributes
python place_image_analyzer.py --batch --places_csv "places.csv" --media_csv "media.csv"

# 2. Build image embeddings from the analysis output
python ingestion/rag_index_images.py --use_existing --analysis_path "place_clip_analysis_data.pkl"

# 3. Build text embeddings (incorporates visual attributes)
python ingestion/rag_index.py --model bge-large

# 4. Combine text + image embeddings into the final index
python ingestion/rag_index_combine.py --model bge-large --text_weight 0.7 --image_weight 0.3

# 5. Start the app
python run_rag.py --embedding-model bge-large

Then open http://localhost:8000

Quick run (after first-time setup)

python run_rag.py --embedding-model bge-large

CLI options

python run_rag.py --port 8080                       # custom port
python run_rag.py --embedding-model minilm           # different embedding model
python place_image_analyzer.py --report              # visual-attribute report
python place_image_analyzer.py --vibes_report         # vibe-category report

Embedding model options

Model	Dimensions	Speed	Quality
MiniLM	384	fastest	good
BGE-Small	384	fast	better
BGE-Base	768	medium	very good
BGE-Large	1024	slowest	best
MPNet	768	medium	very good

Notes

• This was built quickly for a datathon — the pipeline is a linear script sequence, not a package, and has no automated tests.
• .env is git-ignored; if you need Ollama config overrides, create your own local .env (see ollama_wrapper.py for supported settings).

License

MIT — see LICENSE.