Skip to content

README.md

Latest commit

 

History

History
30 lines (20 loc) · 1.27 KB

README.md

File metadata and controls

30 lines (20 loc) · 1.27 KB
Raw

🖼️ Giving Eyes to Your AI: Engineering a Multimodal Agent

A hands-on workshop exploring multimodal AI agents with Haystack.

What You'll Build

  • 📄 Multimodal indexing pipeline (PDFs + images) using CLIP embeddings
  • 🤖 Vision-enabled agent powered by GPT-4o
  • 🔍 RAG tool for searching company policies
  • 💬 Conversational memory for context-aware interactions
  • 🔁 Human-in-the-loop controls for sensitive actions

Get Started

👉 See multimodal_agent_notebook.ipynb for the full interactive experience.

Deploy with Hayhooks

Want to deploy the agent as an API? Check out multimodal-agent/pipeline_wrapper.py — a Python script version of the notebook with Hayhooks integration pre-configured for serving the conversational agent.

Files

The files/ directory contains the sample data used in the workshop:

  • receipt.jpeg — A sample receipt image for the expense reimbursement demo
  • social_budget_policy.md — Company policy document for retrieval

Requirements

  • Python 3.10+
  • OpenAI API key (or your preferred LLM provider)
  • See the notebook for full package installation instructions