This repository contains research experiments for building a Multimodal Question Answering (QA) system capable of understanding information from documents, tables, and images.
The goal of this project is to explore different components required for a multimodal QA pipeline rather than delivering a finalized production system. Each collaborator worked on a different module of the system.
The objective of this research is to investigate how information from multiple modalities can be combined to answer questions from research documents.
The system explores three main capabilities:
- Extracting text and layout information from document images
- Retrieving knowledge from research papers using Retrieval-Augmented Generation (RAG)
- Understanding visual information from images
These components together form the foundation for a Multimodal Question Answering pipeline.
This repository is a collaborative research effort.
Explores text extraction and document structure analysis.
Technologies explored:
- PaddleOCR
- PaddleOCR-VL
- Camelot (table extraction)
Focus areas:
- OCR text detection
- Table structure reconstruction
- Document layout understanding
Branch: ocr-experiments
A Retrieval-Augmented Generation system designed to analyze research papers.
Key capabilities:
- Multi-PDF document processing
- Semantic search using vector embeddings
- Interactive CLI for asking questions about papers
- Fast inference using Groq models
- Structured extraction using GROBID
Technologies used:
- FAISS vector database
- HuggingFace embeddings
- GROBID PDF parser
- Groq LLM inference
Branch: Rag-System
This module explores visual feature extraction techniques for integrating image understanding into the multimodal QA pipeline.
The current implementation includes two experimental approaches:
- Uses OpenAI CLIP (ViT-B/32) model
- Generates normalized image embeddings
- Generates normalized text embeddings
- Computes cosine similarity between image and text representations
File:
image_embedding_hugging_face.py
- Uses YOLOv8 backbone for extracting deep visual features
- Applies Global Average Pooling to obtain compact image embeddings
- Uses SentenceTransformer for generating text embeddings
- Projects both modalities into a shared embedding space
- Computes similarity scores between projected embeddings
File:
image_embedding_yolo.py
Branch:
feature/image-embedding-module
This module contributes toward building a unified multimodal representation space for document image understanding.
Document / Research Paper / Image -> Information Extraction
OCR Pipeline: Text + Table + Layout Extraction
RAG Pipeline: Semantic Retrieval + LLM Reasoning
Image Analysis: Visual Understanding
multimodal-question-answering
│
├── ocr-experiments/ # OCR research using PaddleOCR & Camelot
├── rag-system/ # Research paper RAG assistant
├── image-module/ # Image understanding experiments (coming soon)
├── README.md
This repository represents research exploration and experimentation toward a multimodal QA system.
Implemented components include:
- OCR-based document text extraction
- Table detection and filtering
- Research paper question answering using RAG
Additional work on image understanding will be integrated in future updates.
Python PaddleOCR Camelot FAISS HuggingFace Embeddings GROBID Groq LLM Inference Docker
- Integrate OCR results into the RAG pipeline
- Combine text, tables, and image understanding
- Build an end-to-end multimodal QA system
- Evaluate performance across research documents
This project was created to experiment with modern document intelligence and multimodal AI techniques.
It serves as:
- a research exploration
- a learning project
- a demonstration of applied AI components for recruiters and collaborators.
| Name | Contribution |
|---|---|
| Koushiki Chakraborty | OCR experiments (PaddleOCR, PaddleOCR-VL, Camelot) |
| Ayan Mazumdar | Research Paper RAG Assistant |
| Bristi Kundu | Image understanding experiments |
This project is open-source and available under the MIT License.