📚 Document Embedding Service

A comprehensive OCR and document processing service that extracts text from various document formats, generates embeddings, and stores them in a vector database for AI-powered applications.

🌟 Features

📄 Document Processing

Multi-format support: PDF (text + scanned), images (PNG, JPEG), text files
Advanced OCR: Tesseract OCR with Vietnamese and English language support
Smart text extraction: Handles both digital and scanned documents
Textbook processing: Enhanced metadata parsing for educational materials

🤖 AI-Powered Embeddings

OpenAI Integration: Uses text-embedding-3-small (1536 dimensions)
Intelligent chunking: Configurable chunk sizes with overlap
Batch processing: Efficient handling of multiple documents
Semantic search ready: Optimized for RAG applications

🗄️ Vector Database

Qdrant Integration: Both Cloud and self-hosted options
Scalable storage: Handles large document collections
Fast retrieval: Optimized for similarity search
Metadata filtering: Rich metadata for context-aware search

🚀 Production Ready

FastAPI: Modern, fast web API framework
Docker Support: Complete containerization with docker-compose
Health Monitoring: Built-in health checks and monitoring
Configurable CORS: Flexible cross-origin resource sharing

🏗️ Architecture

┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│   File Upload   │───▶│  OCR & Extract  │───▶│   Text Chunks   │
└─────────────────┘    └─────────────────┘    └─────────────────┘
                                                        │
                                                        ▼
┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│ Vector Database │◀───│   Embeddings    │◀───│ OpenAI API      │
│    (Qdrant)     │    │  (1536 dims)    │    │                 │
└─────────────────┘    └─────────────────┘    └─────────────────┘

🚀 Quick Start

Option 1: Docker (Recommended)

Clone and setup:

git clone <repository-url>
cd ocr
cp .env.example .env

Configure environment:

# Edit .env with your credentials
OPENAI_API_KEY=sk-your-openai-key-here
QDRANT_URL=https://your-cluster.region.gcp.cloud.qdrant.io:6333
QDRANT_API_KEY=your-qdrant-key

Start service:

# Development mode
docker-compose up --build

# Production mode
docker-compose -f docker-compose.prod.yml up --build -d

Access API:
- API: http://localhost:8000
- Documentation: http://localhost:8000/docs
- Health: http://localhost:8000/health

Option 2: Local Development

Prerequisites:

# Install Tesseract OCR
# Windows: Download from https://github.com/UB-Mannheim/tesseract/wiki
# Ubuntu: sudo apt-get install tesseract-ocr tesseract-ocr-vie
# macOS: brew install tesseract

Setup Python environment:

python -m venv .venv
source .venv/bin/activate  # On Windows: .venv\Scripts\activate
pip install -r requirements.txt

Configure environment:

cp .env.example .env
# Edit .env with your API keys

Run the service:

uvicorn main:app --host 0.0.0.0 --port 8000 --reload

📚 API Documentation

Core Endpoints

1. Extract Text Only

POST /extract
Content-Type: multipart/form-data

file: <document file>

Response:

{
  "text": "Extracted text content...",
  "filename": "document.pdf",
  "file_type": "application/pdf",
  "char_count": 1234,
  "status": "success"
}

2. Process & Store Document

POST /upload-document
Content-Type: multipart/form-data

file: <document file>
metadata: <optional JSON string>

Response:

{
  "document_id": "doc_abc123",
  "filename": "document.pdf",
  "chunks_count": 15,
  "status": "success",
  "message": "Document processed and indexed successfully"
}

3. Process Textbook (Enhanced Metadata)

POST /upload-textbook
Content-Type: multipart/form-data

file: <textbook file>
book_name: "Sách Toán"
publisher: "Cánh Diều"
grade: "Lớp 3"                    # Optional
product_name: "Custom name"       # Optional

Response:

{
  "document_id": "doc_textbook_123",
  "filename": "sach_toan.pdf",
  "chunks_count": 25,
  "status": "success",
  "message": "Textbook processed successfully. Book: Sách Toán - Cánh Diều - Lớp 3"
}

4. Batch Upload

POST /upload-batch
Content-Type: multipart/form-data

files: <multiple document files>

5. Health Check

GET /health

Response:

{
  "status": "healthy",
  "qdrant_connected": true
}

Management Endpoints

DELETE /document/{document_id} - Delete document
GET /document/{document_id}/metadata - Get document metadata
GET /collections/{collection_name}/info - Get collection info

⚙️ Configuration

Environment Variables

Variable	Description	Default	Required
`OPENAI_API_KEY`	OpenAI API key	-	✅
`QDRANT_URL`	Qdrant Cloud URL	-	If using cloud
`QDRANT_API_KEY`	Qdrant API key	-	If using cloud
`QDRANT_HOST`	Qdrant host	localhost	If local
`CHUNK_SIZE`	Text chunk size	800	❌
`CHUNK_OVERLAP`	Chunk overlap	150	❌
`MAX_FILE_SIZE_MB`	Max upload size	100	❌
`COLLECTION_NAME`	Qdrant collection	cs_chatbot_docs	❌
`ALLOWED_ORIGINS`	CORS origins	*	❌

Supported File Types

PDF: application/pdf
Images: image/png, image/jpeg, image/jpg
Text: text/plain, text/markdown
Code: text/x-python
JSON: application/json

🐳 Docker Usage

Using Makefile (Recommended)

# Setup
make setup          # Copy .env.example to .env
make dev            # Start development
make prod           # Start production
make local          # Start with local Qdrant

# Monitoring
make logs           # View logs
make health         # Check health
make status         # Container status

# Maintenance
make clean          # Clean everything
make restart        # Restart services

Manual Docker Commands

# Development
docker-compose up --build

# Production
docker-compose -f docker-compose.prod.yml up --build -d

# With local Qdrant
docker-compose --profile local-qdrant up --build

🔧 Development

Project Structure

ocr/
├── main.py                 # FastAPI application
├── config/
│   └── config.py          # Configuration settings
├── services/
│   ├── embedding_service.py    # Document processing
│   ├── document_processor.py   # Text extraction
│   ├── embedder.py            # OpenAI integration
│   └── qdrant_service.py      # Vector database
├── model/
│   └── schemas.py         # Pydantic models
├── requirements.txt       # Dependencies
├── Dockerfile            # Container definition
├── docker-compose.yml    # Development setup
└── docker-compose.prod.yml # Production setup

Adding New Features

New file type support:
- Add MIME type to SUPPORTED_FILE_TYPES in config
- Implement extraction logic in document_processor.py
Custom embeddings:
- Extend embedder.py with new provider
- Update embedding_service.py initialization
Enhanced metadata:
- Modify schemas in model/schemas.py
- Update processing logic in services

Testing

# Run tests locally
pytest

# Run tests in Docker
docker-compose exec ocr-embedding-service python -m pytest

# API testing
curl -X GET http://localhost:8000/health
curl -X GET http://localhost:8000/docs

🌐 Production Deployment

1. Environment Setup

# Production .env
OPENAI_API_KEY=sk-prod-key-here
QDRANT_URL=https://prod-cluster.region.gcp.cloud.qdrant.io:6333
QDRANT_API_KEY=prod-qdrant-key
ALLOWED_ORIGINS=https://your-app.com,https://admin.your-app.com
DEBUG=false
MAX_FILE_SIZE_MB=50

2. Docker Deployment

# Production deployment
docker-compose -f docker-compose.prod.yml up --build -d

# With scaling
docker-compose -f docker-compose.prod.yml up --scale ocr-embedding-service=3 -d

3. Reverse Proxy (Nginx)

# nginx.conf
upstream ocr_backend {
    server localhost:8000;
}

server {
    listen 80;
    server_name your-api.com;
    
    location / {
        proxy_pass http://ocr_backend;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
    }
}

📈 Monitoring

Health Checks

# Service health
curl http://localhost:8000/health

# Container health  
docker inspect --format "{{.State.Health.Status}}" ocr-embedding-api

Logs

# Application logs
docker-compose logs -f ocr-embedding-service

# Specific timeframe
docker-compose logs --since 1h ocr-embedding-service

Resource Monitoring

# Container stats
docker stats

# Using Makefile
make stats

🤝 Contributing

Fork the repository
Create feature branch: git checkout -b feature/amazing-feature
Commit changes: git commit -m 'Add amazing feature'
Push to branch: git push origin feature/amazing-feature
Open Pull Request

Code Standards

Follow PEP 8 for Python code
Use type hints
Add docstrings for functions
Write tests for new features

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

🆘 Support

Common Issues

Tesseract not found:

# Install Tesseract OCR
# Make sure it's in system PATH
tesseract --version

OpenAI API errors:
- Check API key validity
- Verify account credits
- Check rate limits
Qdrant connection issues:
- Verify URL and API key
- Check network connectivity
- Ensure collection exists

Docker issues:

# Reset everything
docker-compose down -v
docker system prune -f
docker-compose up --build

Getting Help

📧 Email: your-email@domain.com
💬 Issues: Create GitHub issue
📚 Documentation: /docs endpoint
🐳 Docker Help: See DOCKER_README.md

🎯 Roadmap

Support for more file formats (DOCX, PPTX)
Multiple embedding providers
Advanced text preprocessing
Batch processing optimization
Web dashboard for document management
Kubernetes deployment
API authentication and authorization

Made with ❤️ for document processing and AI applications

Name		Name	Last commit message	Last commit date
Latest commit History 19 Commits
config		config
model		model
services		services
.dockerignore		.dockerignore
.env.example		.env.example
.gitignore		.gitignore
DOCKER_README.md		DOCKER_README.md
Dockerfile		Dockerfile
Makefile		Makefile
README.md		README.md
base.py		base.py
docker-compose.override.yml		docker-compose.override.yml
docker-compose.prod.yml		docker-compose.prod.yml
docker-compose.yml		docker-compose.yml
main.py		main.py
requirements.txt		requirements.txt

Folders and files

Latest commit

History

Repository files navigation

📚 Document Embedding Service

🌟 Features

📄 Document Processing

🤖 AI-Powered Embeddings

🗄️ Vector Database

🚀 Production Ready

🏗️ Architecture

🚀 Quick Start

Option 1: Docker (Recommended)

Option 2: Local Development

📚 API Documentation

Core Endpoints

1. Extract Text Only

2. Process & Store Document

3. Process Textbook (Enhanced Metadata)

4. Batch Upload

5. Health Check

Management Endpoints

⚙️ Configuration

Environment Variables

Supported File Types

🐳 Docker Usage

Using Makefile (Recommended)

Manual Docker Commands

🔧 Development

Project Structure

Adding New Features

Testing

🌐 Production Deployment

1. Environment Setup

2. Docker Deployment

3. Reverse Proxy (Nginx)

📈 Monitoring

Health Checks

Logs

Resource Monitoring

🤝 Contributing

Code Standards

📄 License

🆘 Support

Common Issues

Getting Help

🎯 Roadmap

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages