AI Document Processing & Analytics Pipeline

An end-to-end automated pipeline for ingesting, classifying, chunking, and retrieving 1,200+ synthetic financial documents with quantitative evaluation reporting and full SQL analytics tracking.

Built as a production-style ML data pipeline demonstrating document intelligence at scale — from raw text ingestion through TF-IDF classification, hybrid semantic retrieval (FAISS + BM25), and quantitative evaluation with Precision@K, MRR, NDCG, and per-class F1 metrics. Every stage is tracked in a SQLite database with analytical SQL queries for performance reporting.

Architecture

PDF/Text Input ──► Document Loader ──► Text Processor (Chunking)
                                            │
                        ┌───────────────────┤
                        ▼                   ▼
                 TF-IDF Classifier    Embedding Engine
                   (sklearn)        (sentence-transformers)
                        │                   │
                        ▼                   ▼
                  SQLite Database     FAISS + BM25 Index
                  (classifications,   (hybrid retrieval)
                   analytics)               │
                        │                   ▼
                        └──────► Evaluation Framework
                                 (Precision@K, MRR, NDCG,
                                  confusion matrix, latency)
                                        │
                                        ▼
                                 SQL Analytics Report
                                 + JSON Evaluation Report

Key Results

Metric	Value
Documents Processed	1,200
Document Types	6 (invoice, bank statement, loan agreement, pay stub, tax form, insurance policy)
Classification Accuracy	100% (TF-IDF + Logistic Regression, 80/20 train/test split)
Classification F1 (macro)	1.0
Avg Classification Confidence	94.3%
Retrieval MRR	0.772
Retrieval NDCG	0.793
Retrieval Precision@1	0.743
Retrieval Precision@5	0.703
Avg Query Latency	80ms (median 14ms, P99 474ms)
Text Chunks Produced	6,364
Processing Throughput	7,400+ docs/sec (chunking)
Total Pipeline Time	35.2s (full), 1.8s (skip-retrieval)
Unit Tests	23/23 passing

Project Structure

ai-doc-processing-suite/
├── run_pipeline.py              # Main entry point — runs end-to-end
├── generate_documents.py        # Generates 1,200 synthetic financial documents
├── config.yaml                  # Pipeline configuration
├── requirements.txt             # Python dependencies
│
├── pipeline/                    # Reusable pipeline components
│   ├── document_loader.py       # Loads documents from directory structure
│   ├── text_processor.py        # Recursive text chunking with overlap
│   ├── classifier.py            # TF-IDF + Logistic Regression classifier
│   ├── retriever.py             # Hybrid FAISS (dense) + BM25 (sparse) retrieval
│   └── evaluator.py             # Precision@K, MRR, NDCG, latency benchmarking
│
├── sql/                         # SQL layer
│   ├── schema.sql               # Database schema (6 tables, 5 indexes)
│   ├── analytics.sql            # 10 analytical SQL queries for reporting
│   └── db_manager.py            # Python SQLite wrapper with batch operations
│
├── tests/                       # Unit tests
│   ├── test_pipeline.py         # Tests for all pipeline components
│   └── test_sql.py              # Tests for database operations
│
├── notebooks/                   # Exploratory notebooks (OCR, RAG, model experiments)
│   ├── Analyze_a_Scanned_PDF.ipynb
│   ├── Full_RAG_Pipeline.ipynb
│   ├── Document_Classification_Before_Retrieval.ipynb
│   └── ...
│
├── data/                        # Generated at runtime
│   ├── generated/               # 1,200 synthetic documents (6 subdirectories)
│   ├── manifest.json            # Document metadata manifest
│   ├── ground_truth.json        # Evaluation ground truth
│   └── pipeline.db              # SQLite database
│
└── results/                     # Evaluation output
    └── evaluation_full_pipeline.json

Quick Start

# Clone
git clone https://github.com/ShamsRupak/ai-doc-processing-suite.git
cd ai-doc-processing-suite

# Install dependencies
pip install -r requirements.txt

# Run the full pipeline (generates docs, classifies, embeds, retrieves, evaluates)
python run_pipeline.py

# Or run without retrieval (faster, no sentence-transformers needed)
python run_pipeline.py --skip-retrieval

# Run tests
python -m pytest tests/ -v

Pipeline Steps

The pipeline runs 6 steps sequentially:

1. Document Generation — Creates 1,200 synthetic financial documents across 6 types using randomized templates with realistic data (names, amounts, dates, account numbers)
2. Document Ingestion — Loads all documents, computes stats, registers in SQLite
3. Text Processing — Recursive chunking with configurable size (512 chars) and overlap (128 chars), producing 6,364 chunks
4. Classification — TF-IDF vectorization (5,000 features, unigram+bigram) + Logistic Regression with train/test evaluation, per-class precision/recall/F1, and confusion matrix
5. Embedding & Retrieval — Builds FAISS dense index (sentence-transformers/all-MiniLM-L6-v2) + BM25 sparse index with 0.7/0.3 ensemble weighting
6. Evaluation & SQL Analytics — Computes Precision@K, MRR, NDCG, latency percentiles, throughput benchmarks, and runs 10 analytical SQL queries

Run Individual Steps

# Generate documents only
python run_pipeline.py --step generate

# Generate + classify (no retrieval)
python run_pipeline.py --step classify

SQL Database

The pipeline tracks everything in SQLite with 6 tables:

• documents — Document registry with metadata (1,200 rows)
• chunks — Text chunks with word/char counts (6,364 rows)
• classifications — Predictions with confidence and correctness (1,200 rows)
• queries — Query execution log with latency (35 rows)
• query_results — Retrieved documents per query with rank and relevance
• evaluation_runs — Full metric snapshots per pipeline run

Analytics Queries

10 pre-built SQL queries in sql/analytics.sql:

1. Document distribution by type
2. Classification accuracy by document type
3. Classification confusion matrix
4. Retrieval performance by query type
5. Top performing queries (by relevance score)
6. Worst performing queries
7. Document retrieval frequency
8. Chunk size distribution
9. Pipeline run comparison
10. End-to-end pipeline summary

Query the Database Directly

sqlite3 data/pipeline.db

-- Classification accuracy by type
SELECT actual_type, COUNT(*) AS total,
       ROUND(100.0 * SUM(CASE WHEN is_correct THEN 1 ELSE 0 END) / COUNT(*), 2) AS accuracy
FROM classifications GROUP BY actual_type;

-- Retrieval latency stats
SELECT ROUND(AVG(latency_ms), 2) AS avg, ROUND(MIN(latency_ms), 2) AS min,
       ROUND(MAX(latency_ms), 2) AS max FROM queries;

-- Pipeline summary
SELECT COUNT(*) AS docs FROM documents;

Pipeline Components

DocumentLoader

Loads .txt and .pdf files from a structured directory, using subdirectory names as document type labels. Supports manifest-based metadata enrichment.

TextProcessor

Recursive character-level text splitter with configurable chunk size, overlap, and minimum length. Uses a separator hierarchy (paragraph → sentence → word) to find natural break points. Produces 6,364 chunks from 1,200 documents at 7,400+ docs/sec.

DocumentClassifier

TF-IDF (5,000 features, unigrams + bigrams, sublinear TF) with Logistic Regression. Achieves 100% accuracy with 94.3% average confidence on financial document classification. Supports train/test evaluation with per-class metrics, confusion matrix, and top feature extraction for interpretability. Top discriminative features: statement and transfer ref (bank statements), invoice and inv (invoices), borrower and shall (loan agreements).

HybridRetriever

Combines FAISS (dense, cosine similarity on sentence-transformer embeddings) with BM25 (sparse, keyword matching). Ensemble weighting is configurable (default 0.7 dense / 0.3 sparse). Achieves MRR 0.772 and NDCG 0.793 on 35 ground truth queries across 6 document types.

PipelineEvaluator

Computes classification metrics (accuracy, precision, recall, F1), retrieval metrics (Precision@K for K=1,3,5,10; MRR; NDCG), and performance metrics (latency percentiles P50/P95/P99, throughput benchmarks). Generates JSON reports and formatted console summaries.

Configuration

All pipeline parameters are in config.yaml:

documents:
  num_documents: 1200
  types: [invoice, bank_statement, loan_agreement, pay_stub, tax_form, insurance_policy]

pipeline:
  chunk_size: 512
  chunk_overlap: 128
  embedding_model: "sentence-transformers/all-MiniLM-L6-v2"

retrieval:
  ensemble_weights:
    dense: 0.7
    sparse: 0.3

Tech Stack

• Python — Core pipeline
• SQL (SQLite) — Document registry, classification tracking, query analytics
• scikit-learn — TF-IDF vectorization, Logistic Regression, evaluation metrics
• FAISS — Dense vector similarity search
• sentence-transformers — Document/query embeddings (all-MiniLM-L6-v2)
• rank-bm25 — Sparse keyword retrieval
• NumPy / Pandas — Data processing
• pytest — Testing

Notebooks

The notebooks/ directory contains exploratory Jupyter notebooks used during development:

• Full_RAG_Pipeline — LangChain + ChromaDB + FAISS + TinyLlama hybrid retrieval
• Document_Classification_Before_Retrieval — LLM-based classification with Mistral 7B
• Experimenting_with_Model_Performance_TinyLlama — TinyLlama Q&A with Gradio
• Integrating_Open_Source_LLMs_into_RAG — LlamaIndex + Mistral 7B RAG
• Analyze_a_Scanned_PDF — Tesseract OCR extraction with bounding boxes
• Python_Libraries_for_Data_Extraction — PyPDF2 vs pdfplumber vs PyMuPDF comparison
• Resume_Parser_with_PyMuPDF — Structured field extraction from resumes

These notebooks informed the design of the modular pipeline and demonstrate the progression from prototype to production-ready components.

License

MIT