Skip to content

document_preprocessor.MD

Latest commit

 

History

History
69 lines (45 loc) · 2.49 KB

document_preprocessor.MD

File metadata and controls

69 lines (45 loc) · 2.49 KB
Raw

document_preprocessor.py

This module handles document parsing and text extraction for all supported file types in DocMind. It transforms raw documents into structured text chunks that are embedded and indexed for later retrieval.

Responsibilities

  • Extract text from PDFs, images (via OCR), DOCX, and TXT files
  • Normalize and split text into clean paragraph-level chunks
  • Add these chunks to the FAISS vector index
  • Remove any old indexed data for the same document ID
  • Provide both single-document and batch preprocessing functions

Supported File Types

Extension Handler Function Method
.pdf extract_text_from_pdf() PyMuPDF blocks
.png, .jpg, .jpeg extract_text_from_image() Tesseract OCR
.docx extract_text_from_docx() python-docx
.txt extract_text_from_txt() Basic file read

Key Functions

preprocess_document(session_id: str, file_path: str, doc_id: str)

  • Detects file type and delegates to the correct extractor
  • Splits the extracted text into cleaned chunks (\n delimited)
  • Calls remove_from_metadata(session_id, doc_id) and then add_to_metadata(session_id, doc_id, chunks), both function are related to meta_store.py

preprocess_batch(session_id: str, file_paths: List[str])

  • Processes multiple files in sequence
  • Skips failed documents but logs the error
  • Returns a list of successfully processed doc_ids

🔍 Extraction Behavior

  • PDFs: Extracted block by block using page.get_text("blocks"), with newlines stripped for paragraph cohesion.
  • Images: Processed using pytesseract.image_to_string() with --psm 6 for better structure; double-spaced lines inferred as paragraphs.
  • DOCX: All non-empty paragraphs concatenated.
  • TXT: Raw text read from file, no transformation.

🔗 Dependencies

  • PyMuPDF, pytesseract, Pillow, docx, shutil, os, typing
  • add_to_metadata(session_id, doc_id, chunks) and remove_from_metadata(session_id, doc_id) from meta_store.py
  • logger from core/logger.py for structured logs

Notes

  • Document ID (doc_id) is derived from the uploaded file's name (including extension)

Related Modules

meta_store.py – adds/removes chunks and embedding for the specified document in the metadata

pipeline_routes.py – calls preprocess_batch(session_id, file_paths) during analysis