A comprehensive Information Retrieval (IR) system that combines web scraping, indexing techniques, fuzzy matching, and spelling correction to provide accurate and tolerant search over a real-world book dataset.
This project implements a complete search engine pipeline, starting from collecting data through web scraping to building efficient search indexes and supporting tolerant query processing.
The dataset contains 1,000 books collected from Books to Scrape, and the system supports exact search, phrase search, multi-term queries, and spelling correction.
- Web scraping of 1,000 books
- Text preprocessing and normalization
- Inverted Index construction
- Positional Index for phrase search
- K-Gram Index for tolerant retrieval
- Single-term search
- Multi-term (AND) search
- Exact phrase search
- Spelling correction using:
- K-Gram candidate generation
- Jaccard Similarity
- Edit Distance (Levenshtein)
- Soundex phonetic matching
- Ranked search results
| Category | Technologies |
|---|---|
| Programming Language | Python |
| Web Scraping | BeautifulSoup, Requests |
| Data Processing | Pandas, Regex |
| NLP | NLTK |
| Machine Learning Utilities | Scikit-learn |
| Indexing | Inverted Index, Positional Index, K-Gram |
| Similarity | Jaccard Similarity, Edit Distance |
| Phonetic Matching | Soundex |
- Source: Books to Scrape
- Number of documents: 10,000 books
Each record includes:
- Book title
- Category
- Product description
- Collect book data using web scraping.
- Clean and preprocess the text.
- Build the Inverted Index.
- Build the Positional Index.
- Build the K-Gram Index.
- Process user queries.
- Apply spelling correction when needed.
- Retrieve and rank matching documents.
Open the notebook and execute all cells sequentially.
The notebook will:
- Scrape the dataset
- Preprocess the documents
- Build all indexes
- Execute search queries
- Demonstrate spelling correction
- Display ranked search results
Retrieve documents containing a specific keyword.
Search multiple terms using AND logic with TF-based ranking.
Locate documents containing an exact phrase using positional indexing.
Automatically suggests corrections for misspelled queries using:
- K-Gram Index
- Jaccard Similarity
- Edit Distance
- Soundex
Input:
retrival
Output:
retrieval
The correction is selected based on similarity scores and edit distance before executing the search.
- TF-IDF ranking
- BM25 ranking
- Boolean query parser
- Wildcard search
- Query auto-completion
- Web interface using Streamlit or Flask