🔍 Distributed Search Engine (DSE)

A high-performance, containerized, distributed search engine built with FastAPI and Docker. This project implements a scalable architecture for dynamically fetching results from external sources (Wikipedia, StackOverflow), partitioning the data into chunks, and distributing them across multiple worker nodes for parallel processing and aggregation.

🏗️ System Architecture

The project is structured around a Gateway-Worker pattern. The central Gateway fetches real-time data from external APIs (Wikipedia, StackOverflow), partitions the content, and distributes it to multiple worker nodes that process and attribute the data in parallel.

Phase 1: Single Node (Single Machine)

Initial implementation focused on a standalone node managing its own inverted index and search logic.

Phase 2: Multiple Nodes (Single Machine)

Introduction of the Gateway Service, which orchestrates search queries across multiple containers running on the same host via Docker Compose.

Phase 3: Static Nodes (Multiple Machine) (Currently 3 static nodes)

Phase 3: Static Nodes (Multiple Machine) (Currently 3 static nodes)

The goal of Phase 3 is to achieve full distribution across multiple physical or virtual machines, implementing more resilient discovery and load balancing.

✨ Key Features & Optimizations

1. Dual-Stream Tokenization (Code vs. Prose)

• Files: index_services.py
• Mechanism: During startup document indexing, text content is concurrently tokenized into two distinct streams:

  Stream	Tokenizer	Behavior	Example
  Prose	tokenize_prose()	Extracts clean alpha-numeric terms, discarding common stop words (the, is, at, etc.)	"python scripting" → ["python", "scripting"]
  Code	tokenize_code()	Retains programming syntax characters like underscores, dots, and hyphens	"user_id str.replace" → ["user_id", "str.replace"]

• Advantage: Standard prose tokenizers split code elements like ctx.execute() into generic ctx and execute tokens, losing technical context. Keeping syntax structures intact ensures precise search matching on exact API references and variable names.

2. Metadata-Weighted Scoring (Social Boost)

• Files: search_services.py, index_services.py
• Mechanism: Documents are enriched with interactive metadata (such as upvote and edit counts). Results are ranked dynamically based on this logarithmic scoring system: $$\text{final_score} = \text{text_match_score} \times \left(1 + \frac{\text{score}}{100 + \text{score}}\right)$$
• Advantage: This ensures highly upvoted/established answers receive a logarithmic authority boost over low-quality matches, prioritizing proven solutions without letting authority overshadow query relevance.
  • 5 upvotes $\rightarrow$ ~5% boost
  • 100 upvotes $\rightarrow$ ~50% boost
  • 10,000 upvotes $\rightarrow$ ~99% boost (caps gracefully)

3. Source-Aware Search Modes

• Files: gateway/main.py, App.jsx
• Mechanism: The user interface supports three query filters that map cleanly to targeted datasets:

  Mode	External APIs	Local Index
  🌐 All (Default)	Wikipedia + StackOverflow	Prose Index
  📖 Wikipedia	Wikipedia Only	Prose Index
  💻 StackOverflow	StackOverflow Only	Code Index

• Advantage: Fine-tunes performance and output quality. Conceptual queries use Wikipedia indices, whereas technical queries target code token streams from StackOverflow.

4. Distributed Processing of External Results

• Files: gateway/main.py
• Mechanism: Real-time Wikipedia/StackOverflow responses are split into chunks and distributed asynchronously across active background worker nodes for parallel metadata enrichment and attribution.
• Advantage: Simulates horizontal scaling on clusters, unloading query parsing bottlenecks from the central gateway to compute nodes.

5. Startup Index Building

• Files: main.py
• Mechanism: Scans internal storage folders on node boot-up and automatically structures the code and prose inverted indexes in-memory (build_index()).
• Advantage: Replaces manual index triggers; the cluster is search-ready the moment servers boot.

6. Premium Responsive Light UI

• Files: App.jsx, index.css, LogoAnimation.jsx, MatrixBackground.jsx
• Mechanism: Features a premium light Wedgwood blue and cream theme.
  • Includes a background canvas displaying animated colliding nodes and binary streams.
  • Incorporates a .webm intro animation that plays dynamically on page load before cross-fading into the static brand logo.
  • Uses premium glassmorphic frosted-blur overlays behind form inputs and results.
  • Disables user-drag and selections on visual assets to preserve brand styling integrity.

---

🛠️ Tech Stack

• Language: Python 3.9+
• Web Framework: FastAPI
• Frontend: React, Tailwind CSS, Framer Motion
• ASGI Server: Uvicorn
• HTTP Client: HTTPX (for asynchronous node calls)
• Containerization: Docker & Docker Compose
• Data Sourcing: Wikipedia API & StackOverflow (StackExchange) API.

---

🚀 Getting Started

Prerequisites

• Docker Desktop installed and running.
• Node.js (v16+) and Python 3.9+ (for local development).

Quick Launch

1. Clone the Repository:

   git clone https://github.com/AnshMNSoni/Distributive-Search-Engine.git
   cd Distributive-Search-Engine

2. Spin up the Cluster:

   docker-compose up --build

3. Access the Search API:

   • Gateway Search: http://localhost:8000/search?q=your_keyword
   • Interactive API Docs (Swagger UI): http://localhost:8000/docs/

4. UI Experience:

   cd frontend
   npm install
   npm run dev

---

📋 Current Scope & Roadmap

• Phase 1: Standalone Indexing & Search
• Phase 2: Dockerized Multi-Node Aggregation
• Phase 3: Dynamic Wikipedia & StackOverflow Integration
• Feature: Premium Light UI with interactive collision-matrix background
• Feature: Startup Auto-indexing & Dual-Stream Tokenizers
• Feature: Logarithmic Social Boost Metadata Ranking
• Feature: Dynamic Node Registration & Heartbeats
• Feature: Fault-tolerant Querying (Handle node timeouts gracefully)

---

🤝 Contributing

Contributions are welcome! Please feel free to submit a Pull Request or open an issue for any bug reports or feature requests.

1. Fork the Project
2. Create your Feature Branch (git checkout -b feature/AmazingFeature)
3. Commit your Changes (git commit -m 'Add some AmazingFeature')
4. Push to the Branch (git push origin feature/AmazingFeature)
5. Open a Pull Request

🟢 Active Contributors

📜 License

This project is licensed under the MIT License.