P.R.A.K.S.

A general-purpose search engine, similar to Google or DuckDuckGo.

Demo

demo.mp4

Screenshots

Features

• A web crawler built from scratch
• An indexer using the TF-IDF algorithm, also built from scratch
• A website with an interface similar to other search engines, such as Google or DuckDuckGo
• An extensive test suite of 50+ unit/integration tests for all services

Usage

First, clone the repo:

git clone https://github.com/dastarruer/search-engine/
cd search-engine

Then, create a .env file in the root of search-engine/ with the following environment variables:

# Change these to be whatever you like
DB_PORT=5432
DB_NAME=postgres
DB_USER=postgres
DB_ENDPOINT=db
DB_PASSWORD=123

Finally, run the following command to start all services at once (crawler, indexer, and website):

docker compose up -d

To use the website, go to http://localhost:80 in your web browser of choice.

Note

Please note that there will not be many search results at first. But as the crawler and indexer continue to gather more and more results, both the accuracy and number of search results will increase.

Connecting to Postgres

To connect to the database, run the following command:

PGPASSWORD=<DB_PASSWORD> psql -h localhost -p <DB_PORT> -U <DB_USER> -d <DB_NAME>

If you're using the `.env. file example provided above, this becomes:

PGPASSWORD=123 psql -h localhost -p 5432 -U postgres -d postgres

To get the number of crawled pages:

SELECT COUNT(*) FROM pages;

To get the number of indexed pages:

SELECT COUNT(*) FROM pages WHERE is_indexed = true;