Crawlitics is an AI-powered, Python-based eCommerce web crawler, scraper, and data analyst.
It combines asynchronous web automation using Playwright, NLP-powered matching via sentence-transformers, cosine similarity and fuzzy matching, modular scraping powered by Crawl4AI and structured data extraction provided by the Gemma3 local LLM model (via Ollama). Crawlitics is purpose-built for collecting, analyzing, and comparing product data across online retailers.
- Asynchronous crawling and scraping across multiple eCommerce platforms (e.g. Ozone, Emag, Technomarket)
- Automated and intelligent UI interaction using Playwright (filters, price sliders, next pages, etc.)
- Support for custom user-defined filters (e.g. brand, model, RAM, storage, color, price range, etc.)
- Semantic and accurate filter matching via SentenceTransformers, cosine similarity and fuzzy matching
- Seamless integration with Ollama (and the Gemma3 model) for structured data extraction and analysis (in the future) and PostgreSQL for data storing
- Dynamic concurrency and auto-throttling
- Extensible scraping logic with site-specific configs and selectors
- Clone the repo
git clone https://github.com/Dimitar0528/crawlitics.git
cd crawlitics- Install dependencies with:
pip install -r requirements.txt- Create a .env file with the following data needed to set up your local PostgreSQL database:
DB_HOST=YOUR_HOST
DB_PORT=YOUR_PORT
DB_NAME=YOUR_DB_NAME
DB_USER=YOUR_DB_USER
DB_PASSWORD=YOUR_DB_PASSWORD- Run the main python file
python main.pyCrawlitics supports local large language model inference using Ollama. There are two main ways to install and use Ollama:
- Download Ollama from https://ollama.com
- Install and / or run a LLM model (e.g. gemma3, llama3.1, mistral or others) with:
ollama run gemma3:latest- Download Ollama Docker Image via:
docker pull ollama/ollama- Start the Docker container with:
docker run --rm --gpus=all -d -v ollama_data:/root/.ollama -p 11434:11434 --name ollama ollama/ollama - Install and / or run a LLM model with:
docker exec -it ollama ollama run gemma3 - [✅] Semantic matching for multiple types of user filters (brand, RAM, storage, color, etc.)
- [✅] Dynamic JSON schema generation for structured product output
- [🔜] LLM-powered AI agent for interactive product data analysis
- [🔜] Export to CSV/JSON with rich metadata
This project is licensed under the MIT License
It also includes components from the Crawl4AI project, which are licensed under the Apache License 2.0.
This product includes software developed by UncleCode (https://x.com/unclecode) as part of the Crawl4AI project (https://github.com/unclecode/crawl4ai).