A flexible Python web scraper with an interactive menu system that accepts ANY website URL and extracts links, headings, images or text content. User chooses exactly what data to extract and whether to save as Excel, CSV or both.
👉 Watch the full execution demo on YouTube
- Accepts any website URL as input — works on any public website
- Shows exactly what was found on the page before extracting
- Interactive menu — user picks what type of data to extract
- User chooses output format — Excel, CSV or both
- Automatically names output files with website domain and timestamp
- Professional error handling — never crashes on bad URLs
| Feature | Description |
|---|---|
| Universal Compatibility | Works on any publicly accessible website |
| Interactive Menu | User controls exactly what gets extracted |
| 4 Data Types | Links, Headings, Images, or Text paragraphs |
| Flexible Output | Excel, CSV, or both simultaneously |
| Smart Naming | Output files named with domain and timestamp |
| Auto URL Fix | Converts relative URLs to absolute automatically |
| Full Error Handling | Handles timeouts, HTTP errors, connection issues |
| Page Summary | Shows count of all found elements before extraction |
| Option | What Gets Extracted | Columns |
|---|---|---|
| 1 — Links | All clickable links on the page | Link Text, URL |
| 2 — Headings | All H1 H2 H3 H4 headings | Heading Type, Heading Text |
| 3 — Images | All images on the page | Image URL, Alt Text |
| 4 — Text | All paragraph text content | Paragraph Text |
| 5 — Everything | All 4 types at once | Separate files for each |
- Python 3.14
- BeautifulSoup4 — HTML parsing
- Requests — HTTP requests with full headers
- Pandas — data processing and file export
- Openpyxl — Excel file creation
- Urllib — URL parsing for base URL detection
- Datetime — timestamp generation for filenames
Step 1 — Install libraries:
pip install requests beautifulsoup4 pandas openpyxl
Step 2 — Run the scraper:
python universal_scraper.py
Step 3 — Follow the interactive menu:
Enter the website URL to scrape: https://books.toscrape.com
--- WHAT WAS FOUND ON THIS PAGE ---
Links found: 220
Headings found: 12
Images found: 51
Paragraphs: 8
--- WHAT DO YOU WANT TO EXTRACT? ---
1. All Links
2. All Headings
3. All Images
4. All Text/Paragraphs
5. Everything
Enter your choice (1-5): 1
--- OUTPUT FORMAT ---
1. Excel (.xlsx)
2. CSV (.csv)
3. Both Excel and CSV
Enter your choice (1-3): 3
==================================================
UNIVERSAL WEBSITE SCRAPER
Built by Aryan — Python Developer
==================================================
Enter the website URL to scrape: https://quotes.toscrape.com
Connecting to: https://quotes.toscrape.com
Successfully connected!
--- WHAT WAS FOUND ON THIS PAGE ---
Links found: 54
Headings found: 3
Images found: 0
Paragraphs: 12
Extracting data...
Saved to Excel: scraped_quotes.toscrape.com_20260702_1430_links.xlsx
Total records saved: 54
--- SCRAPING COMPLETE ---
Website: https://quotes.toscrape.com
Time: 2026-07-02 14:30
Thank you for using Universal Website Scraper!
| File | Description |
|---|---|
| universal_web_scraper.py | Main scraper with interactive menu |
| scraped_books.toscrape.com_links.csv | Sample — links from books site |
| scraped_quotes.toscrape.com_headings.xlsx | Sample — headings from quotes site |
| scraped_quotes.toscrape.com_images.xlsx | Sample — images from quotes site |
| scraped_quotes.toscrape.com_headings.csv | Sample — headings from quotes site |
| scraped_realpython_links.xlsx | Sample — links from jobs site |
| scraped_realpython_links.csv | Sample — links from jobs site |
| README.md | Project documentation |
- How to build reusable Python functions for different scraping tasks
- How to create interactive command line menus with input()
- How to parse and reconstruct URLs using urllib
- How to export the same data to multiple formats simultaneously
- How to auto-generate filenames using domain names and timestamps
- How to make a tool flexible enough to handle any website
- Marketing teams extracting all links from competitor websites
- SEO analysts collecting all headings for content analysis
- Researchers gathering text content from multiple sources
- Developers auditing all images on a website
- Data analysts extracting structured content from any webpage
Built by Aryan — BS Computer Engineering Student, COMSATS University Islamabad. Specializing in Python web scraping, automation, and data collection.