Project Description and Overview

This project implements a web-based summarization tool that leverages Large Language Models (LLMs) to provide concise summaries of web pages. Given a URL, the system fetches the content, processes it, and generates a summary - essentially creating a "Reader's Digest" version of the internet.

Setup Instructions

Prerequisites Python 3.8 or higher Jupyter Lab or Jupyter Notebook Git (for version control)

Installation Steps

1. Clone the repository bash git clone https://github.com/your-username/llm-web-summarizer.git cd llm-web-summarizer

2. Create and activate a virtual environment

bash python -m venv venv .\venv\Scripts\activate

3. Install required dependencies

bash pip install -r requirements.txt

4. Set up environment variables

Create a .env file in the project root

5. Add your OpenAI API key: OPENAI_API_KEY=your_api_key_here (You must open an OpenAI account, pay at least 5$ and generate API key)

How to Run the Notebooks

1. Launch Jupyter Lab bash jupyter lab

2. Navigate to the notebooks directory In the Jupyter Lab interface, open the notebooks folder

3. Run the main lab notebook

• Open lab_01_web_summarizer.ipynb
• Execute cells in order by selecting a cell and pressing Shift+Enter
• Follow the instructions and examples provided in the notebook

4. Experiment with the code

• Try different URLs to see how the summarizer works
• Modify parameters to understand their effects
• Add print statements to explore intermediate results

Key Features of the Code Implementation:

Automatic Content Fetching: Content is fetched automatically when the Website object is created Uses proper headers to mimic a browser request Handles HTTP errors gracefully Content Processing: Removes scripts and styles that don't contain useful content Cleans up whitespace and line breaks for better readability Extracts the page title for context Summarization: Uses OpenAI's ChatCompletion API Includes the page title and URL in the prompt for better context Handles long content by truncating if necessary Uses appropriate temperature for focused summaries Error Handling: Catches network errors during content fetching Handles API errors during summarization Provides meaningful error messages Environment Configuration: Uses python-dotenv to load environment variables Gets OpenAI API key from environment variables Usage Example:

Create a Website instance

site = Website("https://example.com")

Generate a summary

site.summarize()

Display the summary

site.display_summary()

Notes:

1. Environment Setup: Create a .env file in your project root with:

OPENAI_API_KEY=your_api_key_here

Install required packages:

pip install requests beautifulsoup4 openai python-dotenv

2. JavaScript-heavy Sites: This implementation won't work well with sites that rely heavily on JavaScript (like OpenAI's website) For those sites, you would need a browser automation tool like Selenium or Playwright

3. Token Limits: The implementation includes basic truncation for long content For production use, you might want more sophisticated chunking strategies

EXPLANATION OF FILES:

src/utils.py This file contains utility functions that support the main web summarizer functionality:

clean_text(): Removes extra whitespace and special characters from text truncate_text(): Intelligently truncates text to a specified length while trying to preserve sentence boundaries is_valid_url(): Validates URLs and checks if they're accessible extract_domain(): Extracts the domain name from a URL extract_metadata(): Extracts metadata (title, description, author, date) from BeautifulSoup objects estimate_tokens(): Provides a rough estimate of token count for text (useful for API limits) src/init.py This file makes the src directory a Python package and provides convenient imports:

Imports: Imports the main Website class and all utility functions Package Metadata: Includes version, author, and email information all: Explicitly defines what gets imported with from src import * These files provide a clean separation of concerns, with the main summarizer logic in web_summarizer.py and supporting utilities in utils.py. The init.py file makes it easy to import everything from the package with a single statement.

The tests/init.py file is quite simple and serves two main purposes:

Package Identification: It tells Python that the tests directory should be treated as a Python package. This allows you to import modules from this directory and run tests using test runners like pytest or unittest. Documentation: The docstring provides a clear description of what the test package contains and what it tests. This is helpful for developers who are new to the project or who need to understand the testing structure. Key Points: The file is intentionally minimal because test packages typically don't need to expose any functionality to the rest of the codebase. The docstring serves as documentation for anyone exploring the project structure. No actual code is needed in this file unless you have specific setup requirements for your tests (like common fixtures or test configurations). How to Use: With this file in place, you can:

Run all tests using a test runner like pytest:

bash pytest tests/ Import test modules if needed (though this is less common):

python from tests.test_web_summarizer import TestWebsite

The test discovery mechanisms in various tools will properly recognize the tests directory as a test package. This simple implementation follows Python best practices for test package organization while keeping the code clean and maintainable.

An important point on API costs (which are optional! No need to spend if you don't wish) During the course, I'll suggest you try out the leading models at the forefront of progress, known as the Frontier models. I'll also suggest you run open-source models using Google Colab. These services have some charges, but I'll keep cost minimal - like, a few cents at a time. And I'll provide alternatives if you'd prefer not to use them.

Please do monitor your API usage to ensure you're comfortable with spend; I've included links below. There's no need to spend anything more than a couple of dollars for the entire course. Some AI providers such as OpenAI require a minimum credit like $5 or local equivalent; we should only spend a fraction of it, and you'll have plenty of opportunity to put it to good use in your own projects. During Week 7 you have an option to spend a bit more if you're enjoying the process - I spend about $10 myself and the results make me very happy indeed! But it's not necessary in the least; the important part is that you focus on learning.

FREE ALTERNATIVE TO PAID APIs

Early in the course, I show you an alternative if you'd rather not spend anything on APIs: Any time that we have code like: openai = OpenAI() You can use this as a direct replacement: openai = OpenAI(base_url='http://localhost:11434/v1', api_key='ollama') And also replace model names like gpt-4o-mini with llama3.2. For week 1 day 1, you can find this in week1/solutions/day1_with_ollama.ipynb.

Below is a full example:

You need to do this one time on your computer

!ollama pull llama3.2

from openai import OpenAI MODEL = "llama3.2" openai = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")

response = openai.chat.completions.create( model=MODEL, messages=[{"role": "user", "content": "What is 2 + 2?"}] )

print(response.choices[0].message.content)