Implement content filtering for web pages

[codegen-sh]

Problem

The current web content retrieval implementation extracts all text from a webpage, including navigation menus, footers, ads, and other non-essential content. This can waste tokens, dilute the relevant information, and make it harder for the LLM to focus on the important content.

Proposed Solution

Implement content filtering to focus on the main content of web pages:

1. Use heuristics or machine learning to identify the main content of a webpage
2. Filter out non-essential elements like navigation, footers, ads, etc.
3. Prioritize the most relevant content for processing

Implementation Details

• Explore and integrate content extraction libraries like:
  • newspaper3k for news articles
  • readability-lxml for general web content
  • trafilatura for more advanced content extraction
• Add a new configuration option WEB_CONTENT_FILTERING_ENABLED (default: true)
• Modify the WebBaseLoader usage to apply content filtering
• Implement fallback mechanisms when content filtering fails
• Add logging to track the effectiveness of content filtering

Expected Outcome

• The bot will focus on the most relevant content from web pages
• Token usage will be more efficient
• Response quality will improve by reducing noise from non-essential content
• The bot will be able to handle a wider range of websites effectively