Implement content chunking for large web pages

[codegen-sh]

Problem

The current web content retrieval implementation loads the entire webpage content into memory at once. For very large pages, this can cause memory issues and exceed token limits of the LLM models.

Proposed Solution

Implement a chunking mechanism for large web content to process it in manageable pieces:

1. Split large web content into smaller chunks (e.g., paragraphs or sections)
2. Process each chunk separately or implement a streaming approach
3. Combine the results or prioritize the most relevant chunks

Implementation Details

• Add a new configuration option WEB_CONTENT_CHUNK_SIZE (default: 4000 tokens)
• Integrate LangChain's text splitters (e.g., RecursiveCharacterTextSplitter) to chunk content
• Modify the answer_webcontent method to handle chunked content
• Implement a strategy for processing chunks (e.g., process most relevant chunks first)
• Update the prompt template to work with chunked content

Expected Outcome

• The bot will be able to process larger web pages without memory issues
• Content will be processed more efficiently
• The quality of responses for large web content will improve
• The application will be more stable when handling large content