Health Assistant Chatbot

Building a Modern Health Assistant with Next.js and Upstash

Introduction

In this tutorial, we'll dive into how we built a modern Health Assistant application using Next.js and Upstash. Health Assistant project is a RAG chat app that is trained on health related data. Our goal is to create an interactive platform that uses AI to provide insights and advices to users for their health related questions.

Tech Stack

• Data Collection: scrapy crawler
• Application: Next.js
• Vector Database: Upstash
• LLM Orchestration: Langchain.js
• Generative Model: OpenAI
• Middleware: Vercel AI
• Rate Limiting: Upstash

This is a Next.js project bootstrapped with create-next-app.

Getting Started

What is a RAG Chatbot?

Retrieval-Augmented Generation (RAG) is an AI model that optimizes its responses by including information retrieved from a knowledge base based on the user input. It has 2 main phases:

1. Retrieval Phase: When a user asks a question, the chatbot first searches through a large database of previously stored knowledge bases to find relevant information. Vector databases provides the opportunity for similarity searches and retrieves related data.
2. Generation Phase: The relevant information obtained during the retrieval phase is then fed into a generative AI model. Once the response is created, it is displayed to the user in real-time, using the streaming properties.

Data Collection And Storage

Data collection of this project is handled by Scrapy, an open-source and powerful web-crawling framework written in Python. We start by initializing a Scrapy project and customizing our spider based on the data source. The parse_page function in the spider collects selected sections' data, splits them into chunks, generates vector embeddings for them, and uploads those embeddings to the Upstash Vector Database.

To run the crawler, follow these steps:

• Clone the repository: git clone https://github.com/YOUR_GITHUB_ACCOUNT/Health-Assistant-Chat-Bot
• Create a .env file in the health_scraper folder as in the example. If you don't already have an Upstash Vector Database, create one here and set 1536 as the vector dimensions. Similarly, if you dont have an Open AI key, creare one here.
• Then, simply run docker compose up. This will create a container running your crawler. If you want to run the crawlers in this project, two crawlers should be ran seperately, in order of docker compose up collect links and docker compose up fetch_content.

To customize your crawler, you can change the code segment that handles data extractions as in this example elements = response.xpath("//div[contains(@class, 'content-repository-content')]//p | //div[contains(@class, 'content-repository-content')]//li").

❗ Note: Running crawler may take time. To see the progress, you can check check the logs or monitor your vector database from your Upstash account.

Data Retrieval And Response Generation

First, run the development server:

npm run dev
# or
yarn dev
# or
pnpm dev
# or
bun dev

Open http://localhost:3000 with your browser to see the result.

Deploy on Vercel

The easiest way to deploy your Next.js app is to use the Vercel Platform from the creators of Next.js.

Check out our Next.js deployment documentation for more details.