OpenCourse — Quick Run Instructions

Minimal steps to run the full system locally (databases + services).

Architecture

The system is organized into separate layers to keep responsibilities clear:

• Application Layer (Node.js/Express): exposes REST APIs, handles authentication and client requests.
• Sync-Harvesting Layer (Python): ingests and normalizes course data from Coursera and Udemy, writes to MongoDB.
• Spark Layer (PySpark / FastAPI): computes TF-IDF and vector embeddings, and stores vectors in Qdrant.
• Database Layer: MongoDB stores course metadata and analytics; Qdrant serves vector similarity search.
• Frontend Layer (React): user interface for browsing and searching courses. Services communicate via MongoDB and Qdrant; Docker Compose is used to orchestrate local deployments. Frontend layer is communicated with the application layer.

Prerequisites

• Docker & Docker Compose: for containerized deployment.
• .env file: create a .env file in the project root with necessary environment variables (see below).
• Kaggle account: for Udemy dataset access (set KAGGLE_USERNAME and KAGGLE_KEY in .env).
• Account in OpenCourse system: to log in and trigger syncing procedures.
• Internet access: required for initial data fetching, and syncing from Coursera and Udemy

Run Instructions

1. Start services with a single Docker Compose command (recommended)

Run this single instruction from the repository root to build images (pulling fresh bases), recreate services and start everything in detached mode:

# from repository root (Windows / PowerShell)
docker compose up -d --build --force-recreate --renew-anon-volumes

Or using the legacy CLI name:

docker-compose up -d --build --force-recreate --renew-anon-volumes

This single command is sufficient to build and start the entire application stack (application, frontend, sync-harvester, spark, MongoDB, Qdrant).

Services started by Compose include the application, sync-harvester, spark (when configured), MongoDB and Qdrant.

2. Populate databases (initial harvest)

docker-compose exec sync-harvester python harvester_v1.py

This fetches courses, writes normalized documents to MongoDB, and triggers vector generation.

3. Access the Frontend Open your browser and navigate to http://localhost:3000 to access the OpenCourse frontend.

4. Common checks and ports

• Application API: http://localhost:5000
• Spark/ML API: http://localhost:16000
• Sync API: http://localhost:8000
• Frontend: http://localhost:3000
• Qdrant Dashboard: http://localhost:6333/dashboard
• MongoDB: mongodb://localhost:27017

Environment Variables

• Create a .env in the project root with the required environment variables listed below. The repository contains a sample .env at the project root — copy or edit it for your environment.

Sample .env variables (example values included; replace with secure/real values):

MONGO_INITDB_ROOT_USERNAME=admin
MONGO_INITDB_ROOT_PASSWORD=password
MONGODB_URI=mongodb://admin:password@mongodb:27017/OpenCoursesDB?authSource=admin
MONGODB_DB_NAME=OpenCoursesDB
MONGODB_COLLECTION=courses
MONGODB_ANALYTICS_COLLECTION=course_analytics
MONGODB_REC_COLLECTION=recommendations
JWT_SECRET=<your-jwt-secret>
VECTORS_DB_URI=http://qdrant:6333
VECTORS_DB_NAME=VectorsDB
VECTORS_COLLECTION=course_vectors
INTERNAL_API_KEY=<internal-api-key>
KAGGLE_USERNAME=<kaggle-username>
KAGGLE_KEY=<kaggle-key>

Environment notes:

• MONGODB_URI should point to the MongoDB service used by the app (the Compose network service name is mongodb by default).
• JWT_SECRET is used to sign authentication tokens and must be kept secret.
• VECTORS_DB_URI should point to the Qdrant service (Compose service name qdrant).
• INTERNAL_API_KEY is used for internal requests between services; set a shared secret.
• If you prefer to run databases standalone, start mongodb and qdrant containers manually (expose ports 27017, 6333, 6334).

Create an Account

To trigger syncing procedures, you need an account in the OpenCourse system. Follow these steps:

1. Download Postman or use curl to interact with the API.
2. Send a POST request to http://localhost:5000/auth/register with JSON body:

{
  "username": "testuser",
  "password": "testpassword"
}

3. You can now log in via the frontend at http://localhost:3000 using these credentials.

Need the README to include more detailed setup (env vars, troubleshooting, or development commands)? I can expand it.

License

This project is licensed under the GNU General Public License v3.0. See the LICENSE file for details.