WebIndexer - Microservices Architecture

A comprehensive web indexing and search platform built with .NET 9, Next.js, TypeSense, and MassTransit for event-driven architecture.

Architecture Overview

This project follows a microservices architecture with the following components:

• WebIndexer Gateway (Port 5106) - API Gateway using YARP reverse proxy
• WebIndexer API (Port 5237) - Main API service with PostgreSQL database
• WebIndexer Search (Port 5062) - Search service with TypeSense integration
• Web Indexer UI - Next.js frontend application
• Shared Kernel - Shared domain library

Prerequisites

Before running the application, ensure you have the following installed:

Required Software

• .NET 9 SDK
• Node.js 18+
• pnpm (recommended) or npm
• PostgreSQL 15+
• RabbitMQ
• TypeSense

External Services Setup

1. PostgreSQL Database

# Install PostgreSQL and create database
createdb WebIndexer
# Or using psql:
psql -U postgres
CREATE DATABASE WebIndexer;
CREATE USER admin WITH PASSWORD 'admin123';
GRANT ALL PRIVILEGES ON DATABASE WebIndexer TO admin;

2. RabbitMQ Message Broker

# Using Docker (recommended)
docker run -d --name rabbitmq -p 5672:5672 -p 15672:15672 rabbitmq:3-management

# Or install locally and start service
# Windows: Start RabbitMQ service
# Linux/Mac: sudo systemctl start rabbitmq-server

3. TypeSense Search Engine

# Using Docker (recommended)
docker run -d --name typesense -p 8108:8108 -v/tmp/typesense-data:/data typesense/typesense:0.25.1 --data-dir /data --api-key=mysecretapikey --listen-port 8108 --enable-cors

# Or install locally
# Follow TypeSense installation guide for your OS

Project Structure

src/
├── Frontend/
│   └── web-indexer-ui/          # Next.js frontend application
├── SharedKernel/
│   └── SharedKernel/            # Shared domain library (NuGet package)
├── WebIndexer/
│   └── WebIndexer/              # Main API service
├── WebIndexerGateway/
│   └── WebIndexerGateway/       # API Gateway service
└── WebIndexerSearch/
    └── WebIndexerSearch/        # Search service

Running the Application

Step 1: Build Shared Kernel Package

First, build and pack the shared kernel library:

cd src/SharedKernel/SharedKernel
dotnet pack

This creates the WebIndexer.SharedKernel.1.0.0.nupkg file that other services depend on.

Step 2: Start External Services

Make sure PostgreSQL, RabbitMQ, and TypeSense are running:

# Start PostgreSQL (if not running as service)
# Windows: Start PostgreSQL service
# Linux/Mac: sudo systemctl start postgresql

# Start RabbitMQ (if using Docker)
docker start rabbitmq

# Start TypeSense (if using Docker)
docker start typesense

Step 3: Run Backend Services

Open multiple terminal windows and run each service:

Terminal 1: WebIndexer API Service

cd src/WebIndexer/WebIndexer
dotnet restore
dotnet ef database update  # Apply database migrations
dotnet run

Service URL: http://localhost:5237

Terminal 2: WebIndexer Search Service

cd src/WebIndexerSearch/WebIndexerSearch
dotnet restore
dotnet run

Service URL: http://localhost:5062

Terminal 3: WebIndexer Gateway

cd src/WebIndexerGateway/WebIndexerGateway
dotnet restore
dotnet run

Service URL: http://localhost:5106

Step 4: Run Frontend Application

cd src/Frontend/web-indexer-ui
pnpm install
pnpm dev

Frontend URL: http://localhost:3000

Service Endpoints

WebIndexer Gateway (Main Entry Point)

• Base URL: http://localhost:5106
• Health Check: GET /health
• Gateway Info: GET /
• Search API: GET /api/search?query={query}&page={page}&perPage={perPage}
• Search Stats: GET /api/search/stats
• Create Website: POST /websites

WebIndexer API (Direct Access)

• Base URL: http://localhost:5237
• Create Website: POST /websites

WebIndexer Search (Direct Access)

• Base URL: http://localhost:5062
• Search: GET /api/search?query={query}&page={page}&perPage={perPage}
• Stats: GET /api/search/stats
• Swagger UI: http://localhost:5062/swagger

Configuration

Database Configuration

The main API service uses PostgreSQL with the following connection string:

Host=localhost;Database=WebIndexer;Username=admin;Password=admin123

Message Broker Configuration

All services use RabbitMQ with default credentials:

• Host: localhost
• Username: guest
• Password: guest

TypeSense Configuration

The search service connects to TypeSense with:

• Host: localhost
• Port: 8108
• API Key: mysecretapikey
• Protocol: http

Development Workflow

1. Database Migrations

When making changes to the database schema:

cd src/WebIndexer/WebIndexer
dotnet ef migrations add MigrationName
dotnet ef database update

2. Testing API Endpoints

Use the provided .http files to test endpoints:

• src/WebIndexer/WebIndexer/WebIndexer.http - Test main API
• src/WebIndexerGateway/WebIndexerGateway/WebIndexerGateway.http - Test gateway
• src/WebIndexerSearch/WebIndexerSearch/WebIndexerSearch.http - Test search service

3. Frontend Development

The Next.js frontend is configured to connect to the gateway at http://localhost:5106. The API service configuration is in src/Frontend/web-indexer-ui/app/services/api.ts.

Troubleshooting

Common Issues

1. Database Connection Failed

   • Ensure PostgreSQL is running
   • Verify connection string in appsettings.json
   • Check if database and user exist

2. RabbitMQ Connection Failed

   • Ensure RabbitMQ is running
   • Check if port 5672 is available
   • Verify credentials in configuration

3. TypeSense Connection Failed

   • Ensure TypeSense is running on port 8108
   • Check API key configuration
   • Verify CORS settings

4. Port Already in Use

   • Check if another instance is running
   • Use netstat -an | findstr :PORT (Windows) or lsof -i :PORT (Mac/Linux)
   • Kill the process or change port in launchSettings.json

5. Shared Kernel Package Not Found

   • Ensure you've built the shared kernel package first
   • Run dotnet pack in the SharedKernel directory
   • Check if the .nupkg file exists in the nupkgs folder

Service Dependencies

The services have the following startup order dependencies:

1. External services (PostgreSQL, RabbitMQ, TypeSense)
2. WebIndexer API (creates database schema)
3. WebIndexer Search (creates TypeSense collections)
4. WebIndexer Gateway (proxies to other services)
5. Frontend (connects to gateway)

Logs and Debugging

• Each service logs to console with different log levels
• Check individual service logs for specific errors
• Use Swagger UI at http://localhost:5062/swagger for search service testing
• Gateway provides detailed request/response logging

Production Deployment

For production deployment, consider:

1. Environment Variables: Replace hardcoded values with environment variables
2. Docker Containers: Containerize each service for easier deployment
3. Load Balancing: Use a proper load balancer instead of YARP for production
4. Security: Implement proper authentication and authorization
5. Monitoring: Add application monitoring and health checks
6. SSL/TLS: Configure HTTPS for all services

Technology Stack

• Backend: .NET 9, ASP.NET Core, Entity Framework Core
• Database: PostgreSQL
• Search: TypeSense
• Message Broker: RabbitMQ with MassTransit
• API Gateway: YARP (Yet Another Reverse Proxy)
• Frontend: Next.js 15, React 19, TypeScript, Tailwind CSS
• Package Manager: pnpm
• Architecture: Microservices, Domain-Driven Design (DDD), CQRS