Local LLM on Kubernetes

This project provides a simple and straightforward way to deploy a local Language Model (LLM) on a Kubernetes cluster. It uses Ollama to serve the LLM and Open WebUI as a web interface for interacting with the model.

Features

• Local LLM Deployment: Run your own LLM locally on your Kubernetes cluster.
• Ollama Integration: Uses Ollama to serve a variety of open-source language models.
• Web Interface: Provides a user-friendly web interface with Open WebUI.
• Kubernetes-Native: Deploys all components as Kubernetes resources (Deployments, Services, Ingress).
• Customizable: Easily customize the LLM model and other configurations.
• Pre-configured Admin: The Open WebUI admin user can be pre-configured for immediate use.

Prerequisites

• A running Kubernetes cluster.
• kubectl installed and configured to connect to your cluster.
• An Ingress controller (like NGINX Ingress Controller) running in your cluster to expose the web UI.

Getting Started

1. Clone the repository or download the manifest file: You will need the k8sManifests.yaml file.

2. Customize the manifest (optional): You can edit the k8sManifests.yaml file to change the default configuration. For example, you can change the LLM model to be pulled by Ollama.

   In the ollama deployment, find the initContainers section and change the args to pull a different model:

         args: ["ollama serve & sleep 5 && ollama pull <your-model-name>"]

   By default, it pulls phi3.5.

3. Apply the manifest:

   kubectl apply -f k8sManifests.yaml

   This will create all the necessary Kubernetes resources:

   • A PersistentVolumeClaim for Ollama's model storage.
   • A Deployment for Ollama.
   • A Service for Ollama.
   • A Deployment for Open WebUI.
   • A Service for Open WebUI.
   • An Ingress resource to expose the Open WebUI.

4. Access the Web UI: Once the Ingress is set up, you can access the Open WebUI in your browser. The default configuration uses the hostname chat.local.opscale.ir. You might need to add a corresponding entry to your /etc/hosts file or configure your DNS to point this hostname to your Ingress controller's IP address.

   The default admin credentials are:

   • Email: admin@local.opscale.ir
   • Password: ChangeMe123!

Configuration

Ollama

• Model: To change the model, modify the args of the model-puller init container in the ollama deployment in k8sManifests.yaml.
• Resources: You can adjust the CPU and memory resources for the ollama container based on the model you are using.

Open WebUI

• Admin User: The admin user is configured via environment variables in the open-webui deployment in k8sManifests.yaml. You can change the ADMIN_EMAIL, ADMIN_PASSWORD, and ADMIN_NAME values.
• Sign-up: The ENABLE_SIGNUP environment variable is set to false by default to disable user sign-ups.

Architecture

The project consists of the following components:

graph LR
    subgraph User
        U[Browser]
    end

    subgraph Kubernetes Cluster
        Ingress(chat.local.opscale.ir)
        WebUI[Open WebUI]
        Ollama[Ollama]
        PVC(Model Storage)

        U -- HTTPS --> Ingress
        Ingress -- HTTP --> WebUI
        WebUI -- "Talks to" --> Ollama
        Ollama -- "Loads/Saves Models" --> PVC
    end

    style U fill:#f9f,stroke:#333,stroke-width:2px

Loading

• Ollama Deployment: A Kubernetes Deployment that runs the Ollama server. It uses a PersistentVolumeClaim to store the downloaded LLM models. An initContainer is used to pull the desired model on startup.
• Ollama Service: A ClusterIP service that exposes the Ollama server within the cluster.
• Open WebUI Deployment: A Kubernetes Deployment that runs the Open WebUI. It is configured to connect to the Ollama service.
• Open WebUI Service: A ClusterIP service that exposes the Open WebUI within the cluster.
• Ingress: A Kubernetes Ingress resource that exposes the Open WebUI service to the outside world via a hostname.