Skip to content

PREREQUISITES.md

Latest commit

 

History

History
143 lines (99 loc) · 3.6 KB

PREREQUISITES.md

File metadata and controls

143 lines (99 loc) · 3.6 KB
Raw

Prerequisites

Setup requirements for running Granite Switch tutorials.

Hardware Requirements

Tutorial Type GPU Required
Quickstart (micro model) Yes (CUDA)
HuggingFace Scripts (3B) Yes (CUDA)
vLLM/Mellea (8B) Yes (CUDA)
Large Models (30B) Yes (CUDA)

Software Requirements

Python Version

Python 3.10+ is required.

Base Installation

pip install granite-switch

HuggingFace Backend

For direct model inference with HuggingFace Transformers:

pip install "granite-switch[hf,compose]"

This includes:

  • transformers for model loading and generation
  • torch with CUDA support
  • peft for LoRA operations
  • Compose tools for model building

vLLM Backend

For production inference with vLLM:

pip install "granite-switch[vllm]"

This includes:

  • vllm>=0.19.1 for high-performance inference
  • Tensor parallelism support for multi-GPU setups

Mellea Integration

Mellea provides high-level wrapper functions for adapter invocation:

pip install mellea

Notebook Dependencies

For running Jupyter notebooks:

pip install jupyter chromadb tqdm httpx python-dotenv

Model Access

Base Models

Available on HuggingFace Hub:

Model Size Use Case
ibm-granite/granite-4.0-micro 3B Quick demos, testing
ibm-granite/granite-4.1-3b 3B Development, single GPU
ibm-granite/granite-4.1-8b 8B Production, single GPU
ibm-granite/granite-4.1-30b 30B Production, multi-GPU

Adapter Libraries

Official IBM Granite adapter libraries (r1.0):

Library Adapters Purpose
ibm-granite/granitelib-rag-r1.0 5 RAG adapters (rewrite, answerability, citations, etc.)
ibm-granite/granitelib-core-r1.0 3 Core adapters (certainty, requirements, attributions)
ibm-granite/granitelib-guardian-r1.0 4 Guardian adapters (harm check, policy, factuality, etc.)

HuggingFace Authentication

For accessing gated models:

# Interactive login
huggingface-cli login

# Or set token directly
export HF_TOKEN=your_token_here

Starting a vLLM Server

For tutorials using Mellea or the vLLM backend:

# Single GPU
python -m vllm.entrypoints.openai.api_server \
    --model <path-or-hf-repo> \
    --port 8000 \
    --host 0.0.0.0

# Multi-GPU (tensor parallelism)
python -m vllm.entrypoints.openai.api_server \
    --model <path-or-hf-repo> \
    --port 8000 \
    --host 0.0.0.0 \
    --tensor-parallel-size 2

Verify the server is running:

curl http://localhost:8000/v1/models

External Resources

Resource URL Description
Mellea https://github.com/generative-computing/mellea Framework for adapter invocation
Granite Models https://huggingface.co/ibm-granite Official IBM Granite models
Granite Libraries https://huggingface.co/collections/ibm-granite/granite-libraries Adapter libraries collection
vLLM Docs https://docs.vllm.ai/ vLLM documentation

Next Steps

Once prerequisites are installed, proceed to:

  1. Hello Mellea - Mellea adapters with vLLM (recommended start)
  2. Hello Adapter - Minimal HuggingFace example
  3. Learning Paths - Choose your path based on use case