llmcheck

Universal health checker & benchmark for OpenAI-compatible LLM endpoints.

llmcheck lets you quickly test, benchmark, and compare any OpenAI-compatible chat-completions endpoint — OpenAI, Anthropic, OpenRouter, Groq, NVIDIA NIM, Xiaomi MiMo, DeepSeek, local Ollama, LM Studio, or your own router. One command per provider, real timing, real tokens-per-second.

pip install llmcheck

# Quick health check
export OPENAI_API_KEY=sk-...
llmcheck check --profile openai -m gpt-4o-mini

# Compare 3 providers on the same prompt
llmcheck compare openai/gpt-4o-mini groq/llama-3.3-70b-versatile nvidia/meta/llama-3.3-70b-instruct

Why?

If you build with multiple LLM providers (you probably do), you've hit this:

• "Is the endpoint up?"
• "Which provider gives me the best tokens/sec right now?"
• "Is streaming working on this proxy?"
• "Is my fallback actually faster than the primary?"

llmcheck answers all of those in seconds, without you writing curl one-liners or boilerplate.

Features

• ✅ Single check — verify endpoint + model is healthy, with TTFB, total latency, tokens/sec
• 📊 Benchmark — N iterations with concurrency, p50/p95 stats
• 🆚 Compare — side-by-side report across multiple providers
• 🌊 Streaming-aware — measures real time-to-first-token, not just full-response time
• 🔌 Built-in profiles — OpenAI, Anthropic, OpenRouter, Groq, NVIDIA, MiMo, DeepSeek, Ollama, LM Studio, 9router, more
• 🔐 Secret-safe — keys come from env vars by default; never echoed
• 📦 Multiple outputs — pretty terminal table, JSON, markdown (for CI / dashboards)

Install

pip install llmcheck

Or from source:

git clone https://github.com/sherrie-ai/llmcheck
cd llmcheck
pip install -e .

Usage

llmcheck check — one-shot health check

# With a built-in profile
llmcheck check --profile openai -m gpt-4o-mini

# With a custom endpoint
llmcheck check \
  --base-url https://integrate.api.nvidia.com/v1 \
  --api-key-env NVIDIA_API_KEY \
  --model meta/llama-3.3-70b-instruct

# Streaming check (gives true TTFB)
llmcheck check --profile openai -m gpt-4o-mini --stream

llmcheck benchmark — run N iterations and get p50/p95

llmcheck benchmark --profile groq -m llama-3.3-70b-versatile -n 10 -c 3

llmcheck compare — side-by-side across providers

llmcheck compare \
  openai/gpt-4o-mini \
  groq/llama-3.3-70b-versatile \
  nvidia/meta/llama-3.3-70b-instruct \
  9router/Kiro \
  --prompt "Write a haiku about networking." \
  --max-tokens 64 \
  --stream

Output:

                     LLM Endpoint Check
┏━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━┳━━━━━━┓
┃ Provider ┃ Model                   ┃ Status ┃   TTFB  ┃   Total ┃   t/s   ┃ Note ┃
┡━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━╇━━━━━━┩
│ openai   │ gpt-4o-mini             │  ✓ OK  │   234ms │   812ms │  78.5/s │ ...  │
│ groq     │ llama-3.3-70b-versatile │  ✓ OK  │   119ms │   402ms │ 285.0/s │ ...  │
│ nvidia   │ meta/llama-3.3-70b-...  │  ✓ OK  │   312ms │   908ms │  74.2/s │ ...  │
│ 9router  │ Kiro                    │  ✓ OK  │   289ms │   721ms │  82.1/s │ ...  │
└──────────┴─────────────────────────┴────────┴─────────┴─────────┴─────────┴──────┘

llmcheck profiles list — see what's available

llmcheck profiles list

llmcheck profiles add — save your own

llmcheck profiles add my-router \
  --base-url http://localhost:20128/v1 \
  --default-model Kiro \
  --api-key-env ROUTER_API_KEY

Built-in profiles

Profile	Endpoint	Default Env Var
openai	api.openai.com/v1	OPENAI_API_KEY
anthropic	api.anthropic.com/v1	ANTHROPIC_API_KEY
openrouter	openrouter.ai/api/v1	OPENROUTER_API_KEY
groq	api.groq.com/openai/v1	GROQ_API_KEY
nvidia	integrate.api.nvidia.com/v1	NVIDIA_API_KEY
deepseek	api.deepseek.com/v1	DEEPSEEK_API_KEY
mimo	api.mimo.xiaomi.com/v1	MIMO_API_KEY
9router	localhost:20128/v1	ROUTER_API_KEY
ollama	localhost:11434/v1	(none — uses ollama)
lmstudio	localhost:1234/v1	(none — uses lm-studio)

Output formats

llmcheck check --profile openai -m gpt-4o-mini --format json | jq
llmcheck compare ... --format markdown -o report.md

JSON output is structured and stable, suitable for CI pipelines and dashboards.

CI usage example

# .github/workflows/llm-health.yml
- name: Check LLM endpoints
  env:
    OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
    GROQ_API_KEY: ${{ secrets.GROQ_API_KEY }}
  run: |
    pip install llmcheck
    llmcheck compare openai/gpt-4o-mini groq/llama-3.3-70b-versatile \
      --format markdown -o $GITHUB_STEP_SUMMARY

Library use

import asyncio
from llmcheck import check_endpoint

async def main():
    result = await check_endpoint(
        base_url="https://api.openai.com/v1",
        api_key="sk-...",
        model="gpt-4o-mini",
        provider="openai",
        stream=True,
    )
    print(result.summary())

asyncio.run(main())

License

MIT

Status

llmcheck is in early beta. API is stable for the documented commands; internals may move. Issues and PRs welcome.