Skip to content
BranchesTags

Latest commit

 

History

1 Commit

Folders and files

NameName
Last commit message
Last commit date
bin
bin
 
 
config
config
 
 
src
src
 
 
.gitignore
.gitignore
 
 
LICENSE
LICENSE
 
 
README.md
README.md
 
 
pyproject.toml
pyproject.toml
 
 
requirements.txt
requirements.txt
 
 

Repository files navigation

🐾 Supplier Sniffer — 供应商嗅探器

Predictive supplier availability monitoring & auto-failover for LLM API gateways. 面向 LLM API 网关的预测性供应商可用性监控与自动故障转移系统。

License: MIT

📐 Architecture / 架构

┌─────────────────────────────────────────────────────────────────┐
│                        CRON (every N s)                         │
└──────────────────────────┬──────────────────────────────────────┘
                           │
                           ▼
┌─────────────────────────────────────────────────────────────────┐
│  supplier-sniffer.sh  (Main Orchestrator / 主编排器)              │
│                                                                  │
│  ┌─────────────────────┐    ┌──────────────────────────────┐    │
│  │ Phase 1: Metrics    │    │ supplier-adapter.py           │    │
│  │  采集多维指标        │───▶│   Probe / List / Detect      │    │
│  │  • latency / ttfb   │    │   (curl + timing + TLS)      │    │
│  │  • model_count       │    └──────────────────────────────┘    │
│  │  • http_code         │                                        │
│  │  • dns_ms / tls_left │                                        │
│  └──────────┬───────────┘                                        │
│             ▼                                                    │
│  ┌──────────────────────────────────────────────────────────┐   │
│  │ Phase 2: Legacy Window (simple sliding window 滑动窗口)    │   │
│  │ window.json  →  decision: UP / DEGRADED / DOWN           │   │
│  └──────────────────────────┬───────────────────────────────┘   │
│                             ▼                                    │
│  ┌──────────────────────────────────────────────────────────┐   │
│  │ Phase 3: Predictive Analysis (趋势预测分析)                │   │
│  │ supplier-predictor.py  →  timeseries.json                 │   │
│  │  • latency_slope  • model_attrition  • error_diversity    │   │
│  │  • payload_shrink • tls_expiry       • dns_jitter         │   │
│  │  ──► Verdict: GREEN / YELLOW / ORANGE / RED              │   │
│  └──────────────────────────┬───────────────────────────────┘   │
│                             ▼                                    │
│  ┌──────────────────────────────────────────────────────────┐   │
│  │ Phase 4: Response (响应)                                  │   │
│  │ supplier-sniffer-helper.py                                │   │
│  │  • RED    → predictive_pause (pre-emptive failover)       │   │
│  │  • ORANGE → set_frequency 30s (intensify probes)          │   │
│  │  • YELLOW → monitor only (仅监控)                         │   │
│  │  • GREEN  → predictive_resume (restore normal)            │   │
│  └──────────────────────────────────────────────────────────┘   │
│                                                                  │
│  ┌──────────────────────────────────────────────────────────┐   │
│  │ Bottom Floor: Hard-failure (硬故障兜底)                   │   │
│  │ window threshold ≥ 4 failures  →  .SUPPLIER_DOWN flag    │   │
│  │  → pause gateway + cron                                   │   │
│  └──────────────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────────────┘

🚀 Quick Start / 快速开始

# 1. Clone / navigate to project
cd /tmp/supplier-sniffer

# 2. Copy example configs
mkdir -p ~/.hermes/.supplier_sniffer
cp config/suppliers.json.example ~/.hermes/.supplier_sniffer/suppliers.json
cp config/predictive_config.json.example ~/.hermes/.supplier_sniffer/predictive_config.json

# 3. Set your API keys (in ~/.hermes/.env or export directly)
export INFERAICHAT_API_KEY="sk-xxx"
export OPENAI_API_KEY="sk-xxx"

# 4. Test a probe
python3 src/supplier-adapter.py list
python3 src/supplier-adapter.py inferaichat

# 5. Run a full monitoring cycle
bash bin/supplier-sniffer.sh

# 6. Set up cron (every 2 minutes)
crontab -e
*/2 * * * * /tmp/supplier-sniffer/bin/supplier-sniffer.sh

⚙️ Configuration Guide / 配置指南

suppliers.json — Supplier definitions / 供应商定义

{
  "openai": {
    "name": "OpenAI",
    "base_url": "https://api.openai.com",
    "models_endpoint": "/v1/models",
    "auth": {
      "type": "bearer",
      "env_var": "OPENAI_API_KEY"
    },
    "response": {
      "data_keys": ["id"],
      "container_keys": ["data"]
    },
    "rate_limit_code": 429,
    "timeout": 30,
    "tls_domain": "api.openai.com"
  }
}
Field Description / 说明
base_url Root URL / 根地址
models_endpoint Path to list models / 模型列表路径
auth.type bearer / header / basic / query
auth.env_var Env var name holding the key / 密钥环境变量名
response.container_keys Keys containing model arrays / 承载模型数组的字段
tls_domain Domain for TLS cert check / TLS 证书检查域名

predictive_config.json — Predictive analysis thresholds / 预测分析阈值

{
  "windows": { "short": 6, "medium": 15, "long": 30 },
  "thresholds": {
    "latency_slope_ms_per_tick": 50,
    "model_attrition_count": 1,
    "response_size_shrink_pct": 20,
    "error_diversity_min_codes": 2,
    "dns_spike_ratio": 2.0,
    "tls_days_min": 7
  },
  "weights": {
    "latency_trend": 25,
    "model_attrition": 30,
    "error_diversity": 20,
    "payload_shrink": 15,
    "tls_expiry": 25,
    "dns_jitter": 10
  },
  "levels": { "GREEN": 0, "YELLOW": 50, "ORANGE": 70, "RED": 85 }
}

.env — Environment file / 环境变量文件

Stored at ~/.hermes/.env (referenced by all components):

INFERAICHAT_API_KEY='sk-xxx'
OPENAI_API_KEY='sk-xxx'
ANTHROPIC_API_KEY='sk-ant-xxx'
GOOGLE_API_KEY='AIzaXXX'

⌨️ CLI Usage / 命令行使用

supplier-adapter.py — Adapter commands

# List all configured suppliers / 列出所有配置的供应商
python3 src/supplier-adapter.py list

# Probe a specific supplier / 探测指定供应商
python3 src/supplier-adapter.py openai
# → {"supplier":"openai","reachable":true,"latency_ms":342,"model_count":286,...}

# Auto-detect an unknown provider / 自动探测未配置的供应商
python3 src/supplier-adapter.py detect https://custom-api.example.com CUSTOM_API_KEY

supplier-sniffer-helper.py — Failover management

# Status check / 状态查询
python3 src/supplier-sniffer-helper.py status

# Hard-failure pause / 硬故障暂停 (gateway + cron)
python3 src/supplier-sniffer-helper.py pause

# Resume after recovery / 恢复后解除暂停
python3 src/supplier-sniffer-helper.py resume

# Predictive pause / 预测性暂停
python3 src/supplier-sniffer-helper.py predictive_pause
python3 src/supplier-sniffer-helper.py predictive_resume

# Set adaptive probe frequency / 设置自适应探测频率
python3 src/supplier-sniffer-helper.py set_frequency 30

Cron monitoring / 定时监控

# Direct execution — log output → sniffer.log
bash bin/supplier-sniffer.sh

# Cron (every 2 minutes)
*/2 * * * * /tmp/supplier-sniffer/bin/supplier-sniffer.sh

Logs: ~/.hermes/.supplier_sniffer/sniffer.log

🌐 Environment Variables / 环境变量表

Variable / 变量 Default / 默认值 Description / 说明
SNIFFER_DIR ~/.hermes/.supplier_sniffer State directory (timeseries, flags, alerts) / 状态目录
SNIFFER_HOME ~/.hermes Hermes home directory / Hermes 主目录
SNIFFER_CONFIG $SNIFFER_HOME/config.yaml Path to gateway config.yaml / 网关配置路径
INFERAICHAT_API_KEY — API key for InferAI Chat
OPENAI_API_KEY — API key for OpenAI
ANTHROPIC_API_KEY — API key for Anthropic
GOOGLE_API_KEY — API key for Google Gemini

All API keys are loaded from ~/.hermes/.env by default, or directly from the environment.

📊 Predictive Analysis Levels / 预测分析等级

Level / 等级 Confidence / 置信度 Action / 行为
🟢 GREEN 0–49 Normal — clear any predictive pause, reset frequency to 120s / 正常 — 清除预测暂停，重置探测频率
🟡 YELLOW 50–69 Monitor — log signals, no action / 监控 — 记录信号，不操作
🟠 ORANGE 70–84 Intensify — increase probe frequency to 30s / 加强 — 探测频率提升至 30s
🔴 RED 85–100 Pre-emptive pause — pause gateway + cron before hard failure / 预防性暂停 — 在硬故障前暂停网关和定时任务

Six signal dimensions / 六个信号维度

Signal / 信号 Weight Trigger / 触发条件
Latency trend / 延迟趋势 25 Slope > 50ms/tick
Model attrition / 模型减少 30 Count drops ≥ 1
Error diversity / 错误多样性 20 ≥ 2 distinct 5xx codes
Payload shrink / 响应缩小 15 Size drops > 20%
TLS expiry / 证书到期 25 Days left < 7
DNS jitter / DNS抖动 10 Latest/baseline ratio > 2.0

🔁 Auto-Failover / 自动故障转移

Two-layer failover architecture / 双层故障转移架构：

Layer / 层级 Mechanism / 机制 Response Time / 响应时间
Predictive (预测层) Trend analysis → ORANGE/RED → pre-emptive pause ~30s–2min (before failure)
Hard-failure (硬故障层) Sliding window (4/6 failures) → .SUPPLIER_DOWN flag ~2–6min (after failure)

What happens during failover / 故障转移过程：

  1. Gateway config.yaml → api_key set to "paused-by-sniffer" (sentinel value)
  2. All cron jobs paused (idempotent, marked as supplier_outage)
  3. .SUPPLIER_DOWN / .SUPPLIER_PREDICTIVE flag written
  4. On recovery → gateway restored to real API key, cron resumed, flags cleared

Priority / 优先级： Hard-failure flag (.SUPPLIER_DOWN) > Predictive flag (.SUPPLIER_PREDICTIVE)

📁 File Layout / 文件结构

/tmp/supplier-sniffer/
├── bin/
│   └── supplier-sniffer.sh        # Main cron orchestrator
├── src/
│   ├── supplier-adapter.py        # Probe / list / detect CLI
│   ├── supplier-sniffer-helper.py # Pause/resume gateway & cron
│   └── supplier-predictor.py      # Trend analysis → verdict
├── config/
│   ├── suppliers.json.example     # Supplier definitions template
│   └── predictive_config.json.example # Predictive config template
└── README.md                      # This file

~/.hermes/.supplier_sniffer/       # Runtime state (SNIFFER_DIR)
├── suppliers.json                 # Active supplier config
├── predictive_config.json         # Active predictive config
├── timeseries.json                # Time series data (max 120 points)
├── window.json                    # Sliding window (6 points)
├── sniffer.log                    # Operation log
├── .ADAPTER_ALERT                 # Adapter failure alert
├── .ADAPTIVE_FREQUENCY            # Adaptive probe interval
└── .tls_cache/                    # Cached TLS expiry values

📄 License / 许可证

MIT License — feel free to use, modify, and distribute.

See LICENSE for full text.

About

AI supplier availability sniffer — multi-vendor health monitoring with predictive trend analysis

Resources

Readme
MIT license
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages