Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions .github/workflows/deploy.yml
Original file line number Diff line number Diff line change
Expand Up @@ -440,6 +440,16 @@ jobs:
assert "News Sentry" in public_app_html or "NewsSentry" in public_app_html
PY

- name: Live quality gate with HEAD probe
run: |
set -euo pipefail
python tools/cloudflare_live_quality_check.py \
--base-url "${BASE_URL}" \
--api-url "${API_URL}" \
--min-featured 100 \
--max-latest-age-hours 24 \
--min-summary-ready 500

- name: Verify SEO/GEO public site
run: |
set -euo pipefail
Expand Down
106 changes: 106 additions & 0 deletions .github/workflows/public-translation-backfill.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,106 @@
name: Public Translation Backfill

on:
workflow_dispatch:
inputs:
execute:
description: "Execute generated SQL against remote D1"
required: true
default: "false"
type: choice
options: ["false", "true"]
targets:
description: "Comma-separated target priority list"
required: false
default: "france,south-korea,india,japan,germany,italy,canada,united-kingdom,vietnam,new-zealand,ireland,china-watch-en"
limit:
description: "Maximum patches in this run"
required: false
default: "200"
daily_limit:
description: "Maximum patches allowed for the day"
required: false
default: "1000"

permissions:
contents: read

jobs:
public-translation-backfill:
runs-on: ubuntu-latest
timeout-minutes: 30
env:
CLOUDFLARE_API_TOKEN: ${{ secrets.CLOUDFLARE_API_TOKEN }}
CLOUDFLARE_ACCOUNT_ID: ${{ secrets.CLOUDFLARE_ACCOUNT_ID }}
GEMINI_API_KEY: ${{ secrets.GEMINI_API_KEY }}
OPENROUTER_API_KEY: ${{ secrets.OPENROUTER_API_KEY }}
OPENROUTER_API_KEY_2: ${{ secrets.OPENROUTER_API_KEY_2 }}
NVIDIA_API_KEY: ${{ secrets.NVIDIA_API_KEY }}
NVIDIA_API_KEY_2: ${{ secrets.NVIDIA_API_KEY_2 }}
OPENCODE_API_KEY: ${{ secrets.OPENCODE_API_KEY }}
OPENCODE_API_KEY_2: ${{ secrets.OPENCODE_API_KEY_2 }}
REKA_API_KEY: ${{ secrets.REKA_API_KEY }}
AGNES_API_KEY: ${{ secrets.AGNES_API_KEY }}
AGNES_API_KEY_2: ${{ secrets.AGNES_API_KEY_2 }}
DEEPSEEK_API_KEY: ${{ secrets.DEEPSEEK_API_KEY }}
GROQ_API_KEY: ${{ secrets.GROQ_API_KEY }}

steps:
- uses: actions/checkout@v4

- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: "3.12"

- name: Set up Node
uses: actions/setup-node@v4
with:
node-version: "22"
cache: npm
cache-dependency-path: frontend/cloudflare/package-lock.json

- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install -e ".[api]"
cd frontend/cloudflare
npm ci

- name: Plan public translation backfill
run: |
set -euo pipefail
python tools/cloudflare_d1_public_translation_backfill.py \
--targets "${{ github.event.inputs.targets }}" \
--limit "${{ github.event.inputs.limit }}" \
--daily-limit "${{ github.event.inputs.daily_limit }}" \
--output-sql /tmp/news-sentry-public-translation-backfill.sql \
--generate-missing \
--dry-run

- name: Generate public translation D1 SQL
if: github.event.inputs.execute == 'true'
run: |
set -euo pipefail
python tools/cloudflare_d1_public_translation_backfill.py \
--targets "${{ github.event.inputs.targets }}" \
--limit "${{ github.event.inputs.limit }}" \
--daily-limit "${{ github.event.inputs.daily_limit }}" \
--output-sql /tmp/news-sentry-public-translation-backfill.sql \
--generate-missing \
--transaction
if grep -q '^-- no public translation patches ready' /tmp/news-sentry-public-translation-backfill.sql; then
echo "provider_quota_exhausted_or_no_ready_patches=true"
exit 1
fi

- name: Execute public translation D1 SQL
if: github.event.inputs.execute == 'true'
working-directory: frontend/cloudflare
run: npx wrangler d1 execute ns-db --remote --file /tmp/news-sentry-public-translation-backfill.sql

- name: Dry-run receipt
if: github.event.inputs.execute != 'true'
run: |
echo "execute=false; generated SQL only"
head -20 /tmp/news-sentry-public-translation-backfill.sql
6 changes: 3 additions & 3 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ flowchart TB
end

subgraph ADAPTERS["外部适配"]
AP["AI Providers<br/>Gemini → DeepSeek → Groq → Cloudflare"]
AP["AI Providers<br/>Gemini → DeepSeek → Groq → Cloudflare → OpenRouter → NVIDIA/Agnes/OpenCode/Reka"]
RT["Runtime Adapters<br/>RSS-Bridge / API"]
end

Expand Down Expand Up @@ -173,7 +173,7 @@ flowchart TD
| 规则 | 内容 | 参考 |
|------|------|------|
| **前端策略** | CLI-first,FastAPI + Vanilla JS 可选,无重型框架 | ADR-0025(替代 ADR-0010) |
| **AI Provider** | 内置 chain: Gemini → DeepSeek → Groq → Cloudflare Workers AI | ADR-0005 |
| **AI Provider** | 内置 chain: Gemini → DeepSeek → Groq → Cloudflare Workers AI → OpenRouter → NVIDIA/Agnes/OpenCode/Reka | ADR-0005 |
| **分类存储** | L0-L3 走 `metadata.classification`,不做顶层字段 | ADR-0009 |
| **实现语言** | Python 3.11+ / Pydantic v2 | ADR-0012 |
| **配置管理** | 所有国家参数入 config/,禁止硬编码到 src/ | ADR-0015 |
Expand Down Expand Up @@ -371,6 +371,6 @@ flowchart LR
- **测试规模**:2,738 tests, 85% 覆盖率, ruff=0, mypy=0, frontend=0
- **监控目标**:81 targets (italy, china-watch-en, japan, germany, france + 76 更多)
- **信源规模**:244 源 (147 RSS + 97 API),覆盖 81 个 target
- **AI Provider**:内置 chain: Gemini → DeepSeek → Groq → Cloudflare Workers AI
- **AI Provider**:内置 chain: Gemini → DeepSeek → Groq → Cloudflare Workers AI → OpenRouter → NVIDIA/Agnes/OpenCode/Reka
- **部署方式**:Cloudflare Pages + Workers + D1/R2;Cloudflare Containers 承接过渡期 Python/RSS-Bridge 后台面;VPS/Tunnel 仅作 legacy rollback,不是运行依赖
- **可选组件**:`[api]` FastAPI + Web UI(管理后台 + 公开新闻阅读器)
116 changes: 106 additions & 10 deletions config/provider/routes.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
# ADR: ADR-0005
#
# AI Provider 路由配置 — 内置多 Provider 链式兜底版
# 所有 AI 能力由内置 Provider 直接调用:gemini → deepseek → groq → cloudflare_workers_ai → local。
# 所有 AI 能力由内置 Provider 直接调用:gemini → deepseek → groq → cloudflare_workers_ai → openrouter → local。
# 各 Provider 通过自身的 API key 环境变量鉴权,不再经过中间层 sidecar。
#
routes_version: "2.0.0"
Expand Down Expand Up @@ -107,9 +107,21 @@ routes:
output_schema_ref: null
audit: true
fallback_route_ids:
- fallback.local
- translate.openrouter
notes: "翻译兜底链:Groq 失败后由 Cloudflare Workers AI 接管"

- route_id: translate.openrouter
task_type: translate
provider: openrouter
model: google/gemma-4-26b-a4b-it:free
timeout_seconds: 90
max_cost_usd_per_call: 0.0
output_schema_ref: null
audit: true
fallback_route_ids:
- translate.nvidia
notes: "翻译兜底链:Cloudflare Workers AI 失败后由 OpenRouter 免费模型接管"

- route_id: translate.batch
task_type: translate
provider: gemini
Expand All @@ -124,15 +136,51 @@ routes:

- route_id: translate.nvidia
task_type: translate
provider: gemini
model: gemini-2.5-flash
provider: nvidia
model: deepseek-ai/deepseek-v4-flash
timeout_seconds: 90
max_cost_usd_per_call: 0.0
output_schema_ref: null
audit: true
fallback_route_ids:
- translate.deepseek
notes: "历史 route_id 兼容:当前指向 Gemini"
- translate.agnes
notes: "FreeLLMAPI 迁移后直连 NVIDIA NIM,不再经过本地 sidecar"

- route_id: translate.agnes
task_type: translate
provider: agnes
model: agnes-2.0-flash
timeout_seconds: 90
max_cost_usd_per_call: 0.0
output_schema_ref: null
audit: true
fallback_route_ids:
- translate.opencode
notes: "FreeLLMAPI 迁移后直连 Agnes AI"

- route_id: translate.opencode
task_type: translate
provider: opencode
model: deepseek-v4-flash-free
timeout_seconds: 90
max_cost_usd_per_call: 0.0
output_schema_ref: null
audit: true
fallback_route_ids:
- translate.reka
notes: "FreeLLMAPI 迁移后直连 OpenCode Zen"

- route_id: translate.reka
task_type: translate
provider: reka
model: reka-flash-3
timeout_seconds: 90
max_cost_usd_per_call: 0.0
output_schema_ref: null
audit: true
fallback_route_ids:
- fallback.local
notes: "FreeLLMAPI 迁移后直连 Reka"

# ── 聚类 / 复核 / 增强 ────────────────────────────────

Expand Down Expand Up @@ -193,20 +241,68 @@ routes:
output_schema_ref: null
audit: true
fallback_route_ids:
- translate.deepseek
- ai.enrichment.openrouter
notes: "低频 AI 增强合并任务:翻译、摘要、聚类、复核 — Gemini 主路由"

- route_id: ai.enrichment.openrouter
task_type: ai_enrichment
provider: openrouter
model: google/gemma-4-26b-a4b-it:free
timeout_seconds: 120
max_cost_usd_per_call: 0.0
output_schema_ref: null
audit: true
fallback_route_ids:
- ai.enrichment.nvidia
notes: "低频 AI 增强兜底:Gemini 不可用时由 OpenRouter 免费模型接管"

- route_id: ai.enrichment.nvidia
task_type: ai_enrichment
provider: deepseek
model: deepseek-chat
provider: nvidia
model: deepseek-ai/deepseek-v4-flash
timeout_seconds: 120
max_cost_usd_per_call: 0.0
output_schema_ref: null
audit: true
fallback_route_ids:
- ai.enrichment.agnes
notes: "低频 AI 增强兜底:NVIDIA NIM 直连"

- route_id: ai.enrichment.agnes
task_type: ai_enrichment
provider: agnes
model: agnes-2.0-flash
timeout_seconds: 120
max_cost_usd_per_call: 0.0
output_schema_ref: null
audit: true
fallback_route_ids:
- ai.enrichment.opencode
notes: "低频 AI 增强兜底:Agnes AI 直连"

- route_id: ai.enrichment.opencode
task_type: ai_enrichment
provider: opencode
model: deepseek-v4-flash-free
timeout_seconds: 120
max_cost_usd_per_call: 0.0
output_schema_ref: null
audit: true
fallback_route_ids:
- ai.enrichment.reka
notes: "低频 AI 增强兜底:OpenCode Zen 直连"

- route_id: ai.enrichment.reka
task_type: ai_enrichment
provider: reka
model: reka-flash-3
timeout_seconds: 120
max_cost_usd_per_call: 0.0
output_schema_ref: null
audit: true
fallback_route_ids:
- fallback.local
notes: "AI 增强兜底:DeepSeek 接管,失败回退本地规则"
notes: "低频 AI 增强兜底:Reka 直连"

# ── 判断任务 ──────────────────────────────────────────

Expand Down
23 changes: 18 additions & 5 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ News Sentry 是一个**开源 AI 新闻情报与 OSINT 监控平台**,持续
| 类型检查 | mypy strict, 零错误 |
| Web 框架 | FastAPI + Uvicorn |
| 存储 | Cloudflare D1 + R2;SQLite/文件系统用于本地开发与容器过渡 |
| AI Provider | Gemini → DeepSeek → Groq → Cloudflare (链式降级) |
| AI Provider | Gemini → DeepSeek → Groq → Cloudflare Workers AI → OpenRouter → NVIDIA/Agnes/OpenCode/Reka (链式降级) |
| 部署 | Cloudflare Pages + Workers + D1/R2;Cloudflare Containers 作为 Python/RSS-Bridge 过渡运行面 |
| 前端 | Vite + React + Tailwind CSS |
| 测试 | pytest 3,013 tests / vitest |
Expand All @@ -43,7 +43,7 @@ News Sentry 是一个**开源 AI 新闻情报与 OSINT 监控平台**,持续
NewsSentry/
├── src/news_sentry/ # 核心 Python 源码 (~24,000 lines, 102 files)
│ ├── adapters/ # 外部适配器
│ │ ├── providers/ # AI Provider (12 个 Provider)
│ │ ├── providers/ # AI Provider adapters
│ │ │ ├── base.py # AIProvider Protocol
│ │ │ ├── openai_provider.py # OpenAI-compatible 基类
│ │ │ ├── gemini_provider.py # Gemini (主)
Expand All @@ -52,6 +52,7 @@ NewsSentry/
│ │ │ ├── cloudflare_workers_ai_provider.py # CF (兜底翻译)
│ │ │ ├── anthropic_provider.py # Anthropic (可选)
│ │ │ ├── openrouter_provider.py # OpenRouter (可选)
│ │ │ ├── freeapi_compat_providers.py # NVIDIA/OpenCode/Reka/Agnes 直连备用
│ │ │ ├── rules_provider.py # 本地规则引擎
│ │ │ ├── libretranslate_provider.py
│ │ │ └── mymemory_provider.py
Expand Down Expand Up @@ -140,6 +141,8 @@ flowchart TB
subgraph PROVIDERS["AI Provider Chain"]
direction LR
G["Gemini"] --> DS["DeepSeek"] --> GR["Groq"] --> CF["Cloudflare"]
CF --> OR["OpenRouter"] --> NV["NVIDIA"] --> AG["Agnes"]
AG --> OC["OpenCode"] --> RK["Reka"]
end

subgraph STORAGE["文件存储"]
Expand Down Expand Up @@ -207,7 +210,7 @@ flowchart LR
### Provider Chain

```
Gemini (首选) → DeepSeek (备选 1) → Groq (备选 2) → Cloudflare Workers AI (兜底翻译)
Gemini (首选) → DeepSeek (备选 1) → Groq (备选 2) → Cloudflare Workers AI → OpenRouter → NVIDIA → Agnes → OpenCode → Reka → local
↓ 失败 ↓ 失败 ↓ 失败
自动降级到下一个节点,无需人工干预
```
Expand All @@ -220,7 +223,12 @@ AIProvider (Protocol)
│ ├── GeminiProvider (api_key: GEMINI_API_KEY)
│ ├── DeepSeekProvider (api_key: DEEPSEEK_API_KEY)
│ ├── GroqProvider (api_key: GROQ_API_KEY)
│ └── OpenRouterProvider
│ ├── OpenRouterProvider
│ └── FreeAPI migrated providers
│ ├── NvidiaProvider (api_key: NVIDIA_API_KEY[_2])
│ ├── AgnesProvider (api_key: AGNES_API_KEY[_2])
│ ├── OpenCodeProvider (api_key: OPENCODE_API_KEY[_2])
│ └── RekaProvider (api_key: REKA_API_KEY)
├── AnthropicProvider — Anthropic Messages API
├── CloudflareWorkersAIProvider — 翻译专用
├── LibreTranslateProvider / MyMemoryProvider — 翻译备选
Expand Down Expand Up @@ -253,6 +261,11 @@ routes:
| `DEEPSEEK_API_KEY` | DeepSeek | platform.deepseek.com |
| `GROQ_API_KEY` | Groq | console.groq.com |
| `CLOUDFLARE_ACCOUNT_ID` + `CLOUDFLARE_API_TOKEN` | Cloudflare | dash.cloudflare.com |
| `OPENROUTER_API_KEY[_2]` | OpenRouter | openrouter.ai |
| `NVIDIA_API_KEY[_2]` | NVIDIA NIM | build.nvidia.com |
| `AGNES_API_KEY[_2]` | Agnes AI | platform.agnes-ai.com |
| `OPENCODE_API_KEY[_2]` | OpenCode Zen | opencode.ai |
| `REKA_API_KEY` | Reka | platform.reka.ai |

---

Expand Down Expand Up @@ -288,7 +301,7 @@ flowchart TB
QUEUE --> FLOW["Workflows / Durable Objects"]
FLOW --> WORKER
WORKER -. "Access-gated fallback only" .-> C["Cloudflare Containers<br/>Python / FastAPI / RSS-Bridge transition"]
WORKER -.-> API["AI APIs<br/>(Gemini/DeepSeek/Groq/Workers AI)"]
WORKER -.-> API["AI APIs<br/>(Gemini/DeepSeek/Groq/Workers AI/OpenRouter/NVIDIA/Agnes/OpenCode/Reka)"]
```

### 运行面
Expand Down
Loading
Loading