feat(ad-diagnosis): add storage layer, export tools, credibility CIs, and diagnostic charts

Storage abstraction (src/core/storage/):
- Strategy Pattern: S3CompatibleBackend (R2/S3/MinIO) and LocalHTTPBackend
- CLOUDFLARE_R2_ACCOUNT_ID auto-builds R2 endpoint URL
- Switchable via STORAGE_BACKEND env var, no caller changes needed

Output tools (src/mcp/servers/output/tools/):
- export_html: markdown to styled HTML with image upload via storage backend
- export_csv: real DictWriter implementation with storage upload fallback

Ad diagnosis credibility dimensions:
- Statistical sufficiency flag: orders_reliability (low/medium/high)
- ACOS 95% CI via Wilson score interval on CVR
- ITS/DML/CausalImpact effect confidence intervals
- Within-sample directional backtest (hit rate vs delta_orders)
- LP keyword|MATCH_TYPE output bug fixed

Diagnostic charts (6 per ASIN, uploaded to R2, embedded in report):
- Daily performance trend: spend bar + orders line + ACOS with CI band
- ITS causal fit: pre-trend, counterfactual, shaded effect area, CI title
- Keyword ACOS x Orders quadrant scatter (bubble = spend)
- Placement ACOS vs configured bid adjustment grouped bar
- Inventory burn-down curve with stockout date and risk zone
- Competitor price distribution box-plot with own price line

Docs + deploy: README .env table updated, ARCHITECTURE/DEV_GUIDE storage
section added, deploy_ubuntu.sh .env template extended with full storage block.
Ignore src/core/telemetry/step_history.json (runtime-generated).

Author: YoTro

AWS/.gitignore

@@ -78,3 +78,4 @@ output/*.csv
 output/*.json
 tests/*.csv
 tests/test_input.csv
+src/core/telemetry/step_history.json

AWS/README.md

@@ -56,63 +56,111 @@ It employs a **Hybrid Intelligence** model:
 
    ```dotenv
    # ── LLM Providers ──────────────────────────────────────────
-   GEMINI_API_KEY=your_gemini_api_key_here
-   ANTHROPIC_API_KEY=your_anthropic_api_key_here   # Required only when using Claude
-
-   # Default provider: "gemini" | "claude"
    DEFAULT_LLM_PROVIDER=gemini
+   GEMINI_API_KEY=
+   ANTHROPIC_API_KEY=
+   DEEPSEEK_API_KEY=
+   LOCAL_MODEL_PATH=models/llm/qwen2.5-3b-instruct-q4_k_m.gguf
+   MAX_LLM_OUTPUT_TOKENS=
 
-   # Path to local GGUF model file (optional, for offline mode)
-   LOCAL_MODEL_PATH=models/llm/your_model.gguf
+   # ── Amazon Ads API ─────────────────────────────────────────
+   AMAZON_ADS_DEFAULT_STORE=US
+   AMAZON_ADS_CLIENT_ID=
+   AMAZON_ADS_CLIENT_SECRET=
+   AMAZON_ADS_REFRESH_TOKEN_US=
+   AMAZON_ADS_PROFILE_ID_US=
+   AMAZON_ADS_FALLBACK_ASIN_US=
+
+   # ── Amazon SP-API / LWA ────────────────────────────────────
+   AMAZON_LWA_CLIENT_ID=
+   AMAZON_LWA_CLIENT_SECRET=
+   AMAZON_SP_API_REFRESH_TOKEN_US=
 
    # ── Feishu / Lark Bots ─────────────────────────────────────
    # Naming rule: FEISHU_{BOT_NAME_UPPER}_{FIELD}
-   # Add as many bots as needed by following the same pattern.
 
    # amazon_bot (primary bot)
    FEISHU_AMAZON_BOT_APP_ID=
    FEISHU_AMAZON_BOT_APP_SECRET=
-   FEISHU_AMAZON_BOT_USER_ACCESS_TOKEN=   # Optional: for user-level Bitable access
-   FEISHU_AMAZON_BOT_WEBHOOK_URL=         # Optional: for incoming webhook messages
+   FEISHU_AMAZON_BOT_USER_ACCESS_TOKEN=
+   FEISHU_AMAZON_BOT_WEBHOOK_URL=
 
    # test_bot (secondary / staging bot)
    FEISHU_TEST_BOT_APP_ID=
    FEISHU_TEST_BOT_APP_SECRET=
    FEISHU_TEST_BOT_USER_ACCESS_TOKEN=
    FEISHU_TEST_BOT_WEBHOOK_URL=
 
-   # ── Third-party Services ───────────────────────────────────
+   # ── Third-party Market Data ────────────────────────────────
    SELLERSPRITE_EMAIL=
    SELLERSPRITE_PASSWORD=
-   XIYOU_PHONE=
-
-   # ── Amazon Advertising API (LWA) ───────────────────────────
-   AMAZON_ADS_CLIENT_ID=
-   AMAZON_ADS_CLIENT_SECRET=
-   AMAZON_ADS_DEFAULT_STORE=US
-   # Multiple stores supported: AMAZON_ADS_REFRESH_TOKEN_{STORE}, AMAZON_ADS_PROFILE_ID_{STORE}
-   AMAZON_ADS_REFRESH_TOKEN_US=
-   AMAZON_ADS_PROFILE_ID_US=
+   XIYOUZHAOCI_PHONE=
+   LINGXING_ACCOUNT=
+   LINGXING_PASSWORD=
 
    # ── Infrastructure ─────────────────────────────────────────
-   # Omit to use local JSON-file cache; set to enable Redis backend
    REDIS_URL=redis://localhost:6379
+   SERVER_IP=
+   SERVER_USER=
+
+   # ── Object Storage ─────────────────────────────────────────
+   # Backend: s3_compatible (R2 / S3 / MinIO)  |  local_http (VPS nginx)
+   STORAGE_BACKEND=s3_compatible
+
+   # Cloudflare R2 (recommended)
+   CLOUDFLARE_R2_ACCOUNT_ID=     # endpoint auto-built from this
+   STORAGE_ACCESS_KEY_ID=
+   STORAGE_SECRET_ACCESS_KEY=
+   STORAGE_BUCKET_NAME=
+   STORAGE_PUBLIC_URL=           # e.g. https://your-domain.com or https://pub-<hash>.r2.dev
+   STORAGE_REGION=auto
+
+   # AWS S3 (omit CLOUDFLARE_R2_ACCOUNT_ID, set real region)
+   # STORAGE_REGION=us-east-1
+
+   # MinIO / self-hosted (set explicit endpoint)
+   # STORAGE_ENDPOINT_URL=https://minio.yourdomain.com
+
+   # VPS local directory + nginx
+   # STORAGE_BACKEND=local_http
+   # STORAGE_LOCAL_DIR=/var/www/files
    ```
 
    | Variable | Required | Description |
    |---|---|---|
    | `GEMINI_API_KEY` | When using Gemini | Google AI Studio API key |
    | `ANTHROPIC_API_KEY` | When using Claude | Anthropic Console API key |
+   | `DEEPSEEK_API_KEY` | When using DeepSeek | DeepSeek API key |
    | `DEFAULT_LLM_PROVIDER` | No (default: `gemini`) | Active LLM backend |
    | `LOCAL_MODEL_PATH` | No | Path to GGUF model for offline inference |
-   | `FEISHU_*_APP_ID` | For Feishu bot | Lark Open Platform App ID |
-   | `FEISHU_*_APP_SECRET` | For Feishu bot | Lark Open Platform App Secret |
-   | `FEISHU_*_USER_ACCESS_TOKEN` | No | User-level token for Bitable write access |
-   | `FEISHU_*_WEBHOOK_URL` | No | Incoming webhook URL for the bot |
+   | `MAX_LLM_OUTPUT_TOKENS` | No | Cap on LLM response length |
    | `AMAZON_ADS_CLIENT_ID` | For Ads API | Login with Amazon Client ID |
    | `AMAZON_ADS_CLIENT_SECRET` | For Ads API | Login with Amazon Client Secret |
    | `AMAZON_ADS_REFRESH_TOKEN_*` | For Ads API | OAuth2 Refresh Token per store |
    | `AMAZON_ADS_PROFILE_ID_*` | For Ads API | Advertising Profile ID per store |
+   | `AMAZON_ADS_FALLBACK_ASIN_US` | No | ASIN used for test/fallback metric calls |
+   | `AMAZON_LWA_CLIENT_ID` | For SP-API | LWA Client ID |
+   | `AMAZON_LWA_CLIENT_SECRET` | For SP-API | LWA Client Secret |
+   | `AMAZON_SP_API_REFRESH_TOKEN_US` | For SP-API | SP-API OAuth2 Refresh Token |
+   | `FEISHU_*_APP_ID` | For Feishu bot | Lark Open Platform App ID |
+   | `FEISHU_*_APP_SECRET` | For Feishu bot | Lark Open Platform App Secret |
+   | `FEISHU_*_USER_ACCESS_TOKEN` | No | User-level token for Bitable write access |
+   | `FEISHU_*_WEBHOOK_URL` | No | Incoming webhook URL for the bot |
+   | `SELLERSPRITE_EMAIL` | For SellerSprite | Account email |
+   | `XIYOUZHAOCI_PHONE` | For Xiyouzhaoci | Account phone number |
+   | `LINGXING_ACCOUNT` | For Lingxing | Account username |
+   | `REDIS_URL` | No | Enables Redis backend in `DataCache` |
+   | `SERVER_IP` | No | VPS IP shown in SSH tunnel hints |
+   | `SERVER_USER` | No | VPS username shown in SSH tunnel hints |
+   | `STORAGE_BACKEND` | No (default: `s3_compatible`) | Storage driver: `s3_compatible` or `local_http` |
+   | `CLOUDFLARE_R2_ACCOUNT_ID` | For R2 | R2 account ID; auto-builds endpoint URL |
+   | `STORAGE_ACCESS_KEY_ID` | For S3/R2/MinIO | S3-compatible access key |
+   | `STORAGE_SECRET_ACCESS_KEY` | For S3/R2/MinIO | S3-compatible secret key |
+   | `STORAGE_BUCKET_NAME` | For S3/R2/MinIO | Target bucket name |
+   | `STORAGE_PUBLIC_URL` | For S3/R2/MinIO | Base URL for public file access |
+   | `STORAGE_REGION` | No (default: `auto`) | Region (`auto` for R2/MinIO, real region for S3) |
+   | `STORAGE_ENDPOINT_URL` | For MinIO/Backblaze | Explicit S3-compatible endpoint override |
+   | `STORAGE_LOCAL_DIR` | For `local_http` | Local directory to write files |
 
 ---
 

AWS/docs/ARCHITECTURE.md

@@ -216,9 +216,26 @@ The AWS (Amazon Web Scraper) V2 project is a **Hybrid Intelligence Agentic Platf
 |   |                    |  | (Local JSON lookup)|  | (Direct IM attach) |    |
 |   | calc_profit        |  | restriction_check  |  | populate_bitable   |    |
 |   | calc_fba_fee       |  | epa_check          |  | send_local_file    |    |
-|   | (+ category bench) |  | patent_risk_calc   |  | send_url_file      |    |
-|   |                    |  |                    |  | send_data_file     |    |
+|   | (+ category bench) |  | patent_risk_calc   |  | export_md          |    |
+|   |                    |  |                    |  | export_html (→ R2) |    |
+|   |                    |  |                    |  | export_csv  (→ R2) |    |
 |   +--------------------+  +--------------------+  +--------------------+    |
+|                                                                              |
+|   ── src/core/storage/  (Strategy Pattern — swappable per .env) ──────────  |
+|                                                                              |
+|   StorageBackend (ABC)                                                       |
+|     upload(key, bytes, content_type) → public_https_url                     |
+|     upload_file(key, path) → public_https_url                               |
+|     delete(key)                                                              |
+|                                                                              |
+|   +──────────────────────+──────────────────────────────────────────────+   |
+|   | S3CompatibleBackend  | R2 / AWS S3 / MinIO — same boto3 client,    |   |
+|   |                      | only endpoint_url changes                    |   |
+|   +──────────────────────+──────────────────────────────────────────────+   |
+|   | LocalHTTPBackend     | VPS dir + nginx/caddy; no cloud dependency   |   |
+|   +──────────────────────+──────────────────────────────────────────────+   |
+|                                                                              |
+|   Switching backend: set STORAGE_BACKEND env var; zero code changes.        |
                                |
                                v
 +==============================================================================+
@@ -543,16 +560,19 @@ The AWS (Amazon Web Scraper) V2 project is a **Hybrid Intelligence Agentic Platf
 
 
 ================================================================================
-                    EXTENSION RULES  (5-Dimensional Orthogonal)
+                    EXTENSION RULES  (6-Dimensional Orthogonal)
 ================================================================================
 
   New Entry Point    EntryPoint adapter  +  Gateway register     Zero change elsewhere
   New Workflow       WorkflowRegistry.register(name, build_fn)   Zero change elsewhere
   New Data Source    New MCP Server  +  Tool Registry register   Zero business change
   New Output Format  Callback subclass  +  CallbackFactory reg   Zero change elsewhere
   Switch Model       Intelligence Router providers register       Zero business change
+  Switch Storage     Set STORAGE_BACKEND env var                 Zero code change
+                     (s3_compatible → R2/S3/MinIO; local_http → VPS nginx)
+                     Add backend: subclass StorageBackend, 1-line factory entry
 
-  Result: Five extension dimensions are orthogonal; changing one leaves others intact
+  Result: Six extension dimensions are orthogonal; changing one leaves others intact
 
 
 ================================================================================

AWS/docs/DEV_GUIDE.md

@@ -152,11 +152,24 @@ This guide reflects the **Domain-Driven Design (DDD)** and **Dual Orchestration*
 ## 5. Directory Mapping (Summary)
 
 *   `src/core/`: Kernel, Models, Telemetry, and shared Utils (Proxy, Cookies, Context).
+    *   `src/core/storage/`: **Storage abstraction layer** (Strategy Pattern). Swap backends via `STORAGE_BACKEND` env var — no code changes.
+        *   `S3CompatibleBackend` — Cloudflare R2 / AWS S3 / MinIO (same boto3 client, different endpoint)
+        *   `LocalHTTPBackend` — VPS local directory served by nginx/caddy
+        *   Usage: `from src.core.storage import get_storage_backend; url = storage.upload(key, bytes, mime)`
 *   `src/entry/`: Entry adapters (CLI, Feishu, etc.).
 *   `src/gateway/`: Auth, Rate Limiting, and Unified Dispatching.
 *   `src/jobs/`: Job management, Checkpoints, and Callbacks.
 *   `src/mcp/servers/`: Microservices providing specific tools.
+    *   `src/mcp/servers/output/tools/export_html.py` — Converts markdown/HTML to styled HTML file; optionally uploads images via storage backend.
+    *   `src/mcp/servers/output/tools/export_csv.py` — Exports records to CSV; uploads via storage backend (falls back to local file if unconfigured).
 *   `src/registry/`: The central hub for Tool, Resource, and Prompt discovery.
 *   `src/intelligence/`: LLM Providers, Routing, and AI Processors.
 *   `src/workflows/`: Sequential, deterministic engine.
 *   `src/agents/`: Autonomous, LLM-driven reasoning.
+
+## 6. Adding a New Storage Backend
+
+1. Subclass `StorageBackend` in `src/core/storage/your_backend.py` — implement `upload`, `upload_file`, `delete`.
+2. Add a branch in `src/core/storage/__init__.py` `get_storage_backend()`.
+3. Set `STORAGE_BACKEND=your_backend` in `.env`.
+4. No changes to `export_html`, `export_csv`, or any caller.

AWS/requirements.txt

@@ -24,6 +24,8 @@ scikit-learn
 numpy
 pandas
 pycausalimpact
+boto3
+markdown
 
 # Encryption
 cryptography

AWS/scripts/deploy_ubuntu.sh

@@ -141,6 +141,38 @@ LINGXING_PASSWORD=
 REDIS_URL=redis://localhost:6379
 SERVER_IP=
 SERVER_USER=
+
+# ── Object Storage (export_html / export_csv image/file upload) ─────────
+# Backend choices: s3_compatible (R2 / S3 / MinIO)  |  local_http (VPS nginx)
+STORAGE_BACKEND=s3_compatible
+
+# --- Cloudflare R2 (recommended) ---
+CLOUDFLARE_R2_ACCOUNT_ID=          # from R2 dashboard; endpoint auto-built from this
+STORAGE_ACCESS_KEY_ID=             # R2 API token → Access Key ID
+STORAGE_SECRET_ACCESS_KEY=         # R2 API token → Secret Access Key
+STORAGE_BUCKET_NAME=               # your R2 bucket name
+STORAGE_PUBLIC_URL=                # e.g. https://pub-<hash>.r2.dev or custom domain
+STORAGE_REGION=auto
+
+# --- AWS S3 (omit CLOUDFLARE_R2_ACCOUNT_ID, set region) ---
+# STORAGE_ACCESS_KEY_ID=
+# STORAGE_SECRET_ACCESS_KEY=
+# STORAGE_BUCKET_NAME=
+# STORAGE_PUBLIC_URL=https://<bucket>.s3.<region>.amazonaws.com
+# STORAGE_REGION=us-east-1
+
+# --- MinIO / self-hosted S3 (set explicit endpoint) ---
+# STORAGE_ENDPOINT_URL=https://minio.yourdomain.com
+# STORAGE_ACCESS_KEY_ID=
+# STORAGE_SECRET_ACCESS_KEY=
+# STORAGE_BUCKET_NAME=
+# STORAGE_PUBLIC_URL=https://files.yourdomain.com
+# STORAGE_REGION=auto
+
+# --- VPS local directory + nginx (no cloud dependency) ---
+# STORAGE_BACKEND=local_http
+# STORAGE_LOCAL_DIR=/var/www/files
+# STORAGE_PUBLIC_URL=https://files.yourdomain.com
 EOF
     echo "⚠️  .env created — edit it and fill in all required values before running the app."
 else

AWS/src/core/storage/__init__.py

@@ -0,0 +1,32 @@
+from __future__ import annotations
+"""
+Storage abstraction layer.
+
+Usage:
+    from src.core.storage import get_storage_backend
+    storage = get_storage_backend()
+    url = storage.upload("images/chart.png", png_bytes, "image/png")
+
+Backend is selected by STORAGE_BACKEND env var:
+    s3_compatible  (default) — R2 / S3 / MinIO
+    local_http               — VPS directory + nginx/caddy
+"""
+import os
+from .base import StorageBackend
+
+
+def get_storage_backend() -> StorageBackend:
+    backend = os.getenv("STORAGE_BACKEND", "s3_compatible").lower()
+    if backend == "s3_compatible":
+        from .s3_compatible import S3CompatibleBackend
+        return S3CompatibleBackend()
+    if backend == "local_http":
+        from .local_http import LocalHTTPBackend
+        return LocalHTTPBackend()
+    raise ValueError(
+        f"Unknown STORAGE_BACKEND={backend!r}. "
+        "Supported: 's3_compatible', 'local_http'."
+    )
+
+
+__all__ = ["StorageBackend", "get_storage_backend"]

AWS/src/core/storage/base.py

@@ -0,0 +1,23 @@
+from __future__ import annotations
+from abc import ABC, abstractmethod
+
+
+class StorageBackend(ABC):
+    """
+    Minimal interface for public-URL file storage.
+
+    Implementations: S3CompatibleBackend (R2 / S3 / MinIO)
+                     LocalHTTPBackend    (VPS dir + nginx/caddy)
+    """
+
+    @abstractmethod
+    def upload(self, key: str, data: bytes, content_type: str = "application/octet-stream") -> str:
+        """Upload bytes and return the public HTTPS URL."""
+
+    @abstractmethod
+    def upload_file(self, key: str, file_path: str, content_type: str = "application/octet-stream") -> str:
+        """Upload a local file and return the public HTTPS URL."""
+
+    @abstractmethod
+    def delete(self, key: str) -> None:
+        """Delete an object (best-effort; implementations may no-op)."""

AWS/src/core/storage/local_http.py

@@ -0,0 +1,58 @@
+from __future__ import annotations
+"""
+Local-HTTP storage backend.
+
+Files are written to a directory on the VPS that is served by nginx/caddy
+over HTTPS.  No S3 dependency needed.
+
+Required env vars:
+  STORAGE_LOCAL_DIR    absolute path served by nginx, e.g. /var/www/files
+  STORAGE_PUBLIC_URL   matching public base URL,      e.g. https://files.yourdomain.com
+
+nginx config example:
+  server {
+      listen 443 ssl;
+      server_name files.yourdomain.com;
+      root /var/www/files;
+      location / { autoindex off; }
+  }
+"""
+import logging
+import os
+from .base import StorageBackend
+
+logger = logging.getLogger(__name__)
+
+
+class LocalHTTPBackend(StorageBackend):
+    def __init__(
+        self,
+        local_dir: str | None = None,
+        public_url: str | None = None,
+    ):
+        self._dir        = local_dir  or os.environ["STORAGE_LOCAL_DIR"]
+        self._public_url = (public_url or os.environ["STORAGE_PUBLIC_URL"]).rstrip("/")
+        os.makedirs(self._dir, exist_ok=True)
+
+    def upload(self, key: str, data: bytes, content_type: str = "application/octet-stream") -> str:
+        dest = os.path.join(self._dir, key)
+        os.makedirs(os.path.dirname(dest), exist_ok=True)
+        with open(dest, "wb") as f:
+            f.write(data)
+        url = f"{self._public_url}/{key}"
+        logger.info(f"[storage] written {key} ({len(data)} bytes) → {url}")
+        return url
+
+    def upload_file(self, key: str, file_path: str, content_type: str = "application/octet-stream") -> str:
+        with open(file_path, "rb") as f:
+            return self.upload(key, f.read(), content_type)
+
+    def delete(self, key: str) -> None:
+        path = os.path.join(self._dir, key)
+        try:
+            os.remove(path)
+            logger.info(f"[storage] deleted {path}")
+        except FileNotFoundError:
+            pass
+        except Exception as e:
+            logger.warning(f"[storage] delete {path} failed (ignored): {e}")