Rebuild eval architecture: 3-oracle groundedness, composite scoring, RAG, hotkey-only validators

.env.compose.example

@@ -102,10 +102,14 @@ EXECUTION_STREAM_NAME=eirel:tasks
 EXECUTION_STREAM_GROUP=execution-workers
 
 # --- Dataset paths (inside containers) ---
+# Filesystem fallback path; the production task source is R2 via
+# OwnerDatasetBinding rows + EIREL_R2_* in owner-api's k3s secret
+# (deploy/k8s/owner-api-hybrid/secret.env). Compose services never
+# read R2 directly — they call owner-api over HTTP.
 EIREL_OWNER_DATASET_ROOT_PATH=/app/data/owner_datasets/families
-EIREL_CALIBRATION_FIXTURES_ROOT_PATH=/app/data/calibration
-EIREL_WORKFLOW_CORPUS_ROOT_PATH=/app/data/workflow_corpus
-EIREL_RESEARCH_TOOL_CATALOG_PATH=/app/data/research_tool_catalog.json
+# (EIREL_WORKFLOW_CORPUS_ROOT_PATH and EIREL_WEB_SEARCH_TOOL_CATALOG_PATH
+# are optional overrides; defaults work for the bundled corpus and
+# search-tool catalog. Set only if you point at a custom path.)
 
 # =============================================================================
 # LLM provider: Chutes-only (enforced by SUPPORTED_PROVIDER_IDS in code)
@@ -143,17 +147,6 @@ EIREL_TAVILY_API_KEY=
 EIREL_WEB_SEARCH_PER_BACKEND_TIMEOUT_SECONDS=10.0
 EIREL_WEB_SEARCH_MOCK=false
 
-# --- X API tool (1 call/task cap) ---
-X_TOOL_SERVICE_HOST_PORT=18086
-EIREL_X_TOOL_URL=http://x-tool-service:8086
-EIREL_X_TOOL_TOKEN=x-tool-token
-EIREL_X_TOOL_API_TOKEN=x-tool-token
-EIREL_X_TOOL_PORT=8086
-EIREL_X_BEARER_TOKEN=
-EIREL_X_API_BEARER=
-EIREL_X_API_CALLS_PER_TASK=1
-EIREL_X_MOCK=true
-
 # --- Sandbox (Python execution) ---
 SANDBOX_TOOL_SERVICE_HOST_PORT=18091
 EIREL_SANDBOX_TOOL_URL=http://sandbox-tool-service:8091
@@ -166,3 +159,14 @@ EIREL_SANDBOX_TOOL_MAX_CODE_BYTES=65536
 # Legacy names still referenced in some code paths
 SANDBOX_SERVICE_URL=http://sandbox-tool-service:8091
 SANDBOX_SERVICE_AUTH_TOKEN=sandbox-tool-token
+
+# --- URL fetch (browser-open) ---
+URL_FETCH_TOOL_SERVICE_HOST_PORT=18087
+EIREL_URL_FETCH_TOOL_URL=http://url-fetch-tool-service:8087
+EIREL_URL_FETCH_API_TOKEN=url-fetch-token
+EIREL_URL_FETCH_DEFAULT_MAX_REQUESTS=10000
+EIREL_URL_FETCH_MAX_RESPONSE_BYTES=1048576
+EIREL_URL_FETCH_TIMEOUT_SECONDS=15.0
+EIREL_URL_FETCH_MAX_REDIRECTS=5
+EIREL_URL_FETCH_PER_HOST_RATE=4
+EIREL_URL_FETCH_PER_HOST_WINDOW_SECONDS=10.0

.env.validator.example

@@ -23,41 +23,97 @@ EIREL_VALIDATOR_HOTKEY_NAME=
 EIREL_VALIDATOR_ACTIVE_FAMILIES=general_chat
 
 # =============================================================================
-# Pairwise judge (eiretes-judge sidecar on your machine)
+# Eiretes judge (Chutes-hosted GLM-5.1-TEE — runs in the eiretes-judge sidecar)
 # =============================================================================
-# Each validator runs its own eiretes-judge and pays its own LLM bill.
-# The judge compares the miner's response against the OpenAI baseline and
-# returns a pairwise verdict + per-dimension scores.
+# Each validator runs its own eiretes-judge sidecar and pays its own LLM bill.
+# All three internal judge roles (pairwise / multi / eval) share ONE
+# Chutes-hosted GLM-5.1-TEE deployment — single TEE-attestable model covers
+# the entire scoring path; validators can verify which model+weights produced
+# the score.
+#
+# Cost ballpark: ~$0.05/judge call × 100 tasks × N miners = ~$5N/run.
 
-EIREL_PROVIDER_CHUTES_API_KEY=
-
-EIREL_JUDGE_BASE_URL=https://llm.chutes.ai/v1
-EIREL_JUDGE_API_KEY=
-EIREL_JUDGE_MODEL=moonshotai/Kimi-K2.5-TEE
-EIREL_JUDGE_TIMEOUT_SECONDS=300
-EIREL_JUDGE_RUBRIC_VERSION=pairwise_general_chat_v1
-EIREL_JUDGE_MAX_TOKENS=8192
+# Eiretes-side judge config — read by eiretes/eval/config.py.
+# Falls back to EIREL_JUDGE_* below when blank (one-release compat).
+EIREL_EVAL_JUDGE_BASE_URL=https://llm.chutes.ai/v1
+EIREL_EVAL_JUDGE_API_KEY=
+EIREL_EVAL_JUDGE_MODEL=zai-org/GLM-5.1-TEE
+EIREL_EVAL_JUDGE_TIMEOUT_SECONDS=300
+EIREL_EVAL_JUDGE_MAX_TOKENS=8192
 
+# Where the validator-engine reaches the eiretes-judge sidecar.
 EIREL_JUDGE_SERVICE_URL=http://eiretes-judge:8095
 
 # =============================================================================
-# OpenAI baseline (ChatGPT reference via Responses API + web_search tool)
+# Validator-side 3-oracle layer (OpenAI + Gemini + Grok)
+# =============================================================================
+# At task-claim time, items tagged ``oracle_source: three_oracle`` get
+# enriched by the validator's 3-oracle fanout: OpenAI + Gemini + Grok in
+# parallel, then the Chutes reconciler synthesizes the consensus claim
+# set. ``expected_claims`` are cached in-process per task and reused
+# across every miner that judges that task within the batch.
+#
+# Each validator independently produces ground truth — decentralized
+# verification, no single-point oracle bottleneck. Cost per validator
+# per cycle: ~100 tasks × 4 LLM calls = ~400 calls (cached across
+# miners). With M validators, total = 400M.
+#
+# Layer activation: ALL THREE oracles + reconciler must have BASE_URL +
+# API_KEY + MODEL configured. Below that, three_oracle items fall back
+# to ``oracle_status="disputed"`` with the template-floor must_not_claim
+# preserved (graceful degradation; not a crash).
+
+# OpenAI oracle
+EIREL_VALIDATOR_ORACLE_OPENAI_BASE_URL=https://api.openai.com/v1
+EIREL_VALIDATOR_ORACLE_OPENAI_API_KEY=
+EIREL_VALIDATOR_ORACLE_OPENAI_MODEL=gpt-5.4
+
+# Gemini oracle (Google Generative Language API; different request shape)
+EIREL_VALIDATOR_ORACLE_GEMINI_BASE_URL=https://generativelanguage.googleapis.com/v1beta
+EIREL_VALIDATOR_ORACLE_GEMINI_API_KEY=
+EIREL_VALIDATOR_ORACLE_GEMINI_MODEL=gemini-3.1-pro-preview
+
+# Grok oracle (xAI's OpenAI-compatible endpoint)
+EIREL_VALIDATOR_ORACLE_GROK_BASE_URL=https://api.x.ai/v1
+EIREL_VALIDATOR_ORACLE_GROK_API_KEY=
+EIREL_VALIDATOR_ORACLE_GROK_MODEL=grok-4.3
+
+# Optional knobs (defaults shown).
+EIREL_VALIDATOR_ORACLE_TIMEOUT_S=120
+EIREL_VALIDATOR_ORACLE_PARALLEL=1
+
+# Pairwise comparator vendor — which of the 3 oracles' answers the
+# pairwise judge sees as the reference. Eliminates the per-task
+# OpenAI baseline call (~$0.05/task × ~30 tasks/cycle ≈ $1.50/cycle
+# saved) by reusing the already-cached oracle answer. Choices:
+# openai | gemini | grok. Default openai. Falls back through the
+# other vendors then to expected_claims[0] when the chosen vendor
+# errored.
+EIREL_VALIDATOR_PAIRWISE_REFERENCE_VENDOR=openai
+
+# =============================================================================
+# Validator-side reconciler (Chutes-hosted zai-org/GLM-5.1-TEE)
 # =============================================================================
-# Each validator calls OpenAI once per claimed task to generate the baseline
-# response that miners are compared against. You pay this bill. The budget
-# cap stops a run if spend exceeds the configured USD amount.
+# Synthesizes the 3 oracle answers into ``expected_claims`` +
+# ``must_not_claim_extras``. Same TEE-attestable model the eiretes-judge
+# uses — single deployment covers the entire scoring path.
+#
+# Set to your Chutes API key — same account as the judge above, or a
+# different one if you want the reconciler funded from a separate ledger.
 
-OPENAI_API_KEY=
-OPENAI_BASELINE_MODEL=gpt-5.4
-OPENAI_BASELINE_BASE_URL=https://api.openai.com
-OPENAI_BASELINE_TIMEOUT_SECONDS=240
-OPENAI_BASELINE_MAX_COST_USD_PER_RUN=10.0
+EIREL_VALIDATOR_RECONCILER_BASE_URL=https://llm.chutes.ai/v1
+EIREL_VALIDATOR_RECONCILER_API_KEY=
+EIREL_VALIDATOR_RECONCILER_MODEL=zai-org/GLM-5.1-TEE
 
 # --- Optional: host-side port overrides ---
 VALIDATOR_ENGINE_HOST_PORT=18010
 EIRETES_JUDGE_HOST_PORT=18095
 
 EIREL_VALIDATOR_BATCH_SIZE=1
+# How many tasks the validator processes concurrently per claim batch.
+# Default is 2; raise for high-throughput operators with adequate
+# upstream LLM rate limits.
+EIREL_VALIDATOR_MAX_PARALLEL=2
 
 # =============================================================================
 # Miner latency SLA (scored at the validator, not invocation)
@@ -69,4 +125,4 @@ EIREL_VALIDATOR_BATCH_SIZE=1
 #   instant: miners pick a fast/non-thinking model.
 #   thinking: full reasoning + tool use allowed, 10-minute hard ceiling.
 EIREL_MINER_INSTANT_LATENCY_BUDGET_SECONDS=120
-EIREL_MINER_THINKING_LATENCY_BUDGET_SECONDS=600
+EIREL_MINER_THINKING_LATENCY_BUDGET_SECONDS=600

README.md

@@ -55,7 +55,10 @@ Started via `docker-compose.yml` at the repo root.
 | `owner-api` | Submission lifecycle, deployment management, run orchestration, score aggregation, chain-publication readiness checks |
 | `metagraph-listener` | Syncs registered neurons from Bittensor chain state; gates miner submission acceptance |
 | `provider-proxy` | LLM provider fan-out with per-run USD budget enforcement |
-| `web-search-tool-service` / `x-tool-service` / `sandbox-tool-service` | Subnet-owned tool endpoints available to miner agents |
+| `web-search-tool-service` | Web search tool exposed to miner agents |
+| `url-fetch-tool-service` | URL fetch / read tool exposed to miner agents |
+| `sandbox-tool-service` | Server-side Python sandbox for verifiable computation |
+| `rag-tool-service` | Indexes per-run document corpora; serves `rag.retrieve` for `rag_required` tasks |
 | `postgres`, `redis` | Storage + coordination |
 
 **Consumer-facing product** (optional — only if you run the subnet's end-user chat product on top of the subnet):
@@ -81,7 +84,7 @@ The subnet launches with a single family:
 
 | Family | Description |
 |--------|-------------|
-| **general_chat** | Multi-turn conversational assistant with optional web search, across `instant` and `thinking` modes. Backed by owner-routed tool services for web search, X, Semantic Scholar, and a server-side Python sandbox for verifiable computation. |
+| **general_chat** | Multi-turn conversational assistant across `instant` and `thinking` modes. Backed by owner-routed tool services: web search, URL fetch, a Python sandbox for verifiable computation, and RAG retrieval over per-run document corpora. |
 
 Additional families are defined on the roadmap and will activate in
 future releases. Enforcement is gated by `EIREL_LAUNCH_MODE=true`.
@@ -94,12 +97,25 @@ future releases. Enforcement is gated by `EIREL_LAUNCH_MODE=true`.
    (docker or k3s, selectable via `OWNER_RUNTIME_BACKEND`).
 3. At each run boundary, the current deployment set is frozen into an
    `EpochTargetSnapshot`; task rows are seeded into
-   `miner_evaluation_tasks` — one row per `(miner, task)`.
+   `miner_evaluation_tasks` — one row per `(miner, task)`. Bundles are
+   sampled across a capability × domain matrix so coverage stays
+   broad. RAG corpora referenced by `rag_required` tasks are indexed
+   into `rag-tool-service` once per run.
 4. Validators claim tasks in batches (with lease TTL), invoke the miner
-   through the owner proxy, judge locally, and submit back.
-5. On run close, per-miner summaries roll up into an
+   through the owner proxy, enrich each task at task-claim time via a
+   3-oracle layer (OpenAI + Gemini + Grok + Chutes reconciler) for
+   `expected_claims`, and judge locally with a multiplicative composite
+   that gates on `grounded_correctness` + `instruction_safety` and
+   incorporates server-attested tool-call attestation from the
+   orchestrator's ledger.
+5. Per-(run, miner, task) `EvalFeedback` rows are persisted server-side
+   when validators submit results; miners read their own rows via a
+   hotkey-signed endpoint.
+6. On run close, per-miner summaries roll up into an
    `AggregateFamilyScoreSnapshot` and `DeploymentScoreRecord` rows drive
-   the carryover / weight-publication path.
+   the carryover / weight-publication path. After close, every
+   submission archive scored in that run becomes publicly downloadable
+   from the leaderboard.
 
 ## Local development
 
@@ -125,12 +141,9 @@ pytest tests/ -q
 
 ```bash
 cp .env.compose.example .env.compose
-cp .env.host.example .env.host
 
 docker compose --env-file .env.compose up -d
-
-set -a && . ./.env.host && set +a
-eirel-ai migrate
+docker compose --env-file .env.compose run --rm api-gateway eirel-ai migrate
 ```
 
 Optional monitoring:
@@ -170,7 +183,9 @@ Postgres/Redis/S3 — see
 | **Submission fee** | `EIREL_SUBMISSION_TREASURY_ADDRESS` (empty disables), `EIREL_SUBMISSION_FEE_TAO` |
 | **Runtime** | `OWNER_RUNTIME_BACKEND` (`docker` / `kubernetes`) |
 | **Launch mode** | `EIREL_LAUNCH_MODE` (`true` → restrict to `general_chat`) |
-| **Internal auth** | `EIREL_INTERNAL_SERVICE_TOKEN` |
+| **Internal auth** | `EIREL_INTERNAL_SERVICE_TOKEN` (orchestrator ↔ owner-api ↔ tool platforms; **not** distributed to validators) |
+| **Provider keys** | `EIREL_PROVIDER_OPENAI_API_KEY`, `EIREL_PROVIDER_ANTHROPIC_API_KEY`, `EIREL_PROVIDER_OPENROUTER_API_KEY`, `EIREL_PROVIDER_CHUTES_API_KEY` |
+| **RAG tool** | `EIREL_RAG_TOOL_URL`, `EIREL_RAG_TOOL_API_TOKEN`, `EIREL_RAG_EMBEDDING_API_KEY` (defaults to `OPENAI_API_KEY`) |
 | **Storage** | `OBJECT_STORAGE_BACKEND` (`filesystem` / `s3`), `ARTIFACT_STORAGE_ROOT` |
 | **Datasets** | `EIREL_OWNER_DATASET_ROOT_PATH` |
 | **Tracing** | `OTEL_*` |
@@ -184,8 +199,9 @@ owner-api                # Control plane
 metagraph-listener       # Chain state sync
 eirel-provider-proxy     # LLM provider fan-out
 web-search-tool-service  # Tool: web search
-x-tool-service           # Tool: X / twitter
+url-fetch-tool-service   # Tool: URL fetch / read
 sandbox-tool-service     # Tool: Python sandbox
+rag-tool-service         # Tool: RAG retrieval over per-run corpora
 # consumer product (optional):
 orchestrator
 execution-worker
@@ -210,8 +226,18 @@ eirel-ai admin neurons list
 ## Security
 
 - Bittensor hotkey signatures + replay protection on all authenticated
-  endpoints (miner submissions, validator task claims/submissions).
-- Internal service tokens for inter-service calls.
+  endpoints. Validators authenticate every owner-api call with their
+  wallet hotkey (claim, result, ledger read, feedback read); miners do
+  the same for submission upload + their own per-task feedback. The
+  operator's `EIREL_INTERNAL_SERVICE_TOKEN` is reserved for inter-service
+  calls within the operator stack (orchestrator ↔ owner-api ↔ tool
+  platforms) — it is **not** distributed to 3rd-party validators.
+- Submission archives are private during the run that scored them;
+  after the run closes, archives become publicly downloadable from
+  the leaderboard (operators can audit; competitors can study).
+- Server-attested tool-call ledger: validators score `tool_attestation`
+  from owner-side `OrchestratorToolCallLog` rows rather than trusting
+  miner-emitted trace frames.
 - Circuit breakers for miner-pod invocation and chain extrinsic
   submission.
 - Pluggable artifact storage (filesystem for dev, S3 for production).