Rebuild eval architecture: 3-oracle groundedness, composite scoring, RAG, hotkey-only validators - #4

Merged

kyron1112567 merged 24 commits into

mainfrom

refactor/eval-architecture-rebuild

May 7, 2026

Commits on Apr 27, 2026

feat: enhance leaderboard with submission metadata including agent details and artifact SHA
kyron1112567
committed

Commits on Apr 28, 2026

Commits on May 4, 2026

refactor: remove honeytoken and trace gate penalty features from the codebase
kyron1112567
committed

Commits on May 7, 2026

feat(schema): add EvalFeedback + OrchestratorToolCallLog tables; per-dimension TaskMinerResult fields
kyron1112567
committed
feat(shared): add safety pipeline + multi-metric scoring + tool pricing table
kyron1112567
committed
feat(tools): server-attested tool-call ledger; url-fetch + mcp-relay services; sandbox session persistence
kyron1112567
committed
feat(rag): rag-tool-service + per-run corpus indexer + miner-pod EIREL_RAG_URL injection
kyron1112567
committed
feat(validator): 3-oracle layer (OpenAI+Gemini+Grok) + Chutes reconciler + oracle cache
kyron1112567
committed
feat(validator): composite scoring with hard gates; pairwise from cached oracle answer; hotkey-signed ledger fetch
kyron1112567
committed
feat(orchestrator): graph-based product runtime; serving picker; promotions; mcp dispatcher
kyron1112567
committed
feat(owner-api): hotkey-signed eval feedback + ledger; tier-1 leaderboard fields; closed-run public submission viewer; auth tightening
kyron1112567
committed
chore: drop legacy x-tool-service, OpenAI baseline client, calibration manager, datasets router
kyron1112567
committed
refactor(shared): drop dead judge_config plumbing from Settings + bundle; tolerate legacy oracle_source values
kyron1112567
committed
chore(deploy): consolidate to EIREL_EVAL_JUDGE_*; drop validator EIREL_INTERNAL_SERVICE_TOKEN + EvalFeedback proxy envs; wire rag-tool-service
kyron1112567
committed
chore: pyproject script entries (rag-tool-service, url-fetch); ensure dataset dir tracked
kyron1112567
committed
docs: README + miner-guide + validator-guide for the new eval architecture
kyron1112567
committed
test: oracles + reconciler + composite + ledger + feedback + product orchestrator + safety coverage
kyron1112567
committed
refactor(tests): update OpenAI/Grok response handling in tests for consistency
kyron1112567
committed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Rebuild eval architecture: 3-oracle groundedness, composite scoring, RAG, hotkey-only validators - #4

Rebuild eval architecture: 3-oracle groundedness, composite scoring, RAG, hotkey-only validators#4
kyron1112567 merged 24 commits into
mainfrom
refactor/eval-architecture-rebuild

Commits on Apr 27, 2026

feat: enhance leaderboard with submission metadata including agent details and artifact SHA

Commits on Apr 28, 2026

feat(orchestrator): chat-stream passthrough; consumer-api becomes thin SSE facade

feat(eval): multi-turn fixtures with live + scripted replay

feat(dashboard): queued submissions page

feat(cost): per-task LLM cost via owner-api ledger injection

fix(validator): burn 1.0 - total_assigned so EIREL_FAMILY_WEIGHTS controls real burn

chore(fee_verifier): boot-time audit log + per-call DEBUG

feat(monitoring): add kube-state-metrics and node-exporter configurations

Commits on May 4, 2026

refactor: remove honeytoken and trace gate penalty features from the codebase

Commits on May 7, 2026

feat(schema): add EvalFeedback + OrchestratorToolCallLog tables; per-dimension TaskMinerResult fields

feat(shared): add safety pipeline + multi-metric scoring + tool pricing table

feat(tools): server-attested tool-call ledger; url-fetch + mcp-relay services; sandbox session persistence

feat(rag): rag-tool-service + per-run corpus indexer + miner-pod EIREL_RAG_URL injection

feat(validator): 3-oracle layer (OpenAI+Gemini+Grok) + Chutes reconciler + oracle cache

feat(validator): composite scoring with hard gates; pairwise from cached oracle answer; hotkey-signed ledger fetch

feat(orchestrator): graph-based product runtime; serving picker; promotions; mcp dispatcher

feat(owner-api): hotkey-signed eval feedback + ledger; tier-1 leaderboard fields; closed-run public submission viewer; auth tightening

chore: drop legacy x-tool-service, OpenAI baseline client, calibration manager, datasets router

refactor(shared): drop dead judge_config plumbing from Settings + bundle; tolerate legacy oracle_source values

chore(deploy): consolidate to EIREL_EVAL_JUDGE_*; drop validator EIREL_INTERNAL_SERVICE_TOKEN + EvalFeedback proxy envs; wire rag-tool-service

chore: pyproject script entries (rag-tool-service, url-fetch); ensure dataset dir tracked

docs: README + miner-guide + validator-guide for the new eval architecture

test: oracles + reconciler + composite + ledger + feedback + product orchestrator + safety coverage

refactor(tests): update OpenAI/Grok response handling in tests for consistency