Skip to content

Rebuild eval architecture: 3-oracle groundedness, composite scoring, RAG, hotkey-only validators - #4

Merged
kyron1112567 merged 24 commits into
mainfrom
refactor/eval-architecture-rebuild
May 7, 2026
Merged

Rebuild eval architecture: 3-oracle groundedness, composite scoring, RAG, hotkey-only validators#4
kyron1112567 merged 24 commits into
mainfrom
refactor/eval-architecture-rebuild

Conversation

@kyron1112567

Copy link
Copy Markdown
Contributor
  • feat: enhance leaderboard with submission metadata including agent details and artifact SHA
  • feat(orchestrator): chat-stream passthrough; consumer-api becomes thin SSE facade
  • feat(eval): multi-turn fixtures with live + scripted replay
  • feat(dashboard): queued submissions page
  • feat(cost): per-task LLM cost via owner-api ledger injection
  • fix(validator): burn 1.0 - total_assigned so EIREL_FAMILY_WEIGHTS controls real burn
  • chore(fee_verifier): boot-time audit log + per-call DEBUG
  • feat(monitoring): add kube-state-metrics and node-exporter configurations
  • refactor: remove honeytoken and trace gate penalty features from the codebase
  • feat(schema): add EvalFeedback + OrchestratorToolCallLog tables; per-dimension TaskMinerResult fields
  • feat(shared): add safety pipeline + multi-metric scoring + tool pricing table
  • feat(tools): server-attested tool-call ledger; url-fetch + mcp-relay services; sandbox session persistence
  • feat(rag): rag-tool-service + per-run corpus indexer + miner-pod EIREL_RAG_URL injection
  • feat(validator): 3-oracle layer (OpenAI+Gemini+Grok) + Chutes reconciler + oracle cache
  • feat(validator): composite scoring with hard gates; pairwise from cached oracle answer; hotkey-signed ledger fetch
  • feat(orchestrator): graph-based product runtime; serving picker; promotions; mcp dispatcher
  • feat(owner-api): hotkey-signed eval feedback + ledger; tier-1 leaderboard fields; closed-run public submission viewer; auth tightening
  • chore: drop legacy x-tool-service, OpenAI baseline client, calibration manager, datasets router
  • refactor(shared): drop dead judge_config plumbing from Settings + bundle; tolerate legacy oracle_source values
  • chore(deploy): consolidate to EIREL_EVAL_JUDGE_*; drop validator EIREL_INTERNAL_SERVICE_TOKEN + EvalFeedback proxy envs; wire rag-tool-service
  • chore: pyproject script entries (rag-tool-service, url-fetch); ensure dataset dir tracked
  • docs: README + miner-guide + validator-guide for the new eval architecture
  • test: oracles + reconciler + composite + ledger + feedback + product orchestrator + safety coverage

…hed oracle answer; hotkey-signed ledger fetch
…oard fields; closed-run public submission viewer; auth tightening
…L_INTERNAL_SERVICE_TOKEN + EvalFeedback proxy envs; wire rag-tool-service
@kyron1112567
kyron1112567 merged commit deabcc6 into main May 7, 2026
1 check passed
@kyron1112567
kyron1112567 deleted the refactor/eval-architecture-rebuild branch May 7, 2026 19:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant