Rebuild eval architecture: 3-oracle groundedness, composite scoring, RAG, hotkey-only validators - #4
Merged
Conversation
kyron1112567
commented
May 7, 2026
Contributor
- feat: enhance leaderboard with submission metadata including agent details and artifact SHA
- feat(orchestrator): chat-stream passthrough; consumer-api becomes thin SSE facade
- feat(eval): multi-turn fixtures with live + scripted replay
- feat(dashboard): queued submissions page
- feat(cost): per-task LLM cost via owner-api ledger injection
- fix(validator): burn 1.0 - total_assigned so EIREL_FAMILY_WEIGHTS controls real burn
- chore(fee_verifier): boot-time audit log + per-call DEBUG
- feat(monitoring): add kube-state-metrics and node-exporter configurations
- refactor: remove honeytoken and trace gate penalty features from the codebase
- feat(schema): add EvalFeedback + OrchestratorToolCallLog tables; per-dimension TaskMinerResult fields
- feat(shared): add safety pipeline + multi-metric scoring + tool pricing table
- feat(tools): server-attested tool-call ledger; url-fetch + mcp-relay services; sandbox session persistence
- feat(rag): rag-tool-service + per-run corpus indexer + miner-pod EIREL_RAG_URL injection
- feat(validator): 3-oracle layer (OpenAI+Gemini+Grok) + Chutes reconciler + oracle cache
- feat(validator): composite scoring with hard gates; pairwise from cached oracle answer; hotkey-signed ledger fetch
- feat(orchestrator): graph-based product runtime; serving picker; promotions; mcp dispatcher
- feat(owner-api): hotkey-signed eval feedback + ledger; tier-1 leaderboard fields; closed-run public submission viewer; auth tightening
- chore: drop legacy x-tool-service, OpenAI baseline client, calibration manager, datasets router
- refactor(shared): drop dead judge_config plumbing from Settings + bundle; tolerate legacy oracle_source values
- chore(deploy): consolidate to EIREL_EVAL_JUDGE_*; drop validator EIREL_INTERNAL_SERVICE_TOKEN + EvalFeedback proxy envs; wire rag-tool-service
- chore: pyproject script entries (rag-tool-service, url-fetch); ensure dataset dir tracked
- docs: README + miner-guide + validator-guide for the new eval architecture
- test: oracles + reconciler + composite + ledger + feedback + product orchestrator + safety coverage
…tails and artifact SHA
…dimension TaskMinerResult fields
…services; sandbox session persistence
…L_RAG_URL injection
…ler + oracle cache
…hed oracle answer; hotkey-signed ledger fetch
…otions; mcp dispatcher
…oard fields; closed-run public submission viewer; auth tightening
…n manager, datasets router
…dle; tolerate legacy oracle_source values
…L_INTERNAL_SERVICE_TOKEN + EvalFeedback proxy envs; wire rag-tool-service
… dataset dir tracked
…orchestrator + safety coverage
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.