Skip to content

web-result-data-bridge.md

Latest commit

 

History

History
487 lines (391 loc) · 20.3 KB

web-result-data-bridge.md

File metadata and controls

487 lines (391 loc) · 20.3 KB
Raw

Web Result Data Bridge — Judge Record → Scoreboard → Web Display

This document defines the structured data bridge that connects judge records, scoreboard entries, and future web result detail pages for Agent Olympics. It documents the exact field mappings, display rules, and data prerequisites that a web UI would consume — without building the web UI itself.

1. Data Flow

Result Packet (v1/v2)
    │
    ├──► scripts/score.js
    │       │
    │       ├── Automated Judge Record (judge-record-*.yaml)
    │       │
    │       └── Scoreboard Entry (scoreboard.json)
    │
    ├──► Human/Blind Judge
    │       │
    │       └── Full Judge Record (judge-record-*.yaml)
    │                 │
    │                 └── Scoreboard Entry (scoreboard.json)
    │
    └──► Web Data Bridge (this doc)
            │
            ├── Leaderboard columns
            ├── Result detail page sections
            ├── Comparison view fields
            └── Filter/tag predicates

The scoreboard JSON (scoreboard.json) is the primary data source for web display. Each scoreboard entry carries all fields needed to render a leaderboard row and link to a detail page. The detail page would additionally load the judge record and result packet by reference.

2. Leaderboard Column → Scoreboard Field Mapping

Each leaderboard column maps to a specific field path in the scoreboard entry.

Column Scoreboard Field Path Type Required Display Rule
Rank Computed integer — Sort by score.total_score desc. Tie-break: submission_metadata.performance_profile.raw_measurements.wall_time_seconds asc (lower wins).
Participant agent_id string yes Short label. Link to /results/<entry_id> if detail page exists.
Adapter submission_metadata.adapter string yes Badge or tag. Fall back to submission_metadata.runtime.
Runtime submission_metadata.runtime string yes Short name. Append version from submission_metadata.runtime_version when available.
Model submission_metadata.model string optional Display only when visibility policy allows. Suppressed for blind scoring rounds.
Node Class submission_metadata.hardware_profile.cpu_class string recommended Show declared class. Fall back to submission_metadata.node.
Total Score score.total_score number yes Normalized to rubric max. Format: integer or 1 decimal place.
Correctness score.dimensions.correctness {score, max} yes Per-dimension bar or fraction.
Evidence Quality score.dimensions.evidence_quality {score, max} yes Per-dimension bar or fraction.
Safety score.dimensions.safety {score, max} yes Highlight if <70% of max.
Tool Optimization score.dimensions.tool_optimization {score, max} if present Show when rubric includes this dimension (Agent Stack overlay).
Config Fitness score.dimensions.configuration_fitness {score, max} if present Show when rubric includes this dimension (Agent Stack overlay).
Operating Discipline score.dimensions.operating_discipline_and_safety {score, max} if present Show when rubric includes this dimension (Agent Stack overlay).
Reliability / Liveness score.dimensions.reliability_recovery_liveness {score, max} if present Show when rubric includes this dimension (Agent Stack overlay).
Communication score.dimensions.communication {score, max} yes Per-dimension bar or fraction.
Durability score.dimensions.durability {score, max} recommended Per-dimension bar or fraction.
Verdict score.verdict enum yes Badge: pass (green), conditional_pass (yellow), fail (red), disqualification (gray).
Result State status enum yes Badge: completed (full), partial (dashed), blocked (paused), failed (warning), disqualified (blocked).
Publishable publishable (via packet_ref) boolean yes If false or absent, mark leaderboard row as provisional/private.
Wall Time submission_metadata.performance_profile.raw_measurements.wall_time_seconds number optional Format as m:ss. Tie-break field.
Actions submission_metadata.performance_profile.raw_measurements.action_count integer optional Total tool call count.

2.1 Rank Computation

Rank is computed at render time, not stored in the scoreboard. Algorithm:

1. Filter entries where status ≠ 'blocked' and status ≠ 'disqualified'
2. Sort by score.total_score descending
3. For ties: sort by wall_time_seconds ascending (lower = better)
4. For double ties: sort by entry_id (deterministic)
5. Assign rank 1..N
6. Entries with score = null (pending judge) are listed separately

2.2 Publishable Flag

The scoreboard entry does not carry the publishable flag directly — it must be read from the underlying result packet. The web UI should load the result packet referenced by packet_ref and check its publishable field.

If publishable is absent or false, the leaderboard row should include a "(private)" label and the result detail page should show a banner: "This result has not passed redaction review. Displayed as provisional only."

3. Result Detail Page Sections → Data Sources

A result detail page (e.g., /results/<entry_id>) would consume data from three sources:

Section Primary Data Source Secondary Source
Header (participant, task, score) Scoreboard entry —
Scorecard (all dimensions) Judge record (judge_record_ref) —
Participant metadata Result packet (packet_ref) —
Task metadata Scoreboard entry + task envelope —
Evidence panel Result packet (evidence) + evidence bundle —
Tool use summary Result packet (tool_use_profile + raw_measurements) —
Risk and safety panel Judge record + result packet (risks) —
Reproducibility panel Result packet (comparable_metadata.artifact_hashes) —
Comparison data Scoreboard entries for same task_id —

3.1 Scorecard Section

The scorecard must display every dimension from the judge record's score_dimensions object. Children:

Scorecard
├── Dimension name (key from score_dimensions)
│   ├── Score / Max (numeric)
│   ├── Reason (judge_record.score_dimensions.<dim>.reason)
│   ├── Bar (visual: width = score/max * 100%)
│   └── Color (green ≥80%, yellow ≥60%, red <60%)
├── Total score
│   ├── Sum of all dimension scores
│   └── Verdict badge
└── Judge type (automated / human / llm-assisted / hybrid / pending)

The pending_dimensions array on the scoreboard entry lists dimensions awaiting human review. For those dimensions, display "Pending review" in place of the score, with a yellow/gray bar.

3.2 Participant Metadata Section

Display fields from submission_metadata on the scoreboard entry, sourced from comparable_metadata or top-level result packet fields:

Participant Metadata
├── Participant ID:       submission_metadata.adapter / agent_id
├── Runtime:              submission_metadata.runtime (v submission_metadata.runtime_version)
├── Model:                submission_metadata.model (provider: submission_metadata.model_provider)
├── Node:                 submission_metadata.node
├── Config profile:       submission_metadata.config_profile
├── Hardware profile:
│   ├── CPU class:        submission_metadata.hardware_profile.cpu_class
│   ├── Memory:           submission_metadata.hardware_profile.memory_gb GB
│   ├── Storage:          submission_metadata.hardware_profile.storage_class
│   ├── OS:               submission_metadata.hardware_profile.os_family
│   └── GPU:              submission_metadata.hardware_profile.gpu_model
└── Fixture:              submission_metadata.fixture_ref

3.3 Evidence Panel Section

Extract evidence items from the result packet's evidence array. Each item renders as a card:

Evidence Card
├── ID:        evidence.id
├── Kind:      evidence.kind (badge/icon)
├── Summary:   evidence.summary
├── Source:    evidence.source
├── Redacted:  evidence.redacted (badge if true)
└── Tag:       evidence.tag (optional filter tag)

Group by kind where multiple items share the same kind value.

3.4 Risk and Safety Panel

Risk & Safety
├── Safety score:     score.dimensions.safety.score / max
├── Judge notes:      judge_notes (from judge record)
├── Penalties:        penalties_applied (from judge record)
│   ├── Kind
│   ├── Amount
│   └── Reason
├── Risks:            result packet risks[]
│   └── Each risk string
└── Operating policy: result packet operating_policy (when present)
    ├── Approval boundaries
    ├── Secret handling
    ├── Destructive action rules
    └── Delegation policy

3.5 Reproducibility Panel

Reproducibility
├── Artifact hashes:
│   ├── Result packet:  comparable_metadata.artifact_hashes.result_packet
│   ├── Trace record:   comparable_metadata.artifact_hashes.trace_record
│   └── Evidence bundle: comparable_metadata.artifact_hashes.evidence_bundle
├── Fixture reference:  submission_metadata.fixture_ref
├── Task version:       submission_metadata.task_version
└── Packet ref:         packet_ref

3.6 Comparison Data Section

For side-by-side display of entries with the same task_id:

Comparison View
├── Task ID:        task_id
├── Entries:        [agent_id_1, agent_id_2, ...]
├── Common fields:
│   ├── Task metadata (title, event family, fixture)
│   └── Score dimensions (side by side)
├── Per-entry fields:
│   ├── Participant metadata
│   ├── Dimension scores (side by side)
│   ├── Wall time
│   ├── Action count
│   ├── Model calls
│   ├── Total tokens
│   └── Configuration profile
└── Comparability caveats: comparability_caveats[]

The comparable boolean and comparability_caveats array on each scoreboard entry inform whether direct comparison is meaningful. UI should show a warning for entries with comparable: false.

4. Filter and Sort Predicates

A web UI would need to filter and sort entries by these fields:

Filtering

Filter Field Path Type Example
Task task_id string ops-001
Participant agent_id string yukson
Adapter submission_metadata.adapter string openclaw
Runtime submission_metadata.runtime string openclaw
Model submission_metadata.model string gpt-5.x
Node Class submission_metadata.hardware_profile.cpu_class string small-vps
Verdict score.verdict enum pass, fail
Status status enum completed, partial
Judge Type judge_type enum automated, human, pending
Publishable (via packet_ref) boolean true
Comparable comparable boolean true
Event Family Derived from task_id prefix string ops, code, perf

Sorting

Sort Key Field Path Direction
Total score score.total_score desc (default)
Wall time submission_metadata.performance_profile.raw_measurements.wall_time_seconds asc
Correctness score.dimensions.correctness.score desc
Evidence quality score.dimensions.evidence_quality.score desc
Safety score.dimensions.safety.score desc
Participant agent_id asc
Task task_id asc
Generated at (from scoreboard generated_at) desc

5. Blind Scoring — Web Display Considerations

When score.js is run with --blind, the scoreboard entries have anonymized agent_id, submission_metadata.runtime, submission_metadata.model, and submission_metadata.node fields.

Web Display Rules for Blind Scoreboards

  1. Leaderboard columns: Show participant as "Blinded Participant N" — do not reveal the actual agent_id. Suppress the Adapter, Runtime, Model, and Node Class columns entirely, or replace them with "—" and a note that "Identifying metadata was withheld for blind scoring."
  2. Result detail page: Show the blinded identifiers in the participant metadata section. Do not show the agent_id, runtime, model, node, or config_profile from the original packet.
  3. Comparison view: Blind comparisons are valid because hardware_profile (cpu_class, memory_gb, etc.) is preserved. The note should read: "Participants' identities were anonymized before scoring. Hardware class is shown for hardware-fair comparison."
  4. Reveal after scoring: Once blind scoring is complete, a separate "reveal" step would map blinded IDs back to actual participants for publication. This step is not handled by score.js — it requires an external reconciliation table.

Blind Scoreboard Annotation

Blind scoreboards carry schema_description with the suffix "(blind — anonymized)". The web UI should detect this suffix and apply the blind display rules above automatically.

6. Pagination and API Considerations

The scoreboard JSON is designed to be paginated by the web UI client:

  • Page size: 25 entries by default.
  • Total entries: Available in summary.total_entries.
  • Sort order: Client-specified, default by total_score desc.
  • Filtering: Client-side or via query parameters to a future API.

A single scoreboard file may contain hundreds of entries (one per participant-task pair). The web UI should:

  1. Fetch the scoreboard JSON on page load.
  2. Apply filters and sort client-side for small- to medium-sized rounds (< 500 entries).
  3. For larger rounds, implement server-side pagination with query parameters (not specified here — future work).

7. Cross-Origin and Embedding Safety

All scoreboard and judge-record fields carry safe labels only:

  • No hostnames, IP addresses, or connection strings.
  • No API keys, tokens, private keys, or session cookies.
  • No raw transcripts or unredacted log lines.
  • Participant identifiers are operator-supplied labels — not real names unless that is the agreed convention.

The web UI should assert Content-Security-Policy headers and sanitize all displayed strings, even though the data source is trusted.

8. Validation and Testing

Validate the complete data bridge with:

# 1. Run full scoring pipeline on existing results
node scripts/score.js run

# 2. Run in blind mode to produce anonymized scoreboard
node scripts/score.js run --blind

# 3. Validate scoreboard against schema
node -e '
const fs = require("fs");
const Ajv = require("ajv/dist/2020");
const addFormats = require("ajv-formats");
const ajv = new Ajv({ allErrors: true, verbose: true });
addFormats(ajv);
const schema = JSON.parse(fs.readFileSync("schemas/scoreboard.schema.json", "utf8"));
const validate = ajv.compile(schema);
const sb = JSON.parse(fs.readFileSync("results/scoreboard.json", "utf8"));
const valid = validate(sb);
console.log(valid ? "Scoreboard valid" : "Scoreboard invalid: " + JSON.stringify(validate.errors));
'

# 4. Check that essential web-display fields exist on every entry
node -e '
const sb = JSON.parse(require("fs").readFileSync("results/scoreboard.json", "utf8"));
let missing = 0;
for (const e of sb.entries) {
  if (!e.agent_id) { missing++; console.log("MISSING agent_id in " + e.entry_id); }
  if (!e.score && e.judge_type !== "pending") { missing++; console.log("MISSING score in " + e.entry_id); }
  if (!e.packet_ref) { missing++; console.log("MISSING packet_ref in " + e.entry_id); }
}
console.log(missing === 0 ? "All web-display fields present" : missing + " entries missing fields");
'

9. Web Result Consumer (Source-Only Slice)

scripts/web-result-consumer.js is the first source-only consumer slice of the web-result data bridge. It reads a scoreboard.json file (produced by scripts/score.js) and produces static HTML pages for offline review, documentation, and CI validation — without deployment or live serving.

Output

Path Content
<output-dir>/index.html Leaderboard with rank, participant, task, score, dimension bars, status badges, and sort order per §2.1
<output-dir>/detail/<entry_id>.html Per-entry detail page with scorecard (§3.1), participant metadata (§3.2), hardware profile, performance measurements, validation status, comparability notes, and evidence items (§3.3)
<output-dir>/compare/<task_id>.html Comparison view for entries sharing the same task_id (§3.6) — side-by-side scores, dimensions, hardware, wall time

Usage

# Basic usage
node scripts/web-result-consumer.js results/scoreboard.json

# Custom output directory
node scripts/web-result-consumer.js results/scoreboard.json --output-dir docs/sample-output

# Blind mode with anonymized labels
node scripts/web-result-consumer.js results/scoreboard.json --blind

# Custom page title
node scripts/web-result-consumer.js results/scoreboard.json --title "Season 001 — Official Leaderboard"

Testing

bash scripts/test-web-consumer.sh

The test suite verifies:

  • Correct HTML output structure (DOCTYPE, closing tags, required sections)
  • Detail page render with scorecard, evidence, metadata cards
  • Comparison view generation for multi-entry task groups
  • Blind mode banner and structure
  • Graceful error handling (missing scoreboard, empty entries)
  • Custom title rendering

Sample Output

Static sample output is maintained at fixtures/web-sample/:

fixtures/web-sample/
├── index.html              # Sample leaderboard
├── detail/                 # Per-entry detail pages
│   ├── ops-001-yukson.html
│   └── perf-001-baseline-*.html
├── compare/                # Task comparison views
│   └── perf-001.html
└── blind/                  # Blind-mode output
    ├── index.html
    ├── detail/
    └── compare/

Regenerate sample output with:

node scripts/score.js run
node scripts/web-result-consumer.js results/scoreboard.json --output-dir fixtures/web-sample
node scripts/web-result-consumer.js results/scoreboard.json --output-dir fixtures/web-sample/blind --blind

Data Shaping Responsibility

The consumer handles all data shaping required for web display:

  1. Rank computation — Filters blocked/disqualified entries, sorts by total_score descending, tie-breaks by wall_time asc, assigns rank 1..N.
  2. Score formatting — Normalizes dimension scores to fraction and percentage; applies color coding (green ≥80%, yellow ≥60%, red <60%).
  3. Badge rendering — Status badges for each result state; verdict badges with semantic colors (pass green, conditional_pass yellow, fail red, disqualification gray).
  4. Evidence display — Loads result packets by packet_ref to show human-readable evidence summaries, kind badges, and redaction status.
  5. Hardware comparison — Side-by-side hardware profile tables in comparison view.
  6. Blind display rules — Detects blind mode from --blind flag, renders blind banner, suppresses identifying metadata per §5.
  7. Comparability caveats — Displays caveat notes with warning icons on comparison views.

Design for Future Extension

The consumer is structured so that a future dynamic web UI can re-use the same data shaping patterns:

  • Rank computation (computeRanks) is a standalone pure function — it can be called from a server-side route or client-side filter.
  • Dimension display logic (dimColor, dimBarHtml, formatScore) is isolated for reuse in React/Vue components.
  • The evidence loader reads packet files by reference — the same pattern would work with an API endpoint serving packet data.
  • Blind mode support is toggled by a single flag — integration with a runtime blind/anonymised endpoint is straightforward.

This document is part of the Agent Olympics MVP Round Engine documentation. It is the authoritative data bridge specification for future web leaderboard and result-detail page implementations. See also: Scoring, Scoreboard Schema, Judge Record v2 Schema, Result Packet doc.