Skip to content

Latest commit

 

History

History
63 lines (48 loc) · 2.4 KB

File metadata and controls

63 lines (48 loc) · 2.4 KB

Release benchmarks

All performance values below come from the production Game Boy-compatible ROM under PyBoy 2.7.0 DMG timing. They are emulator measurements, not physical- hardware measurements.

Token timing

The ROM increments a volatile frame counter from VBlank. Each generation step records inference frames before the selected token is rendered; render frames are tracked separately. PyBoy DMG timing is converted using 59.7275 frames per second.

Measurement Value
Recorded production tokens 71
Average inference 106.55 frames
Average inference 1.784 seconds/token
Worst inference 113 frames
Worst inference 1.892 seconds/token
Average token rendering 3.87 frames

The release gate prefers an average below two seconds and requires every recorded token below 2.5 seconds.

Memory and ROM

Region Use
ROM 131,072 bytes / 8 banks
Static WRAM 4,487 bytes
Minimum stack headroom 3,705 bytes
Bank 0 used 14,201 of 15,872 usable bytes
Bank 0 headroom 1,671 bytes

The Bank 0 hard minimum is 1,024 bytes; 2,048 bytes is a warning preference. The release therefore passes while retaining an advisory warning.

Quality

Held-out quantized model evaluation reports 88.14% token accuracy, 86.84% exact match, 96.71% semantic consistency, zero known contradictions, zero malformed outputs, 88.24% held-out route validity, and 93.55% ending-transition validity. These are specialized dataset metrics, not general language benchmarks.

Parity and coverage

  • 2,664 Python/native deterministic engine vectors.
  • 98 Python/native integer inference contexts with full logits and attention.
  • A complete real-button-input Game Boy route ending at FLED_AT_DAWN.
  • All six endings and route predicates covered by host simulation/tests.
  • 512 sampled integer prompts in the final mask evaluation.
  • Four generated intro and three generated ending segments in ROM verification.

Machine-readable evidence is generated under build/reports/. The compact aggregate is build/reports/release-report.json.

Limitations of the numbers

PyBoy is useful for deterministic regression and relative timing but is not a substitute for an original DMG and cartridge. Automated semantic matching uses known valid reference families and cannot measure atmosphere, surprise, or human preference. The release has not undergone a blinded external user study.