Draft: PS2 Optimization Corpus v2 integration

[PunishedSnake]

Purpose

Integrate the PS2 Optimization Research Library v2 into fhdb-bootstrap-manager on an isolated performance branch without destabilising the known-good HDL installer line.

Base: feature/hdl-game-installer
Head: perf/corpus-v2-integration
Initial audited baseline: 4b5aa8d85e86c9de570a2128b52d1eaa5b334844
Frozen Phase-0 hardware A/B point: 7875b14d837d6332f5edc37f1c12a55527d7dd87 / CI #666
Frozen resume-hash experiment: a43b073c32348e020c234fff64615c8c4cddc98d / CI #706

Engineering rules

• Manifest v2 decides source-of-truth routing.
• PS2_PERFORMANCE_BIBLE.md supplies the optimization workflow.
• Remove work/data/copies before adding specialised machinery.
• Producer/consumer/lifetime/ownership/alignment contracts precede allocator or buffering changes.
• No blind -O3, -ffast-math, global ALIGN(64), MMI/VU/Scratchpad, larger buffers or higher priorities.
• Timing/cache/DMA/FIFO/device claims require a real PS2.
• Each material experiment keeps exact binary identity plus a correctness/error-path A/B gate.

Primary docs:

• docs/CORPUS_V2_IMPLEMENTATION_PLAN.md
• docs/PHASE0_HARDWARE_AB_PROTOCOL.md
• docs/HDL_RESUME_HASH_BENCHMARK.md
• docs/HDL_DATAFLOW_CONTRACTS.md
• docs/HDL_IOP_RAM_BUDGET.md
• docs/ALIGNMENT_CONTRACT_AUDIT.md
• docs/ALLOCATION_LIFETIME_AUDIT.md

Phase 0: measurement foundation

Implemented:

• EE/IOP latency histograms with p50/p95/p99/max;
• source/HDD/SIF/cache/fallback traffic accounting;
• source/toolchain/build provenance;
• R5900 performance-counter harness with state preservation;
• linker map/symbol/instruction audit artifacts;
• host HDDMAN.LOG parser;
• same-source HDL_PROFILE=1/0 builds;
• separate IOP object trees and exact ELF/IRX identity gates;
• interleaved hardware sample templates and comparators.

Frozen CI #666:

                         PROFILE ON   PROFILE OFF
stripped ELF                638388        632884
EE named text               232780        229956
EE functions                   618           609
EE instructions              58246         57539
execute_transaction()         6156          6156
execute_transaction insn      1540          1540
hdl_stream.irx                9861          8405

PROFILE ON ELF   964d5c30613b16e5a160b51d4473000ce6da5740596a785d100d2c68a09686d7
PROFILE OFF ELF  4d1458ebf158c21759d1acdd3a44ecca094a5f9948c9e4461ef4a4beb8f23916
PROFILE ON IRX   8d3dbeabadbb860888b2c3d2072e8344953bea443faefccefce006b234cdb3db
PROFILE OFF IRX  f0b29957560ce2ef35a53e77fa8250f477d7aa6490037f00cdfe2edc04a39751

Static deltas prove PROFILE OFF is actually compiled out. They do not quantify runtime profiler overhead; that remains a real-console A/B.

Phase 1: remove unnecessary work/code

Accepted source-level cuts:

1. direct-fileXio application contract without unused generic POSIX pathname backend;
2. no-op unused libcglue timezone bootstrap;
3. integer-only bounded formatting, removing floating dtoa/vfprintf machinery;
4. transaction-stage metadata reuse: normal stage-4 -> stage-5 reuses the canonical 1024-byte metadata block for mandatory read-back comparison.

Rejected experiments remain documented, including the local draw2d shim which increased .text.

Phase 3: storage/HDL recovery experiment

CI #706 freezes the resume-hash/source-lifetime experiment. It can restore SHA state for already copied data and, for a valid complete PAYLOAD_VERIFIED checkpoint, skip source reopen/fingerprint/ISO-probe work while retaining mandatory full HDD SHA-256 read-back.

Any checkpoint failure falls back to the legacy source path. Journal authority and metadata durability are unchanged.

Status: HIPOTEZA DO TESTU until real PS2 correctness/crash-window/performance validation.

Phase 4: IOP/SIF preparation

Current fast COPY already follows:

USB/BDM -> IOP stage -> ps2hdd
                    -> one SIF DMA copy -> EE SHA consumer

USB producer prefetch is already double-buffered. SIF completion remains synchronous because EE immediately consumes one destination buffer; removing that wait without a new ownership protocol would be a race.

Known current incremental IOP budget for one active stream:

                                  PROFILE OFF    PROFILE ON
IRX loaded sections                    7283 B         8739 B
two-stage allocation                 131135 B       131135 B
max fragment map                      49152 B        49152 B
prefetch stack                         4096 B         4096 B
stream object                           648 B         1336 B
                                  ----------     ----------
known subtotal                       192314 B       194458 B
ThreadMan control objects            UNMEASURED     UNMEASURED
other active IRX/runtime             NOT INCLUDED   NOT INCLUDED

No third stage/ring is accepted before runtime free-memory and exposed-latency evidence.

Phase 5: allocation, lifetime and peak working set

Transaction workspace

CI #724 freezes workspace v1: one transaction-owned 64 KiB / 64-byte-aligned EE workspace is borrowed sequentially by COPY/source-hash and target verification. It removes one alloc/free pair without increasing peak workspace.

PROFILE OFF  ELF 632756 B  execute_transaction 6008 B / 1502 insn
PROFILE ON   ELF 638260 B  execute_transaction 6008 B / 1502 insn

CI #733 workspace v2 extended ownership backwards into source admission. It removed another alloc/free pair but grew the transaction controller by 24 B / 6 instructions and lengthened the buffer lifetime. Rejected/held unless hardware data proves the extra allocator cut worthwhile.

CI #739 tests only source_fingerprint() memalign(64, 64 KiB) -> malloc(64 KiB). Pinned fileXio accepts ordinary caller alignment, but residual edge handling can still cost time. Static footprint is slightly smaller; runtime remains HIPOTEZA DO TESTU.

CI #743 removed two unsupported 64-B static scratch alignments and produced zero section/BSS/text/instruction benefit. Rejected as layout-only source cleanup.

Forensic HDDMETA bounded read-back

Baseline forensic snapshot keeps a complete canonical APAMETA1 image and an equally large read-back copy alive simultaneously.

At 2048 patches:

canonical image              2,170,976 B
baseline full verify         2,170,976 B
baseline pair peak           4,341,952 B

CI #749 bounded v1 reduces read-back scratch to 64 KiB while preserving exact byte comparison, format, slot/non-overwrite policy, truncation detection and trailing-data rejection.

CI #752 bounded v2 removes v1's two fileXioLseek() size-check RPCs. It reads exactly the expected byte count and requires one final one-byte read to report EOF.

bounded pair peak            2,236,512 B
peak reduction               2,105,440 B  (~2.01 MiB)

Bounded v2 vs bounded v1:

.text                        -32 B  both PROFILE modes
EE named text                -36 B  both PROFILE modes
EE instructions               -8    both PROFILE modes
execute_transaction()          0 B / 0 insn
IOP binary                     byte-identical

CI #752 bounded-v2 hashes:

PROFILE OFF  ecd99a7aee199039146cfa8275d2ecbe360b9b486bb290adb3bd30d86ae10a54
PROFILE ON   3ace7ea8730dc7dd56fe6bea078b2aeedc1e7735c5bcc831e7d7b883f65bdd2f

This is a POTWIERDZONE static peak-working-set reduction and still a HIPOTEZA DO TESTU for real-PS2 recovery latency/correctness.

An independent APAMETA1 reference vector is now pinned before any canonical streaming experiment:

reference bytes        2216
reference image SHA256 601ba74fc619738dac19baa2a6cb53054b67803e00b1fccb6bf89c69ef4bab6f

A future streaming serializer must reproduce that byte stream before hardware testing.

Full EE + IOP direct allocator inventory

CI #753 extends the source inventory beyond libc and distinguishes EE heap from IOP SysMem:

allocation calls       25
free calls             72
total events           97
EE/libc events         75
IOP SysMem events      22

IOP direct ownership classes are now documented for:

1. stream object;
2. one- or two-stage stream allocation;
3. direct-BDM USB fragment map;
4. fixed 1024-byte metadata-commit read-back buffer.

The metadata verify allocation remains KEEP because it participates in write -> HIOCFLUSH -> read-back -> memcmp durability verification and costs only 1 KiB once per commit.

ThreadMan backing memory is intentionally not guessed from CreateThread/CreateSema; real runtime IOP inventory is still required.

Current gates

• isolated performance branch
• corpus-v2 implementation plan
• Phase-0 latency/traffic instrumentation
• same-source PROFILE ON/OFF pair
• exact ELF/IRX provenance + preflight/comparators
• real-console SCPH/revision/adapters/active-IRX/workload record
• real-hardware PROFILE ON/OFF overhead A/B
• R5900 empty-scope/deterministic-loop calibration on real EE
• Phase-1 source/libc/work-removal cuts
• Phase-1 real-hardware functional smoke
• Phase-3 producer/consumer/lifetime contracts
• isolated resume-hash/source-lifetime experiment + matched CI identity
• resume-hash correctness/crash-window/performance gate on real PS2
• static incremental IOP RAM budget
• runtime active-IOP-module/free-memory inventory
• Phase-5 direct EE/libc + IOP SysMem allocation/lifetime inventory
• Phase-5 explicit aligned(64) consumer-contract audit
• Phase-5 isolated transaction workspace / fingerprint / bounded-HDDMETA candidates
• Phase-5 real-hardware allocator/forensic acceptance gates
• Phase-2 I-cache/control-flow work, deliberately held behind hardware evidence
• Phase-4 SIF/buffering changes, deliberately held behind attribution + RAM evidence

This PR intentionally remains draft until the real-hardware A/B and regression gates are complete. Static size, architecture diagrams and emulator timing do not promote a hypothesis into a PS2 performance result.