Important
Research prototype: there is no end-user release or download yet. Developers can build the experimental code using the instructions below.
NX Warp is an experimental video codec for rendered VR, developed with the custom WiVRn NX streaming stack. It explores a simple premise: head pose, reusable tiles and renderer information should let a headset reconstruct useful pixels with less work than a conventional whole-frame pipeline.
The current work combines a Vulkan encoder, a Vulkan decoder and an atlas renderer tested on Pico 4. The priority is low latency, full-resolution output and inexpensive reconstruction, especially at low bitrate. Visual approximations are valid experiments when they preserve useful structure and measurably reduce cost. Quality, bitrate and power remain measured tradeoffs.
The immediate target follows the Pico display: 90 Hz / 11.11 ms per update, with centre-first correction scheduling under investigation. The stretch target remains 240 Hz / 4.17 ms per update. Individual stages and repeated-render throughput have crossed parts of that budget; consistent 240 Hz delivery has not been demonstrated. This is a research prototype, with visible artifacts and incomplete quality gates, rather than a production-ready streaming release.
Removing work beats optimizing work.
Figure 1. Actual Pico capture from the native atlas renderer, configured for 2160 × 2160 output per eye. Tile seams and cube trails remain visible. A screenshot establishes the captured appearance, not moving-head quality or display FPS. Capture settings, control image and binary identity.
| Current scope | |
|---|---|
| Application | Rendered stereo VR through custom WiVRn NX |
| Implementation | C++20 and Vulkan compute/graphics; library identifier nxvc |
| Primary measured hardware | Radeon RX 7900 XTX host; Pico 4 / Adreno 650 headset |
| Native output in recent captures | 2160 × 2160 per eye; sequence fixture uses a padded 4352 × 2176 stereo atlas |
| Live integration | WiVRn NX atlas-live; experimental renderer switches remain opt-in |
| Presentation target | 90 → 120 → 144 → 180 → 240 Hz; recent live Pico captures use 90 Hz |
| Evidence | Controlled comparisons, raw timings, fixture/build identities and actual captures |
| Open problems | Completion-time tails, correction scheduling, motion artifacts, compression quality and sustained thermal behavior |
Read next: Architecture · Results · Visual results · Measurement rules · Status · Roadmap · Documentation
The design divides images into 64 × 64 tiles and makes reusable content cheap. The reference codec defines reconstruction behavior; Vulkan implementations and live rendering experiments are checked against their relevant reference or control paths.
The current atlas work separates stored tile content from the mapping used to render it. Reusing content can avoid full-picture reconstruction, while a tile-aware mesh moves warp calculations out of repeated fragment work. Changes to codec references still need explicit synchronization and correct feedback handling.
flowchart TB
subgraph PC["PC"]
direction LR
A["Rendered stereo frame<br/>and prediction inputs"] --> B["Vulkan encoder"]
end
B --> C["Tile stream · WiVRn NX transport"]
subgraph HS["Headset"]
direction LR
D["Vulkan decoder"] --> E["Atlas content<br/>and tile mapping"]
E --> F["Native renderer<br/>pose-aware warp"]
F --> G["OpenXR compositor"]
end
C --> D
D -. "receipt / reference feedback" .-> B
Figure 2. Simplified experimental integration. Encoding, transport, decoding, rendering and compositor scheduling have distinct costs; a gain in one box is not automatically an end-to-end gain.
The next architectural direction is to separate work by what each region needs:
| Region or tile | Intended compute policy | Research question |
|---|---|---|
| Predictable / unchanged | Warp or skip; reuse stored content | Can reconstruction and memory writes disappear? |
| Sparse correction | Update only the residual that matters | Can lightweight work avoid the dense path? |
| Dense / disoccluded | Spend more work on a fallback | Can difficult content stay within a deadline? |
| Fovea | Preserve detail and useful edges | Where does additional compute improve perception most? |
| Periphery / invisible | Cheaper correction, mostly warp, or skip | Can foveation control work as well as quantization? |
These are development priorities, not a claim that every path is integrated. Depth, motion vectors, visibility and object information could improve prediction further. The codec should ultimately describe what the renderer's prediction cannot explain.
A presentation update can reuse a stable correction state and apply a newer pose. This makes 60–120 Hz correction with faster presentation worth investigating. The sequence probe already tests repeated rendering, but currently repeats a static pose; it does not establish this complete pose-aware scheduling architecture.
Correction state A ───────────── B ───────────── C
Presentation warp → warp → warp → warp → warp → …
target: one useful update every 4.17 ms
Stable references, disocclusion handling and bounded image age are essential. Queueing more work can increase throughput while making the displayed image older.
Latest live tradeoff: decode queue priority reduced the mean of two run medians from 27.44 to 24.48 ms encode-to-selection, while fresh source selections fell from 76.52 to 72.67/s. Four animated Pico runs; opt-in, with equal priority still the default. This is not photon latency.
Latest work-omission experiment: exact PLANAR reuse passes CPU-reference and dropped-frame checks. It saves about 0.30 ms in a local-motion standalone decoder fixture, but shows no full-motion decode-time gain. The prototype remains archived; it is not a live latency improvement.
| Profile | Current status | Evidence and scope |
|---|---|---|
| Live wider-ring profile | Enabled opt-in on the custom WiVRn NX / Pico path | 77–82 fresh updates/s, about 4.94 ms decode GPU time and 2.04 ms server encode time; native 512 × 512 centre with a 1024 × 1024 fine-detail boundary. Live and fixture results |
| Synchronized GPU probe | Standalone Pico test; not live integrated | 14 exact-output checks; guide/history reconstruction costs 1.67–1.77 ms. Scope and timings |
| CPU quality prototype | NOT integrated; not Pico measured; separate quality experiment | Alternating-eye prototype results are recorded in the experiment note. Trace image |
| Stretch target | 240 Hz remains unproven | The target is 4.17 ms per update; no live 240 Hz delivery or physical-motion proof is established. Motion-proof archive |
A fresh 90-second Pico scene-motion capture measures 27.65 ms median / 34.84 ms p95 / 37.61 ms p99 from encode start to first render selection across 6,456 selected stereo frames. Canonical frame IDs and WiVRn clock conversion are checked. This excludes application rendering before encode, later presentation work and scanout: it is not motion-to-photon latency. The synchronized-history prototype is not enabled in this baseline.
A standalone Vulkan reconstruction probe keeps both centres fresh and switches both eyes between full detail and current-guide/warped-history reconstruction. 14 independent pixel checks pass on Pico, including signed translation across the packed sampling boundaries, eye seams and invalid-history fallback.
The simple guide/history pass costs 1.67–1.77 ms in isolated Pico runs. Sharing guide decisions across pixels was exact but slower (1.89 ms) and was rejected. These are reconstruction costs, not decoder savings or live latency. The timed inputs are static; GPU motion-quality and complete pipeline integration remain open. The wider-ring streamer stays on its tested profile.
The alternating-eye CPU experiment keeps both centres fresh, refreshes one eye’s peripheral detail per frame, and uses a current half-resolution guide to validate warped history for the other. Synchronized controls refresh both eyes every second frame at the same average detail budget. Both cadence phases are tested because motion can align with the refresh rhythm.
In the synthetic moving fixture, alternating history reduced peripheral MAE from 5.199 for fresh guides alone to 0.239. Synchronized controls measured 0.212–0.267 at the same detail budget. Alternating had higher stereo residual error (0.479 versus 0.306–0.389), so better retained detail is not sufficient to choose it for VR. These are intensity errors on a 0–255 scale, not measured comfort or GPU savings.
Centres, refresh budgets, lost-detail recovery, history reset and ambiguous checker motion have CPU checks. GPU/codec integration and Pico validation remain open. The live wider-ring profile is unchanged.
Wider detail falloff: the opt-in wide PLANAR ring expands the fine-detail boundary from 768×768 to 1024×1024 per eye and moves the coarser bands outward. The native 512×512 centre stays unchanged. These are wider 4-pixel PLANAR cells, not a true half-resolution ring. The 32-frame fixture costs 5.3% more bytes with identical decoded native centres. Two 90-second live Pico motion trials delivered 77–82 fresh updates/s, with about 4.94 ms decode GPU time and 2.04 ms server encode time. The wider profile is enabled; these results show no demonstrated latency reduction.
Historical measured results and experiments
Follow-up: compact decoder workgroups and shorter ready waits includes a corrected, inconclusive workgroup test: its harness did not explicitly enable the specialized shader. The shorter ready-wait trial showed no gain. The next proposed architecture uses a low-resolution guide with retained full-resolution detail, with explicit history poses and stereo-aware repairs. Its CPU quality model is implemented in the guide-history fixture; GPU and codec integration remain proposed.
Adaptive peripheral updates (historical, September 9): the experiment keeps native centres fresh and selectively reuse peripheral fits. Removing unused PLANAR transforms reduced full-resolution offline encode GPU time from about 5.9–6.1 ms to 4.7–4.8 ms. Cadence caching itself adds little measured saving and remains opt-in; decoder skipping and motion-correct tile history are still open. Live Pico motion checks with fractional sender pacing deliver 81–82 fresh updates/s versus 75 in the control, with roughly 25% lower server encode time. Reported source offset is 57–59 ms versus 56 ms; halved latency is not achieved. These timings are not end-to-end latency.
Peripheral shading probe: a coarser outer density map saved only 0.23 ms of presentation GPU time and did not reduce source offset, so it was reverted. Measured comparison. The next proposed approach is staggered temporal tile updates: full-rate centre, half-rate middle, quarter-rate periphery, followed by a separately tested small peripheral blend. This tile scheduler is not implemented yet.
Pacing audit: experimental estimator changes were reverted after incomplete live trials. The benchmark now rejects stalled clients even when their processes survive. A restored 90-second run passed with 71.92 fresh updates/s and 58.75 ms source offset; this is recovery evidence, not a new latency gain. Failure records and validity checks.
Latency follow-up (September 9): disabling glow and dithering preserved the native centre and reduced presentation GPU work to 4.77 / 5.04 ms, versus 5.37 ms with them enabled. Source display-time offset measured 61.68 / 58.33 ms versus 64.41 ms in the intervening control, with 68.27 / 70.37 versus 63.92 fresh updates/s. These sequential trials vary; halving physical latency has not been demonstrated. Removing packet spreading alone did not show the large benefit suggested by comparison with an earlier session. Raw trials, controls, graph and capture.
Broad scene motion (September 9): a moving 195-cube wall on the Pico exposes limits hidden by the earlier small scene. With the headset stationary, the FDM + 4 ms readiness-wait profile delivered 77.22 / 77.02 fresh updates/s, compared with 68.23/s for the original profile. Source display-time offset remained ~70 ms; the earlier 13 ms improvement does not carry over to this workload.
A server rate-control correction now accounts for frame-admission tolerance: the old byte budget assumed fewer frames than the sender actually admitted. Two corrected runs delivered 77.90 / 77.80 fresh updates/s at 90.59 / 90.28 Mbit/s estimated payload, versus 98.57 Mbit/s before. QP reached 40, and payload still exceeds the 76.3 Mbit/s allowance. This is an accounting fix with a quality cost, not a demonstrated latency improvement or strict bitrate cap. Methods, raw motion logs, captures and reproduction.
Earlier small-scene readiness wait (September 9): waiting briefly for a newer completed frame recovers much of the fresh-frame loss from reduced-density shading. Two 4 ms-cap trials reached 86.37 / 85.95 fresh updates/s, versus 83.82/s for the original shading baseline. Source display-time offset fell 70.52 → 57.08 / 57.19 ms; this is not photon latency. The test profile is active on Pico, while both experimental switches default off. No sustained 90-fresh-FPS or physical-motion proof yet. One baseline startup crashed during runtime swapchain creation; its retry passed, and the cause remains unresolved. Methods, controls, raw logs and eye captures.
Fragment-density map experiment (September 9): an opt-in static FDM implementation cut presentation GPU time from 8.37 ms in the baseline to 4.76 / 5.01 ms in two settled 90-second runs, while rendered cadence reached about 89.6/s. Fresh source updates regressed from 83.97/s to 58.08 / 61.12/s; baseline behavior was restored and the FDM switch remains opt-in with default off. Source display-time offset was 69.74 → 55.25 / 55.41 ms. These are last-30 approximately two-second window means, not frame percentiles, and source offset is not photon latency. Six 90-second runs included a full-density attachment control, a JIT-off control, and a final baseline restoration check. There was no physical motion or photon measurement. Methods, metrics, density map and live chart.
Direct compact reconstruction (September 9): flat PLANAR tiles now evaluate only the samples that reach the compact output. Decoder p50 is 8.23 → 7.27 ms with byte-exact retained pixels in UINT and UNORM checks. Two live repeats measured 83.15 / 82.88 fresh updates/s, versus 70.87/s for the previous compact implementation, with the same centre and peripheral sampling settings. Source display-time offset was 70.76 → 68.95 / 69.33 ms, not photon latency. The final APK repeated 83.60 updates/s, with source offset 70.39 ms; latency improvement is small and variable. These stationary tests still fall short of sustained 90 Hz. Methods, tails, raw logs and diagnostic caveats.
Actual application-eye GPU captures now work before Pico's tracking overlay; they verify visible stereo output, not physical head-motion performance. The centre is sharp while the low-bitrate periphery remains deliberately coarse. Left eye · Right eye.
Compact-output experiment (September 9): keep the native 512 × 512 centre while storing quarter-density outer axes in a 1856 × 928 stereo NV12 image. Pico decoder-only p50 fell 11.10 → 8.23 ms, with byte-exact retained samples. The first live mapper regressed; replacing branches recovered that loss. With the extra peripheral filter removed (hardware bilinear sampling remains), two settled live captures measured 71.22 / 72.25 fresh updates/s, versus 53.88/s for the same-APK full-size control, and source display-time offset 73.27 → 69.98 / 69.38 ms. This changes peripheral quality; it is not an equal-quality comparison or photon-latency measurement. The stationary Pico had a tracking dialog covering its view, so visual and physical-motion validation are still outstanding. Compact output remains opt-in. Methods, correctness, rejected variants and raw logs.
Peripheral smoothing capture (September 9): the archived Pico captures show
the filter softening coarse PLANAR edges outside the protected centre. In a
matched hello workload at pacing .1, the last 15 completed windows with
smoothing on (11:24:50–11:25:18) averaged 5.65 ms GPU pass / 46.2 fresh
updates/s; smoothing off (11:25:40–11:26:08) averaged 4.96 ms / 53.7
fresh updates/s. Source offset was 73.84 / 75.24 ms and submit lead
38.49 / 38.41 ms. These are window means rather than frame percentiles, and
workload variance is visible, so no clear latency penalty or win is established.
Capture, script, and raw logs.
Graduated centre falloff: the selected experiment keeps the native 512 × 512 centre, then uses 4 → 8 → 16 → 32-pixel PLANAR cells outward. Paired Pico decode means were 10.206 ms graduated / 10.213 ms control, with identical centre pixels. It remains a stepped two-colour approximation. Before/after images, sampling diagram and rejected experiment.
Native-detail centre experiment (September 9): mixed frames retain native pixel sampling in a 512 × 512 centre per eye (optionally 1024 × 1024), with cheap two-colour PLANAR tiles outside. This corrects a limitation of the earlier all-flat mode: a native-sized output did not preserve native detail. A one-pixel stripe test retains 191.96 levels of alternating-column contrast, versus zero in the flat mode. The Pico quarter-centre decoder averages 10.83 ms, with 13.92 ms p95, in an isolated 60-frame test; this does not establish sustained 90 Hz streaming. Settings, pixel proof and timings.
The matching live Pico smoke test delivered 54.27 fresh updates/s in short active windows; the full capture includes an XR interruption and averages 37.65/s. Actual headset screenshot, logs and exclusions.
Controlled decoded-pixel comparison, not a headset screenshot. The centre uses ordinary INTRA coding; the periphery remains coarse. The stripe comparison uses the 1024-centre mode, while the timing above uses the smaller 512-centre mode.
Live integration boundary: the fast PLANAR renderer is now connected to custom WiVRn NX as an explicit opt-in. The earlier Pico benchmark numbers do not describe the installed streaming client. Interface and supported frames. A 22-second live smoke test reported 87 fresh updates/s; visual and physical head-motion checks remain outstanding. Live evidence and limitations.
Borrowed decoder output is separately opt-in with Android property
debug.wivrn.nx.borrowed_output=1; changing it requires a reconnect. The
three-frame Pico correctness probe and live copy/direct measurements
are archived in the borrowed-output evidence note.
90 Hz centre-first optimization: single-pass admission halves the earlier multi-pass median (5.32–5.45 → 2.73–2.76 ms), with zero deadline misses across two 720-frame native Pico motion runs. One run retains outer pixels for one frame; the other refreshes every tile. This remains an offscreen experiment, not live streaming proof. Paired results and actual capture.
Decoder materialized-reference improvement (historical, September 8): consecutive full-picture frames now reuse an already materialized reference. In the native Pico motion stress control, eligible-frame median decode time falls 121.82 → 66.11 ms with matching output hashes; motion-phase median falls 98.56 → 64.71 ms. This is a real but incomplete improvement: live motion performance and 240 Hz delivery remain unproven. Implementation, validation and timing figure.
The live sender also now treats decoder-worker backlog as an overload signal; previously only deliberate decode-stride drops triggered immediate pacing backoff. WiVRn NX change and limits. Its live cadence benefit has not yet been measured.
The copy shortcut also covers a complete full-resolution INTRA refresh, with matching Pico output hashes. A paired-colour-plane shader experiment showed no speedup and remains disabled. Follow-up tests and limits. The client now rebuilds decoders after seamless reconnect even when settings are unchanged; decoder recreation was observed, but resumed video remains unverified. Reconnect evidence.
Motion regression baseline: isolated native-resolution Pico motion stress now reproduces the reported lag: the original 128-pixel rebuild threshold takes 94.24 ms median per moving-frame decode, versus 1.39 ms during static recovery. A matched decode-plus-render run completes only 17.18 fresh pairs/s after two startup frames. This adversarial synthetic input changes pose while keeping pixels fixed; it is a regression stress case, not a rendered head-turn quality test. Earlier sparse/static results do not characterize this workload. Motion timings and actual Pico offscreen captures. The native pose-transition staging failure is fixed; large-motion performance and reconnect reliability remain open.
Evidence snapshot: 2026-09-08. The rows below use different fixtures and measurement scopes. They must not be added together or interpreted as one unified benchmark.
| Experiment | Observed result | What it establishes |
|---|---|---|
| Remove unused encoder readback | 8.72 → 2.38 ms median, 3.49 ms p99; restored control 8.71 ms | An encoder-stage improvement; four fixture comparisons were bit-identical |
| Opt-in native vertex warp | 6.30 → 3.10 ms, median of GPU window means | Less renderer GPU work at native output; not frame-level p99 |
| Sequential Pico decode + render | 177–178 completed pairs/s, approximately 7.75 ms p99 | Improved offscreen sparse-motion performance; still over the 240 Hz deadline |
| Four renders per correction, long run | 298 renders/s, 74.5 fresh corrections/s; 67.6% within 4.17 ms | Repeated-render capacity over an 81.5-second capture; not paced 240 Hz delivery |
| Renderer GPU timestamp diagnostic | 3.99 ms GPU interval p99, 7.05 ms completion p99 | GPU command intervals and completion latency differ materially |
| CPU completion polling | 324 repeated renders/s, 7.52 ms p99, about 7× CPU time | A costly throughput tradeoff; disabled by default |
A new native 4352 × 2176 proof attempt still fails strict 240 Hz deadlines. The 30-second changing-pixel pan missed 41 / 7,200 deadlines. A separate rendered-camera trajectory (yaw ±60°, pitch ±25°, translation; 720 distinct input frames over three seconds) missed 3 / 720 deadlines, then 17 / 720 on repeat. Worst camera completion was 4.83 ms, above the 4.17 ms budget. These offscreen PLANAR measurements exclude encoding, network and compositor latency.
Raw timings, verified input hashes, camera trajectory and actual Pico captures. Average throughput remains approximately 240 FPS; consistent delivery and live head-motion performance are not established.
An opt-in GPU fit now emits coarse two-region tiles without CPU fitting or an unused warp-prediction pass. Encoder references use a separate padded PLANAR buffer; host-fit and GPU-fit reference checks pass. On the native synthetic fixture, removing unused clears and selecting the exact zero-slope decoder kernel reduced median reconstruction wall time from roughly 31 ms to 10–11 ms. This remains above the 240 Hz budget.
Host conformance passes 260 streams, with three skips. Pico conformance remains incomplete: directional and related cases fail, and synthetic failures reproduce with both new decoder optimizations disabled. These results do not qualify the build for live use. Logs, scope and encoder validation.
The standalone PLANAR probe now has a paired 7,200-frame Android capture on a 4352 × 2176 full-resolution synthetic changing-pixel pan fixture at approximately 157 Mb/s. The fixture is low complexity and all-PLANAR; the probe has no PC encoder, network, XR compositor, head-motion pose warp or live presentation path. With the queue-priority HIGH configuration, the paired runs average approximately 240 scheduled updates/s, while about 9.3% of deadlines are missed. This is a renderer and queueing diagnostic, not evidence of sustained 240 Hz delivery.
Figure 3. Actual Pico output and scientific latency diagnostics from the
standalone probe. total_ms is scheduled arrival to observed completion and
drives the deadline-miss count; interval_ms measures cadence and includes
inter-frame CPU cleanup. The plots exclude cold startup/device and pipeline
creation from the timed window. Archive, raw captures, summary and exact
reproduction.
The experiment supports an approximate GPU PLANAR reconstruction direction and motivates reducing redundant decoder memory work. Those are research directions; no additional numerical decoder claim is made here pending matched validation.
A same-binary pacing follow-up completed four HIGH-priority 7,200-frame runs on the same synthetic fixture. After excluding 24 warmup frames, the three busy-spin configurations missed 0.47–0.82% of 240 Hz deadlines, compared with 6.19% for the sleep control. The runs report approximately 240 scheduled updates/s, but this still does not establish consistent 240 Hz delivery. Busy-wait pacing trades CPU time and thermal headroom; power was not measured. Raw captures, summary script, chart and binary/shader identities.
The encoder was reading back 55 MiB of unused coefficients per frame. Removing that transfer cut measured live Lite encoding time without changing the compared bitstreams. The control was restored to check that the gain followed the change.
Figure 4. Controlled removal of unused GPU→CPU work. The encoder result falls inside 4.17 ms; the full pipeline still has other costs. Methods, profiler data, screenshots and bitstream checks.
The native renderer splits its mesh at tile and foveation boundaries and performs tile homographies in the vertex path. The final same-APK comparison retained full-resolution output and reproduced the reduction from 6.30 to 3.10 ms in GPU window means. It did not establish a consistent encode-to-selection tail-latency improvement.
Figure 5. Prototype renderer comparisons; the final cleaned pair is recorded separately. These GPU aggregates are not per-frame percentiles. Implementation and experiments.
A subsequent live experiment removes a cancelling color-conversion pair using mutable UNORM attachment views over SRGB swapchain images. GPU window medians were 2.7 / 3.2 / 2.6 ms for enabled / disabled / repeat, with no consistent selection-latency win. It remains opt-in. WiVRn NX source, fallback behavior and captures.
The long sequence test completed 24,000 offscreen renders in 81.5 seconds, using four static-pose renders per correction. Its steady render p99 was 7.29 ms. The renderer has useful capacity, but late iterations still prevent a consistent 4.17 ms cadence.
Figure 6. Fresh corrections and repeated renders count different work. The sparse monochrome fixture is not a broad scene-quality test. Raw CSVs, reproducible probe, timestamps and negative controls.
One native-resolution, reversed-order comparison measured encode-start-to-render-selection latency as follows:
| Pipeline | p50 | p95 | p99 |
|---|---|---|---|
| Custom WiVRn NX hardware HEVC | 21.822 ms | 31.159 ms | 33.322 ms |
| NX with experimental target cache | 17.633 ms | 22.321 ms | 23.256 ms |
This is a measured pipeline advantage under those conditions. Bitrate, quality, stereo organization and rendering paths differed. HEVC reached decoded pixels sooner; NX spent less time from decode completion to selection. An earlier native comparison favored HEVC. Neither establishes general codec superiority or photon latency. Controlled pairs and canonical frame mapping.
The following figures come from archived measurements and actual Pico GPU readbacks. The 90 Hz charts use native padded stereo 4352 × 2176, synthetic camera motion and 720 source frames per run. They measure offscreen parsing, upload and rendering; they do not measure live WiVRn NX or motion-to-photon latency. Full visual gallery and regeneration script.
Single-pass scheduling moves the latency distribution toward the full-draw control. The dashed line is the 11.111 ms display-period budget. Each curve represents a separate run.
Individual frames expose stalls that an average FPS number hides. Source frame order is preserved; no warmup samples are removed.
Independent component medians and p99s. These bars must not be summed into an end-to-end percentile; fence waiting overlaps GPU execution.
Both final single-pass runs complete all 720 frames within 11.111 ms. This finite observation does not guarantee future deadlines or sustained thermal behavior.
Skipped work is useful only if retained content remains useful. Single A retains some outer pixels for one frame; Single B updates every tile.
Historical controls: the first scheduler's lifetime-maximum estimate caused prolonged starvation. Expiring measurements helped, but did not establish a hard pixel-age bound. These are different runs from the final single-pass comparison.
New failure case: imposing a 2 or 3 ms admission threshold at 90 Hz leaves the outermost pixels stale for almost eight seconds. The artificial threshold is separate from the 11.111 ms cadence. Raw pressure-test results.
| Fresh complete frame | Forced centre-only update |
|---|---|
 |
 |
The forced-retention control is intentionally discontinuous: each centre contains frame 19 while its outside retains frame 0. Exact RGBA checks verify both regions. Captures are outside timing, and single-pass output matches the full-draw control.
| Forward view | Positive yaw | Negative yaw |
|---|---|---|
 |
 |
 |
Actual readbacks from the earlier 240-FPS source trajectory, including approximately ±60° yaw. These historical images establish changing rendered views, not physical headset tracking or successful 240 Hz presentation. Trajectory, identities and failed deadline tests.
Speed is the immediate Pico priority. The desired low-bitrate appearance preserves useful edges and structure even when texture or exact reconstruction is sacrificed. That is an objective to test, not a property every current image achieves.
| Rendered test material | Deliberate approximation |
|---|---|
 |
 |
| The rendered room corpus exercises stereo, head motion and independently moving content. | Planar tiles trade fidelity for hard-edged facets. The recorded experiment costs 2–4 dB; it is a visual choice, not a free quality gain. |
Figure 7. Existing research figures. Settings, measurements and regeneration commands.
Historical reference-codec quality gates failed on the tested band-limited material. Later timing wins do not close those gates. Live captures still show seams and motion trails, and sustained thermal behavior needs further qualification.
A result belongs with its fixture, build and definition of completion.
- Stage time: encoder or decoder work alone. Its inverse is not complete-pipeline FPS.
- GPU interval: timestamped GPU work, potentially including dependency stalls. Periodic GPU window means cannot supply per-frame p95 or p99.
- Completed offscreen work: a GPU fence confirms completion. Repeated rendering does not create new source frames.
- Render selection: the client selected a decoded frame for rendering. The logged
blitevent is not a photon measurement. - Presentation: paced updates delivered by the runtime and display. Recent Pico captures use 90 Hz; no physical 240 Hz presentation is demonstrated here.
Compare controls in both orders where practical, preserve warmup and session gaps, and report deadline misses alongside p50/p95/p99. Historical “displayed pose age” labels represented a difference between display timestamps, not measured motion-to-photon latency. Full protocol and corrections.
Rejected and inconclusive experiments remain useful evidence: R16 live handoff, shared-load pacing, all-skip uploads, and CPU affinity, color approximation and polling. Less GPU work alone is insufficient justification for more complexity.
| Area | Current state | Entry point |
|---|---|---|
| Reference codec and syntax | CPU reference, conformance material and recorded quality results | Reference, syntax |
| Vulkan encoder | Executable GPU encoder and measured live Lite path | Encoder |
| Vulkan decoder | Compute decode, reconstruction and atlas experiments | Decoder, atlas design |
| Live streaming | Experimental encoder/decoder/renderer integration in the custom fork | WiVRn NX atlas-live |
| Performance probes | Host and Pico evidence; sequential native-render probe | Sequence probe, 240 Hz work |
| Quality evaluation | Synthetic and rendered material, anchors and published gate results | Quality harness, gallery |
| Further design work | Renderer-assisted prediction, compute foveation and hybrid approaches have varying implementation depth | Architecture, hybrid, roadmap |
The design paper records a broader intended system than the measured live path. Consult implementation evidence before treating a design capability as available.
For CPU development, use CMake 3.25+, a C++20 compiler and Ninja with the checked-in presets:
cmake --preset dev
cmake --build --preset dev --parallel
ctest --preset dev --output-on-failureFor Vulkan development, install the Vulkan headers, loader, a usable Vulkan driver and glslc (shaderc). Some supporting components also use glslangValidator:
cmake --preset dev-vk
cmake --build --preset dev-vk --parallel
ctest --preset dev-vk --output-on-failureUse cmake --list-presets to inspect other configurations. Vulkan is off in the default CPU preset. Building this repository does not install the custom headset client.
| Task | Instructions |
|---|---|
| Encode/decode from the reference tools | Reference codec |
| Run Vulkan codec tools | Encoder, decoder |
| Reproduce offscreen native rendering | Sequence probe; Android runs require an NDK and a matching decoder build |
| Reproduce quality comparisons | Quality harness; Python dependencies and external anchors are documented there |
| Work on live VR streaming | Custom WiVRn NX branch and its build instructions |
Progress is measured against 90 → 120 → 144 → 180 → 240 Hz, with full-resolution output and low latency carried through every checkpoint.
- Close the completion-time gap. Separate GPU work, CPU scheduling, synchronization and correction cost; reduce late iterations rather than only average time.
- Decouple corrections from presentation. Keep reference state stable while newer poses drive faster updates. Measure image age and disocclusion failures.
- Remove work by region. Strengthen warp/skip and sparse paths; use foveation, visibility and renderer information to decide what needs computing.
- Prove the whole path. Run matched scene and bitrate comparisons, moving-head captures, long thermal tests and actual presentation measurements on suitable hardware.
Stage fusion, specialized queues and decode-cost-aware rate control are candidates when measurements justify them. Every feature must earn its complexity through quality, bitrate, latency, GPU cost and power behavior. Experiment direction · Broader roadmap and historical gates.
| Document | Purpose |
|---|---|
| Documentation index | Map of the project |
| Design paper | Rationale, alternatives and intended architecture |
| Bitstream syntax | Normative syntax; takes precedence over the design paper |
| Architecture | Modules and interfaces |
| Transport | Wire format, feedback and packet handling |
| 240 Hz experiments | Performance direction, methodology and limitations |
| Research gallery | Figures with settings and regeneration commands |
| Architecture decisions | Recorded implementation choices and tradeoffs |
| Integration design | Design context; use the live fork for current implementation |
Start with CONTRIBUTING.md. Useful work includes reproducible performance experiments, difficult motion and disocclusion fixtures, conformance coverage, and independent hardware measurements.
Codec syntax changes need the reference behavior, documentation and vectors to agree. Lossy visual experiments should state the approximation and compare it against a named control; they do not excuse undefined reconstruction, reference corruption or unsafe synchronization. Keep measurements scoped, retain negative results and run the checks relevant to the change.
Apache-2.0. Licensing does not constitute patent clearance; the repository's review brief records that separate work.
NX Warp builds on the open VR ecosystem: WiVRn, WiVRn NX, Monado, ALVR, and Mesa. The design paper discusses the compression, reprojection and perceptual research behind the project. Branding belongs to the NX family.
Useful pixels. Less work. Measured progress.