GUI ROI - #329
Open
uSpike wants to merge 16 commits into
Open
Conversation
Contributor
Author
|
I'll also note that there's a hidden existing feature to show the ROI points when looking at the lens preview (Single camera) but it's incorrect. This change should fix this. 
|
Contributor
|
Sorry, I deleted feat/panner-unify after merging and it closed your PR. I'll fix it. |
uSpike
marked this pull request as draft
June 9, 2026 14:22
uSpike
marked this pull request as ready for review
July 1, 2026 03:48
Contributor
Author
|
@mohamedtahaguelzim ready to review this PR. I added a small commits to address #397 and a small fix for showing the correct time in the GUI when seeking. |
Contributor
Author
|
Fixed clippy error.. |
PRs target dev; main advances by merging dev at release milestones. CI runs on pushes and PRs to both; the dead v2/v2-consolidation branch triggers are dropped.
…he GPU executor (reco-project#404) * docs: comments state reasons, not provenance Scrub plan-step numbers, review references, dates, and FRICTION-item tags from code comments across the workspace. Comments now state the constraint or reason a piece of code exists; where a historical note carried a still-relevant invariant (FFI layout drift, NPP stream serialization, silent sync fallback), the invariant stays and the story goes. Provenance lives in commit messages and the knowledge base, not in code that outlives the moment. * feat(reco-core): full-range and flip-180 become engine setters Source metadata configuration goes through StitchCore like every other live render parameter, with per-arm dispatch: the CPU executor stores full_range for its kernel and warns on flip_180 (CPU sources reverse buffers at decode). Two more session pipeline reach-throughs gone. * refactor(reco-core): GPU executor owns the Linux zero-copy residency The shared-texture machinery leaves the session: bind groups, decode backpressure channels, CUDA detection pointers, shared views/textures, and the VRAM lookahead pool now live on GpuExecutor as its resident-frame surface (configure_shared_textures, render_shared_slots, render_pool_slot, stage_shared_to_pool, stage_textures_to_pool, release_decode_slots, create_lookahead_pool). SharedTextureSet moves to interop (it bundles interop handles), the VRAM pool to gpu (it is GPU memory management); the pool's budget helpers keep their public surface there. Session arms shrink to orchestration: stage during produce, render from a slot, release after detection. Detection and replay-pack sites clone the Arc-backed views out of the executor so the engine's own methods can borrow mutably. * refactor(reco-core): GPU executor owns the NVMM residency The Jetson DMA-buf import cache and the NvBufSurfTransform detection surfaces move onto the executor (import_nvmm, stage_nvmm_to_pool, setup_nvmm_detection, nvmm_detector_frames). Render and detection consume different handles of the same NVMM frame - the DMA-buf fd renders, the raw NvBufSurface pointer letterboxes for the detector - so both caches live together on the residency component. The session keeps only its public setup_nvmm_detection entry, now a delegation. * refactor(reco-core): GPU executor owns the D3D11 staging residency The D3D11VA staging pool moves from the session onto the executor's Residency component: pool creation (with the lookahead slot sizing), frame staging, slot rendering, and detection-view access become GpuExecutor methods. The session's D3D11 arms delegate; detection holds Arc-cloned views across the engine call instead of borrowing the pool. StitchCore::render_imported_views_at_pose is deleted (the executor renders its own staged slots). Windows arms are compile-checked by the CI windows lane; local gates cover the shared surface. * refactor(reco-core): GPU executor owns the Metal import residency The CVPixelBuffer texture cache moves from the session onto the executor's Residency component behind an import_metal method that returns the four imported Y/UV planes. process_metal_frame collapses into import + the shared imported-NV12 render path (identical render and replay-pack sequence); the macOS copy arm imports through the executor before staging to the pool. macOS arms are compile-checked by the CI macos lane; local gates cover the shared surface. * refactor(reco-core): GPU executor owns NV12 delivery One triple-buffered NV12 converter on the executor (lazy, keyed by output dims so a resize recreates it) replaces the session's eager converter and the engine's separate preview converter. The engine preview tap and the session encode path now read back through the same convert_nv12/flush_nv12 pair, and the session no longer reaches into the pipeline for the render target. Unaligned viewports now round down to NV12-safe dims on every path (previously the session hard-failed at build while preview rounded); the rounding is logged when it changes the dims. * refactor(reco-core): StitchPipeline goes crate-internal The pipeline reach-throughs are gone: StitchCore and StitchSession lose pipeline()/pipeline_mut(), StitchPipeline and GpuSourceBindGroups demote to pub(crate), and the unused render_gpu_frame_at_pose wrapper, render_nv12_to_view, the pipeline gpu_name, and the write-only SharedTextureSet bind_groups field are deleted (the demotion made the dead code checkable). render::pipeline stays a public module for the frame-plane value types and PipelineError it re-exports. StitchCore::gpu() survives as a narrow accessor: reco-gui and the CLI Bayer path create auxiliary resources on the engine's device, and GpuContext is a public consumer-facing type. Public items 574 -> 565. * fix(reco-detect): enable ort's DirectML feature on Windows ort_session registers the DirectML EP on Windows, but without ort's directml cargo feature the registration silently fails and inference falls back to the CPU EP. A Windows target-specific dependency entry carries the feature so every Windows ort build gets GPU inference. * refactor(reco-core): cuda_buf_info accessor on the GPU executor Session code read residency.cuda_buf_info through the field path at three sites; a cloning accessor matches how the other resident-frame state (shared views, D3D11 views) is exposed.
…eco-project#405) * refactor(reco-core): rename the Encoder seam to OutputSink The delivery seam generalizes beyond encoding (snapshots, streams, replay), so the trait says what it is: a sink. Encoder -> OutputSink (submit -> consume), EncodeError -> SinkError, encoder.rs -> sink.rs, AsyncEncodeThread -> SinkThread. The trait gains wants() -> SinkInput, declaring consumed format and residency up front (GPU-resident variants extend the enum without a trait break), and name() for per-sink log identity. Mechanical rename for all consumers; no behavior change. * refactor(reco-core): session delivers through a Vec of attached sinks One attach API replaces three delivery mechanisms: add_sink(sink, SinkOptions) supersedes set_encoder + add_encoder (zero callers) + the encoder/extra_encoders fields. Options are attach-time because they are deployment policy, not sink properties: delivery mode (Threaded with bounded queue + backpressure for lossless encoders, Inline zero-copy for lossy taps that must never block the render loop) and error policy (Abort the run vs log-and-detach while other sinks continue, e.g. a stream dropping while the recording carries on). Sinks wanting anything but NV12 CPU bytes are rejected at attach with a typed error until the CPU session loop delivers per wants(). finish() now finalizes every sink even when an earlier one fails, so no output file is left without its trailer. Fan-out lives in session/sinks.rs as GPU-free helpers with unit tests: identical frames to N sinks, error isolation per sink, finish reaching every sink. * refactor(reco-cli): snapshot writer becomes an OutputSink The (writer, tap-closure) pair and its keep-alive binding collapse into one type attached with inline delivery; the session now owns the writer's lifecycle and finalizes it with the other sinks. The NV12 tap - the last special-cased delivery mechanism - is deleted from the session (Nv12TapFn, set_nv12_tap, clear_nv12_tap, the per-frame tap call). * docs(reco-core): session docs describe sink delivery, not encoders The session module docs still described the pre-sink world (encoder fan-out, NV12 tap). Absolute crate:: paths in the sinks module doc: rustdoc resolves the merged mod-decl + file doc links in the parent scope, where super:: and bare names break. * fix(reco-core): surface the sink worker's real error on a dead channel When the sink thread dies mid-run, submit only ever saw the channel disconnect and returned a generic "sink thread died"; the real error (e.g. the codec rejecting a frame) lived in the JoinHandle and was discarded on Drop when the caller aborted before finish(). submit now joins the worker on disconnect and returns its actual error, so both Abort propagation and Detach warnings carry the true failure reason.
* feat(reco-core): CPU RGBA->NV12 kernel mirroring the GPU converter The CPU delivery path's missing half: BT.709 limited-range NV12 with the GPU compute shader's exact coefficient set, chroma averaging order, and rounding (SYNC_WITH markers on both sides). A GPU-vs-CPU oracle test pins the agreement to 1 LSB on every byte - the converter is deterministic per-pixel math, so unlike the full-stitch oracle there is no sampling noise to admit. The NV12 dimension rounding rule moves to one shared home (render::nv12_cpu::nv12_dims); the GPU executor delegates to it. * feat(reco-core): StitchSession runs over the CPU executor with_executor(Executor) becomes the primitive constructor; with_gpu is the GPU convenience that builds the executor from a SessionConfig. run() dispatches per arm: the CPU arm gets its own synchronous loop - decode, engine submit (detection + pose + software stitch run inside submit_frame_*), CPU NV12 conversion, sink fan-out. No staging ring, no warmup, no zero-copy; resident frames are rejected with a typed error and lookahead logs that it is ignored. Sink attach and finish go executor-agnostic (shared NV12 rounding over the executor's viewport; the flush drain only runs on the GPU arm). The session's DetectionTarget::gpu now returns None on the CPU arm instead of panicking, so reco-autocam falls back to CPU inference EPs. The end-to-end CPU test is the session's first that runs in the default suite - every prior session test needs a GPU. * feat(reco-cli): reco stitch --cpu - the GPU-less batch path --cpu stitches on the CPU executor with no GPU touched anywhere: software render, software decode, and libx264 by default (an explicit --encoder still wins). StitchJob::run picks the strategy in one branch and logs it. Software decode becomes a typed option threaded from the CLI down to the decoder (VideoDecoder::open_input_software -> force_software), replacing reliance on the RECO_NO_HWACCEL env var for this path; the env var stays as the global diagnostic override. Without the typed form, --cpu silently picked NVDEC-with-download decoders and NVENC, breaking the flag's no-GPU promise and leaving CUDA teardown errors at exit. * docs(reco-io): StitchJob doc mentions the CPU path
…U path) (reco-project#407) * feat(reco-core): calibration topology becomes a tagged enum Topology is now { LShape | Cylinder }: serialized with a type tag, deserialized accepting both the tagged form and the legacy untagged L-shape document (missing tag = L-shape, so existing files keep parsing). CylinderTopology carries the mono pre-stitched panorama parameters (focal_length, sweep_deg, screen_rotation_deg, video_height) with the established 180-degree player convention as defaults; Lens::flat covers the distortion-free mono source. validate() enforces lens count against the topology's camera count and cylinder parameter ranges; the axis-offset minimum is scoped to the L-shape (the cylinder's camera sits on the axis). Scene derivation gets one home (SceneGeometry::for_calibration: plane placement for the L-shape, an inert identity for plane-less topologies). L-shape-specific consumers (calibrate optimizer, GUI sliders, preview tuning keys, blend overrides) go through the l_shape()/l_shape_mut() accessors and no-op or warn on seamless topologies. * refactor(reco-core): the executor stitch seam goes N-ary stitch_rgba / stitch_rgba_yuv420p / StitchExecutor::stitch / CpuExecutor::stitch_* take one plane set per camera in projection order instead of a hardcoded (left, right) pair - the composite loop was already N-surface, only the entries pinned the count. The camera count is validated against the projection with a typed error. The GPU executor's synchronous stitch keeps its stereo shape behind the slice (mono GPU programs land with their own bind layout). YuvPlanes/Nv12Planes derive Copy: they are borrowed-slice bundles. CameraId, sources, and the engine submit API deliberately stay stereo-shaped - the N-camera redesign starts from this seam later. * feat(reco-core): CPU cylinder inverse map + analytic coverage The mono cylinder's CPU side: each output pixel casts a ray through the on-axis virtual camera (VirtualCamera's yaw/pitch convention, screen rotation folded into the basis), intersects the cylinder wall, and maps the hit's angle and height to the panorama UV - straight ahead samples u = 0.5 and u grows with yaw, fixing the draft shader's theta convention under which the forward ray fell outside the video. Units are consistent per the cylindrical-player convention: focal_length and video_height share source-pixel units, and video_height omitted means the source's own height (the draft's normalized-1.0 default painted a sliver). CylindricalProjection loses its config struct - the calibration document's CylinderTopology is the single home, mirroring the L-shape. Coverage is the analytic rectangle (yaw = +-sweep/2, pitch = +-atan(h/2r)) via a new CoverageBoundary::rectangular; a projection::for_topology resolver picks the projection a document calls for. * feat: mono submit path + single-input reco stitch for cylinder calibrations The engine gains submit_frame_mono_yuv: detection idles with a one-shot warning (the detection-to-panorama mapping is L-shape-only until the mono mapping lands) and the GPU arm returns a typed error until the mono GPU pass is wired - mono renders on the CPU executor. StereoFrame grows a Mono variant carried by a new single-decoder FfmpegMonoSource behind SmartFileSource::open_mono; every stereo consumer rejects it with the existing typed catch-alls. StitchJob takes an optional right input: the arity is checked against the calibration topology's camera count before any decoder spins up, and the CPU executor now resolves its projection from the document (projection::for_topology) instead of hardcoding the L-shape. The CLI right positional becomes optional - 'reco stitch pano.mp4 -c cylinder.json' imports a pre-stitched 180-degree panorama. * fix(reco-cli): camera blend override goes through the L-shape accessor The gstreamer-gated camera command still poked the topology fields directly; only CI's gstreamer clippy lane compiles it. * fix(reco-core): cylinder view was horizontally mirrored The theta sign made screen-right rays sample the video's left half. theta is now positive toward screen-right (+X at pose zero), so the right half of the output samples the right half of the panorama - caught on real pre-stitched footage. Positive yaw keeps VirtualCamera's sense (toward -X); the sign never surfaces because coverage is symmetric and panners share the basis, but the map's screen orientation must not depend on it. Tests now pin the un-mirrored invariant and the yaw sense separately. * chore(reco-core): scrub the external-player reference from the draft shader The convention is described as what it is; the draft's stale conventions are flagged for the mono GPU pass rewrite. * feat(reco-core): the cylinder honours the calibrated rig orientation Rig tilt is not pitch: panning a tilted rig rolls the viewport progressively toward the edges, because yaw rotates around the rig's tilted up axis. The cylinder map now builds its pan frame exactly like the L-shape (mirrors rig_frame + view_matrix: tilt rotates forward+up around base right, roll rotates up around the tilted forward, screen axes derive from the rotated forward+up pair - the look-at construction; deriving right from the pitch axis silently drops the roll). CylinderTopology loses screen_rotation_deg: it rolled the camera uniformly across the pan, but the player ecosystem's screen-tilt control rotates the painted surface - which is exactly framing.roll in the rig frame. One home, and the edge behavior is now correct. Coverage feeds rig tilt/roll into the existing viewport-roll margining and conservatively shrinks the pitch band by the tilt. Tests pin the band shift at pan center, the edge roll under pan (the tilt signature), and the surface-roll behavior of framing.roll. * fix(reco-core): tilted/rolled mono rigs clamp and orient correctly Two defects collapsed the pose on tilted mono rigs: the placeholder mono scene put the camera at the origin, so the pose-orientation path normalized a zero vector into NaN whenever tilt or roll was nonzero (level rigs early-return, which hid it); and the cylinder's coverage wrongly shrank its pitch band by the rig tilt. The painted band is world-fixed - rig orientation shapes how panning traverses it, not where it is - so coverage stays the analytic rectangle and the clamp's rotated-viewport margining (the same mechanism a tilted L-shape uses) accounts for the edge roll. Caught on real footage: an 18-degree tilt pinned the sweep to a single over-zoomed pose. * fix(reco-core): cylinder coverage no longer shrinks with rig tilt The second half of the tilted-mono fix: the painted band is world-fixed, so its pose-space extent does not depend on the rig orientation; only the clamp's rotated-viewport margining does. An edit-tool anchor mismatch left the conservative shrink in the previous commit. * refactor(reco-core): calibration topology type tag is mandatory Delete the untagged-document fallback in Topology deserialization: pre-1.0, no compat promises, and a silent L-shape default would mask a malformed cylinder doc. Untagged topologies now fail to parse. * fix(reco-io): mono source seeks instead of decode-discarding FfmpegMonoSource::skip_frames decoded and dropped every skipped frame, so a large --start-time decoded the whole prefix. Implement seek with the same strategy as the stereo source: drain short forward skips, respawn the decoder at the target keyframe otherwise. * docs(reco-core): name the mono camera convention, fill review doc gaps VirtualCamera::mono() + MONO_CAMERA_POSITION replace the bare [0, 0, 1] literals repeated across coverage, scene geometry, and the cylinder map docs. Motivate every CylinderMap field, the Copy derives on the plane bundles, and the serde default functions; TODO-mark the three arms stubbed until the mono GPU pass; stop overclaiming in the dyn-dispatch projection test. * chore: bump crossbeam-epoch 0.9.18 -> 0.9.20 RUSTSEC-2026-0204 (published 2026-07-06) flags 0.9.18; fails the cargo-deny and audit CI jobs on every push.
Bumps [cc](https://github.com/rust-lang/cc-rs) from 1.2.62 to 1.2.66. - [Release notes](https://github.com/rust-lang/cc-rs/releases) - [Changelog](https://github.com/rust-lang/cc-rs/blob/main/CHANGELOG.md) - [Commits](rust-lang/cc-rs@cc-v1.2.62...cc-v1.2.66) --- updated-dependencies: - dependency-name: cc dependency-version: 1.2.66 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps gstreamer from 0.25.2 to 0.25.3. --- updated-dependencies: - dependency-name: gstreamer dependency-version: 0.25.3 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
The async file dialogs (reco-project#398) are unparented and therefore non-modal: the whole UI stayed interactive underneath, letting callbacks interleave between opening a picker and applying its result. A scrim overlay now swallows pointer input while a picker is up, restoring modal semantics on every platform without giving up the responsiveness win.
log 0.4.33, uuid 1.23.4, open 5.3.6; actions/checkout v7, actions/cache v6, cache-apt-pkgs-action 1.6.3. Supersedes the stale main-targeted dependabot PRs opened before version updates moved to dev.
Give the modal guard a message card ("Finish in the file picker
window - click here to dismiss") and make a click clear it. The flag
is driven by a best-effort worker/poll pair, so an undismissable scrim
could trap the user if a result is ever dropped; the escape hatch drops
only the visual guard while the Rust-side flag keeps results applying
and still blocks a second picker.
FFmpeg's concat demuxer reported an unknown duration for chained GoPro/DJI segments when the generated manifest only listed file entries. Seeking past the first segment could then fail or stall even when the underlying filesystem, including NFS, supported normal file seeks. Probe each segment duration while building the concat manifest and emit duration entries when all probes succeed. This gives FFmpeg a usable timeline for cross-segment preview scrubbing without adding filesystem-specific path handling.
Manual movement paths updated only the current-frame property after seek or step operations, so the timeline slider moved while the current-time text stayed stale until playback advanced. Route step forward, step backward, relative seek, and debounced slider seek through sync_frame_display so frame count and time text update together after the new frame is rendered.
Use saved field ROI points as pose anchors before coverage clamping to damp short-frame panner jitter. Expose session and StitchJob APIs, add a CLI flag, and enable ROI stabilization for GUI exports with saved ROI anchors.
This was referenced Jul 19, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds ROI selection to the GUI and remove the browser selector.
Type of Change
Checklist
cargo test)cargo clippy --all-targets -- -D warnings && cargo fmt --all -- --check)Screenshots (if applicable)