Skip to content

fix(screen): connect remote screens with trickle ICE - #40

Draft
fergusean wants to merge 4 commits into
leepokai:devfrom
fergusean:fergusean/linux-remote-screen
Draft

fergusean wants to merge 4 commits into
leepokai:devfrom
fergusean:fergusean/linux-remote-screen

Conversation

@fergusean

@fergusean fergusean commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Stream bounded, ordered ICE candidates in both directions when the phone requests trickle and the helper advertises support; retain complete-SDP negotiation otherwise. Remote screens preserve established media when their candidate signaling channel ends during a route change, relay reconnect or brief background trip. Signaling loss remains fatal before ICE connects or both directions complete. Reconnect waits for a usable host route and surfaces terminal upgrade/stopped-channel errors.

macOS and Linux helpers report guarded native connection state, and stale callbacks cannot close or revive replacement sessions. Linux ignores native empty per-transport gathering markers and sends global completion only after every transport finishes; incoming candidate validation and limits remain strict. This prevents GStreamer 1.28.2 from ending a working viewer at gathering completion.

The independent Linux H.264 payload, 8-bit 4:2:0 and system-clock capture fixes landed separately in #104. This PR retains the Tokio callback context and socket/session lifetime needed for trickle; existing keepalive behavior is unchanged. The Linux portal remains lazy, supports permission requests and sends periodic status. Desktop, terminal and Watch clients have no remote-screen viewer; host helper and permission behavior is shared.

Dependencies

#103 and #104 are merged. This head is rebased onto upstream dev at acd8998a, inheriting version 2.17.2 without a feature version bump.

Validation

  • Current host CI passes formatting, all-target Clippy and the complete suite on Linux, macOS and Windows. Linux helper CI and macOS helper CI pass.
  • Current iOS CI times out after one minute in the unchanged repeated-signaling-loss retry-budget test. Unrelated tests also take 52–58 seconds; the initiating cause remains unproved. The combined local Swift suite passes 243 cases / 247 runs. A same-head rerun requires repository admin rights and was refused. Assertions, failure injection, retry budgets and timeouts remain unchanged.
  • Exact-head Linux helper source passes all fifteen native checks on GStreamer 1.22.0 in Debian Bookworm with the repository's Rust 1.99 toolchain. This exercises native pipeline/ICE behavior in a headless container, without portal capture. Native completion regressions preserve a live viewer, subsequent candidates, exactly one aggregate completion and strict malformed-wire rejection; removing the narrow marker guard makes the behavior regressions fail.
  • A native macOS WebRTC viewer connects to the unchanged production Linux helper over direct, forced TURN UDP, TCP and TLS routes. Selected candidate stats confirm the actual relay protocol, including TLS on port 443. Viewer UDP is process-denied for TCP/TLS. Candidate-free offers and answers precede completion in both directions; negotiation takes 558–1,202 ms through answer across these routes.
  • On all four Linux routes, terminating the actual candidate WebSocket after both directions complete preserves the same selected ICE pair for ten seconds. Decoded frame counts increase and exact keyboard markers sent before/after reach an owned test window. These are native viewer checks, separate from the physical iPhone checks below.
  • The installed signed macOS helper, connected through an isolated JSON-RPC fixture to the native viewer, passes direct and forced TURN UDP/TCP/TLS with decoded frames and viewport data-channel responses. Both helper and viewer have UDP denied for TCP/TLS; selected stats confirm the relay protocol and TLS 443. TURN offers/answers precede completion. The fast direct helper gathered before the answer RPC returned. No Mac keyboard/pointer injection is claimed.
  • The actual physical iPhone displays live Linux green-to-purple pattern changes and delivers exact lowercase/uppercase keyboard markers. During a normal background trip, its registered key is disconnected from signaling for approximately fourteen seconds of a twenty-second sample while the media viewer count remains one. The updated pattern is visible on return and further exact input reaches the test window. Physical selected-pair stats were not collected.
  • Older-wire fallback passes direct and TLS 443 with blocked UDP: an omitted phone trickle capability negotiates complete SDP against the current Linux host/helper without candidate subscription; an omitted host trickle field negotiates complete SDP against the current macOS helper without candidate notifications. Both preserve decoded frames and data-channel operation using the app's existing ten-second gathering deadline. These use current native libraries with older wire shapes, rather than actual older physical binaries.
  • The production screen component trees match this PR head. Verification changes no product source or installed runtime. Original identities, grants, keys and dirty worktree contents are preserved; temporary devices, windows, processes and introduced container image are cleaned up. Documentation whitespace, fences and local links are checked non-visually. Screenshots remain private for manual attachment.

This PR remains draft. Remaining acceptance: the upstream iOS CI rerun, physical iPhone forced-relay/UDP-blocked TCP/TLS selected-pair measurements, and the complete old/new physical app/host/helper binary matrix. The native results above establish their stated routes and wire compatibility scopes.

What's New

zh-Hant

  • 遠端螢幕已連線後,即使訊號通道短暫中斷,也能維持畫面連線。

en-US

  • Keep connected remote screens alive through brief signaling interruptions.

@vercel

vercel Bot commented Oct 3, 2026

Copy link
Copy Markdown

@fergusean is attempting to deploy a commit to the leeporic Team on Vercel.

A member of the Team first needs to authorize it.

@fergusean
fergusean force-pushed the fergusean/linux-remote-screen branch 2 times, most recently from cb64f4d to a4d5d99 Compare October 5, 2026 19:00
@leepokai
leepokai changed the base branch from main to dev October 6, 2026 08:32
@leepokai

leepokai commented Oct 6, 2026

Copy link
Copy Markdown
Owner

Great investigation, thanks Sean. fmt, clippy, the six tests and the release build all pass on Linux merged with dev (I've retargeted this PR to dev, where external PRs now land), and the Tokio enter() fix is spot on.

One concern before merging: answering on the first relay candidate can drop the TURN TCP/TLS:443 candidates that arrive later. Since the SDP is non-trickle, networks that block UDP would lose that fallback. Could you wait for a TLS relay candidate (or use a short cap) before answering?

@fergusean
fergusean marked this pull request as draft October 7, 2026 09:09
@fergusean
fergusean force-pushed the fergusean/linux-remote-screen branch from a4d5d99 to caa3993 Compare October 9, 2026 07:12
@fergusean fergusean changed the title fix(linux): stabilize remote screen streaming fix(screen): connect remote screens with trickle ICE Oct 9, 2026
@leepokai

leepokai commented Oct 9, 2026

Copy link
Copy Markdown
Owner

I reran the failed iOS Kit and host workflows against the current head (caa3993) to distinguish transient failures from merge blockers.

  • Windows host retry still fails all-target Clippy: Signaling.tx, Signaling.outgoing, and Screen::helper_candidate are unused on Windows. Please make the platform-specific code compile cleanly under the existing -D warnings check; the other host jobs were cancelled by the matrix failure.
  • iOS Kit retry still fails offlineTapReconnectsBeforeGivingUp(). The two signaling tests listed in the first attempt were not listed as failures on this retry, so their first failures alone do not establish deterministic regressions.

Keeping this draft unmerged until these failures are explained/fixed and the full required matrix passes. The documented physical-device / UDP-blocked TURN fallback validation is also still outstanding.

@fergusean

fergusean commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor Author

Sorry, this is a WIP! I haven't had a chance to review or test, it just got pushed with my other branch updates :)

@leepokai leepokai left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for pushing this forward! The compatibility design is right: trickle only when the phone asks and the helper advertises it, falling back to full SDP otherwise. The candidate limits/validation and the helper-replacement race tests are great too. Since it's still a WIP draft, here's what I'd need before merging:

  1. Signaling loss tears down working media. The candidate subscription's lifetime is tied to the whole screen session on both sides. On the host, Subscription::drop → fail_signaling → screen.close(session) (host/src/screen/signaling.rs), and channel teardown aborts every subscription. On iOS, ScreenSignaling throws unreachable when the stream ends and ScreenSession tears down and reconnects. So a relay ⇄ LAN route change, a relay re-handshake, or a brief background trip closes a healthy P2P stream, and reconnects can exhaust their 3 attempts while the link is still down. Please treat signaling loss as fatal only until ICE connects (or both sides send complete), then quietly drop the subscription and keep the media.
  2. Rebase onto dev. There are real conflicts in apps/screen-linux/src/main.rs and portal.rs: dev made the portal lazy (Arc<Mutex<Option<Portal>>>), added requestPermission, and sends status every 3 s. The sessions.rs/link.rs split will need to route dev's direct h.sessions draining through Sessions::close_all() and add "trickle": true to the new status. Regenerate project.pbxproj with xcodegen rather than merging it.
  3. End-to-end validation on real devices, as noted in the PR: direct, relay, UDP-blocked TURN over TCP/TLS 443, and old/new app × host combinations.

Smaller things:

  • CLAUDE.md still says "non-trickle SDP relayed by screenOffer"; docs/architecture/file-structure.md doesn't list apps/screen-macos-tests/.
  • viewer.ice.expires_at = 0 doubles as a "session failed" flag; an explicit state enum would be clearer.
  • iOS local errors are wrapped as HostError.http(400, …).
  • The Linux streaming fixes (H.264 payload type, 8-bit 4:2:0, provide-clock=false) are independent of trickle ICE; a separate PR would be easier to review and land quickly.

@fergusean
fergusean force-pushed the fergusean/linux-remote-screen branch from d8796de to 5bd6092 Compare October 12, 2026 20:12
@fergusean
fergusean force-pushed the fergusean/linux-remote-screen branch 2 times, most recently from 49a3168 to d6c0e79 Compare October 13, 2026 09:28
fergusean added a commit to fergusean/Codync that referenced this pull request Oct 13, 2026
Use the viewer's H.264 payload and 8-bit 4:2:0 encoder input, negotiate
the answer before capture starts, and keep PipeWire timestamps on the
system clock. These fixes work with complete SDP independently of
trickle ICE and are split from PR leepokai#40 as requested.

Co-authored-by: Codex <codex@openai.com>
@fergusean

fergusean commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor Author

Code updated but I haven't tested it with a real device yet, so keeping it a draft until I do.

@fergusean
fergusean force-pushed the fergusean/linux-remote-screen branch 4 times, most recently from 53e0a20 to 888ab82 Compare October 16, 2026 07:09
fergusean and others added 4 commits October 16, 2026 08:17
Negotiate bounded candidate streams on supported endpoints and retain
connected media through signaling outages. Recover on usable routes,
surface terminal channel errors, and guard helper replacement lifetimes.

Independent Linux video corrections and upstream CI maintenance are
separate PRs; this feature preserves the existing full-SDP fallback.

Co-authored-by: Codex <codex@openai.com>
Ignore GStreamer's empty per-transport end marker before validating
wire candidates. Finish the ordered exchange only when every native
transport has completed gathering.

Cover live viewer preservation, native global completion and strict
wire validation with regressions.

Co-authored-by: Codex <codex@openai.com>
Keep the narrowed remote-screen PR clean under the repository
whitespace check. Test behavior is unchanged.

Co-authored-by: Codex <codex@openai.com>
Construct the existing complete-SDP capture fixture with explicit
non-trickle mode and its candidate buffer after integrating the
upstream capture regressions.

Co-authored-by: Codex <codex@openai.com>
@fergusean
fergusean force-pushed the fergusean/linux-remote-screen branch from 888ab82 to ad54454 Compare October 16, 2026 12:19
@leepokai leepokai added P3: low Nice to have: features, questions, polish. bug Something isn't working labels Oct 18, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working P3: low Nice to have: features, questions, polish.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants