Skip to content

Show screenshares as fullscreen - #5402

Merged
mahibi merged 53 commits into
masterfrom
noid/simplifyCallArchitcture
Nov 27, 2025
Merged

Show screenshares as fullscreen#5402
mahibi merged 53 commits into
masterfrom
noid/simplifyCallArchitcture

Conversation

@mahibi

@mahibi mahibi commented Sep 24, 2025

Copy link
Copy Markdown
Collaborator

This PR will introduce to show screenshares as fullscreen

  • Whenever some shares the screen it will be shown as fullscreen
  • Screenshare can be closed by X button
  • Participants who share a screen have the screenshare icon which can be clicked
  • In screenshare view:
    • zoom by pinching, move by dragging
    • single tap to show the header with participant name and close button
    • double tap to reset zoom level

This PR will also:

  • switch to software based rendering for screenshares so high resolution streams are supported for all devices.
    • Other streams will continue to use hardware based rendering.
  • modify the call screen design, e.g.
    • remove animated call controls so the screenshare buttons are not overlapped. Further improvements may come in the future.
  • migrate more code to Jetpack Compose
  • simplify classes around call architecture (e.g. by replacing classic observer patterns with kotlin flows)

Followup improvements should be:

  • for now, dragging of the self video view is removed. It will be re-implemented whenever the WebRtc lib is updated so we can make use of TextureViewRenderer.
    • with TextureViewRenderer it should also be possible to have rounded corners for the self video view

🖼️ Video

ScreenshareFullscreenHandling.mp4

🚧 TODO

  • fix test testStateSentWithExponentialBackoffWhenAnotherParticipantAdded

🏁 Checklist

  • ⛑️ Tests (unit and/or integration) are included or not needed
  • 🔖 Capability is checked or not needed
  • 🔙 Backport requests are created or not needed: /backport to stable-xx.x
  • 📅 Milestone is set
  • 🌸 PR title is meaningful (if it should be in the changelog: is it meaningful to users?)

@mahibi mahibi added this to the 23.0.0 milestone Sep 24, 2025
@mahibi mahibi self-assigned this Sep 24, 2025
@mahibi mahibi added the 2. developing Work in progress label Sep 24, 2025
@mahibi
mahibi force-pushed the noid/simplifyCallArchitcture branch 2 times, most recently from 02d5796 to b3e4179 Compare October 17, 2025 13:34
@mahibi mahibi changed the title Noid/simplify call architcture simplify call architecture + enlarge screenshares Oct 17, 2025
@mahibi
mahibi force-pushed the noid/simplifyCallArchitcture branch 2 times, most recently from 540b336 to 17424b4 Compare October 27, 2025 14:22
@mahibi
mahibi force-pushed the noid/simplifyCallArchitcture branch 4 times, most recently from 852e38d to 6bb9146 Compare November 4, 2025 13:43
@mahibi
mahibi force-pushed the noid/simplifyCallArchitcture branch 2 times, most recently from 9e811b9 to bb39632 Compare November 7, 2025 11:09
@mahibi
mahibi force-pushed the noid/simplifyCallArchitcture branch 2 times, most recently from 83679e8 to d8f7940 Compare November 12, 2025 15:26
@mahibi
mahibi force-pushed the noid/simplifyCallArchitcture branch 3 times, most recently from c10bda6 to 6c7f054 Compare November 18, 2025 12:34
@mahibi
mahibi requested a review from rapterjet2004 November 18, 2025 12:35
@mahibi mahibi added 3. to review Waiting for reviews and removed 2. developing Work in progress labels Nov 18, 2025
@mahibi
mahibi marked this pull request as ready for review November 18, 2025 12:44
@mahibi mahibi changed the title simplify call architecture + enlarge screenshares Show screenshares as fullscreen Nov 18, 2025
@mahibi
mahibi requested a review from danxuliu November 18, 2025 15:06
@mahibi mahibi mentioned this pull request Nov 18, 2025
5 tasks
@mahibi

mahibi commented Nov 18, 2025

Copy link
Copy Markdown
Collaborator Author

The test testStateSentWithExponentialBackoffWhenAnotherParticipantAdded fails for now:

org.mockito.exceptions.verification.TooManyActualInvocations: 
messageSender.send(
    NCSignalingMessage(from=null, to=null, type=unmute, payload=NCMessagePayload(type=null, sdp=null, nick=null, iceCandidate=null, name=audio, state=null, timestamp=null, reaction=null), roomType=video, sid=null, prefix=null),
    "theSessionId"
);
Wanted 4 times:
-> at com.nextcloud.talk.call.MessageSender.send(MessageSender.java:63)
But was 5 times:
-> at com.nextcloud.talk.call.LocalStateBroadcasterMcu.sendState(LocalStateBroadcasterMcu.java:115)
-> at com.nextcloud.talk.call.LocalStateBroadcasterMcu.sendState(LocalStateBroadcasterMcu.java:115)
-> at com.nextcloud.talk.call.LocalStateBroadcasterMcu.sendState(LocalStateBroadcasterMcu.java:115)
-> at com.nextcloud.talk.call.LocalStateBroadcasterMcu.sendState(LocalStateBroadcasterMcu.java:115)
-> at com.nextcloud.talk.call.LocalStateBroadcasterMcu.sendState(LocalStateBroadcasterMcu.java:115)


	at app//com.nextcloud.talk.call.MessageSender.send(MessageSender.java:63)
	at app//com.nextcloud.talk.call.LocalStateBroadcasterMcuTest.testStateSentWithExponentialBackoffWhenAnotherParticipantAdded(LocalStateBroadcasterMcuTest.kt:370)

Signed-off-by: Marcel Hibbe <dev@mhibbe.de>
Signed-off-by: Marcel Hibbe <dev@mhibbe.de>
Signed-off-by: Marcel Hibbe <dev@mhibbe.de>
Signed-off-by: Marcel Hibbe <dev@mhibbe.de>
Signed-off-by: Marcel Hibbe <dev@mhibbe.de>
Signed-off-by: Marcel Hibbe <dev@mhibbe.de>
Signed-off-by: Marcel Hibbe <dev@mhibbe.de>
Signed-off-by: Marcel Hibbe <dev@mhibbe.de>
…rwrite if it was not offered because it's a voiceOnly call)

Signed-off-by: Marcel Hibbe <dev@mhibbe.de>
add log in getParticipant if not found

Signed-off-by: Marcel Hibbe <dev@mhibbe.de>
Signed-off-by: Marcel Hibbe <dev@mhibbe.de>
keep to use hardware encoder/decoder for other streams.

with hardware encoder/decoder, the stream might freeze for high res screenshares.
Crash log snippet:

2025-11-07 10:37:44.750  9405-11072 MediaCodec              com.nextcloud.talk2                  I  (0xb4000070580ed870) configure surface(0xb4000071f852e4f0) crypto(0x0) flags(0)
2025-11-07 10:37:44.750  9405-11072 MediaCodec              com.nextcloud.talk2                  D  (0xb4000070580ed870) configure format: AMessage(what = 0x00000000) = {
                                                                                                          string mime = "video/avc"
                                                                                                          int32_t width = 4096
                                                                                                          int32_t height = 2304
                                                                                                        }
2025-11-07 10:37:44.750  9405-11072 CCodecConfig            com.nextcloud.talk2                  D  no recognized params in: Dict {
                                                                                                      string vendor.dolby.filter_mimeType.value = "video/avc"
                                                                                                    }
2025-11-07 10:37:44.750  9405-11072 MediaCodec              com.nextcloud.talk2                  D  pass vendor.dolby.filter_mimeType.value: video/avc
2025-11-07 10:37:44.751  9405-11860 SurfaceUtils            com.nextcloud.talk2                  D  connecting to surface 0xb4000071f852e500, reason connectToSurface
2025-11-07 10:37:44.751  9405-11860 MediaCodec              com.nextcloud.talk2                  I  [c2.qti.avc.decoder] setting surface generation to 9630833
2025-11-07 10:37:44.751  9405-11860 SurfaceUtils            com.nextcloud.talk2                  D  disconnecting from surface 0xb4000071f852e500, reason connectToSurface(reconnect)
2025-11-07 10:37:44.751  9405-11860 Surface                 com.nextcloud.talk2                  D  Surface::disconnect
2025-11-07 10:37:44.751  9405-11860 BufferQueueProducer     com.nextcloud.talk2                  D  [SurfaceTexture-1-9405-112](id:24bd00000080,api:3,p:9405,c:9405) disconnect: api 3
2025-11-07 10:37:44.751  9405-11860 SurfaceUtils            com.nextcloud.talk2                  D  connecting to surface 0xb4000071f852e4f0, reason connectToSurface(reconnect-with-listener)
2025-11-07 10:37:44.751  9405-11861 CCodec                  com.nextcloud.talk2                  D  [c2.qti.avc.decoder] buffers are bound to CCodec for this session
2025-11-07 10:37:44.751  9405-11861 CCodecConfig            com.nextcloud.talk2                  D  no c2 equivalents for native-window
2025-11-07 10:37:44.752  9405-11861 CCodecConfig            com.nextcloud.talk2                  D  no c2 equivalents for native-window-generation
2025-11-07 10:37:44.752  9405-11861 CCodecConfig            com.nextcloud.talk2                  D  no c2 equivalents for flags
2025-11-07 10:37:44.753  9405-11861 CCodecConfig            com.nextcloud.talk2                  D  c2 config diff is   c2::u32 raw.size.height = 2304
                                                                                                      c2::u32 raw.size.width = 4096
2025-11-07 10:37:44.754  9405-11861 Codec2Client            com.nextcloud.talk2                  W  query -- param skipped: index = 1107298332.
2025-11-07 10:37:44.754  9405-11861 CCodec                  com.nextcloud.talk2                  D  encoding statistics level = 0
2025-11-07 10:37:44.754  9405-11861 CCodec                  com.nextcloud.talk2                  D  setup formats input: AMessage(what = 0x00000000) = {
                                                                                                      int32_t feature-secure-playback = 0
                                                                                                      int32_t frame-rate = 30
                                                                                                      int32_t height = 2304
                                                                                                      int32_t max-input-size = 13271040
                                                                                                      string mime = "video/avc"
                                                                                                      int32_t priority = 1
                                                                                                      int32_t profile = 8
                                                                                                      int32_t width = 4096
                                                                                                      Rect crop(0, 0, 4095, 2303)
                                                                                                    }
2025-11-07 10:37:44.754  9405-11861 CCodec                  com.nextcloud.talk2                  D  setup formats output: AMessage(what = 0x00000000) = {
                                                                                                      int32_t android._color-format = 0
                                                                                                      int32_t android._video-scaling = 1
                                                                                                      int32_t android._dataspace = 281411584
                                                                                                      int32_t color-standard = 6
                                                                                                      int32_t color-range = 2
                                                                                                      int32_t color-transfer = 3
                                                                                                      int32_t sar-height = 1
                                                                                                      int32_t rotation-degrees = 0
                                                                                                      Buffer hdr-static-info = {
                                                                                                        00000000:  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  ................
                                                                                                        00000010:  00 00 00 00 00 00 00 00  00                       .........
                                                                                                      }
                                                                                                      int32_t sar-width = 1
                                                                                                      Rect crop(0, 0, 4095, 2303)
                                                                                                      int32_t width = 4096
                                                                                                      int32_t feature-secure-playback = 0
                                                                                                      int32_t frame-rate = 30
                                                                                                      int32_t height = 2304
                                                                                                      int32_t max-height = 2160
                                                                                                      int32_t max-width = 4096
                                                                                                      string mime = "video/raw"
                                                                                                      int32_t priority = 1
                                                                                                      int32_t color-format = 2130708361
                                                                                                    }
2025-11-07 10:37:44.754  9405-11861 CCodecConfig            com.nextcloud.talk2                  I  query failed after returning 17 values (BAD_INDEX)
2025-11-07 10:37:44.755  9405-11861 nextcloud.talk2         com.nextcloud.talk2                  E  Failed to query component interface for required system resources: 6
2025-11-07 10:37:44.755  9405-11072 MediaCodec              com.nextcloud.talk2                  I  (0xb4000070580ed870) start
2025-11-07 10:37:44.759  9405-11860 MediaCodec              com.nextcloud.talk2                  E  Codec reported err 0xfffffff4/NO_MEMORY, actionCode 0, while in state 5/STARTING
2025-11-07 10:37:44.759  9405-11860 SurfaceUtils            com.nextcloud.talk2                  D  disconnecting from surface 0xb4000071f852e500, reason disconnectFromSurface
2025-11-07 10:37:44.759  9405-11860 Surface                 com.nextcloud.talk2                  D  Surface::disconnect
2025-11-07 10:37:44.759  9405-11860 BufferQueueProducer     com.nextcloud.talk2                  D  [SurfaceTexture-1-9405-112](id:24bd00000080,api:3,p:9405,c:9405) disconnect: api 3
2025-11-07 10:37:44.775  9405-11072 org.webrtc.Logging      com.nextcloud.talk2                  E  AndroidVideoDecoder: initDecode failed
2025-11-07 10:37:44.775  9405-11072 org.webrtc.Logging      com.nextcloud.talk2                  E  AndroidVideoDecoder: android.media.MediaCodec$CodecException:
2025-11-07 10:37:44.775  9405-11072 org.webrtc.Logging      com.nextcloud.talk2                  E  AndroidVideoDecoder: android.media.MediaCodec$CodecException:  (Ask Gemini)
                                                                                                    	at android.media.MediaCodec.native_start(Native Method)
                                                                                                    	at android.media.MediaCodec.start(MediaCodec.java:2798)
                                                                                                    	at org.webrtc.MediaCodecWrapperFactoryImpl$MediaCodecWrapperImpl.start(MediaCodecWrapperFactoryImpl.java:42)
                                                                                                    	at org.webrtc.AndroidVideoDecoder.initDecodeInternal(AndroidVideoDecoder.java:182)
                                                                                                    	at org.webrtc.AndroidVideoDecoder.initDecode(AndroidVideoDecoder.java:146)
2025-11-07 10:37:44.775  9405-11072 org.webrtc.Logging      com.nextcloud.talk2                  I  AndroidVideoDecoder: release
2025-11-07 10:37:44.775  9405-11072 org.webrtc.Logging      com.nextcloud.talk2                  I  AndroidVideoDecoder: release: Decoder is not running.
2025-11-07 10:37:44.775  9405-11072 org.webrtc.Logging      com.nextcloud.talk2                  I  SurfaceTextureHelper: stopListening()
2025-11-07 10:37:44.775  9405-11072 org.webrtc.Logging      com.nextcloud.talk2                  I  SurfaceTextureHelper: dispose()
2025-11-07 10:37:44.778  9405-11864 org.webrtc.Logging      com.nextcloud.talk2                  I  EglBase14Impl: Using OpenGL ES version 2

Signed-off-by: Marcel Hibbe <dev@mhibbe.de>
Signed-off-by: Marcel Hibbe <dev@mhibbe.de>
Signed-off-by: Marcel Hibbe <dev@mhibbe.de>
Signed-off-by: Marcel Hibbe <dev@mhibbe.de>
to examine why loading spinners keep showing for huge calls with many participants

Signed-off-by: Marcel Hibbe <dev@mhibbe.de>
Signed-off-by: Marcel Hibbe <dev@mhibbe.de>
@mahibi
mahibi force-pushed the noid/simplifyCallArchitcture branch from 14097bf to 328f2a9 Compare November 27, 2025 10:31
This happens e.g. when they don't have permissions to send audio and video.

The fix is to set isConnected = true in the ParticipantUiState creation.

At first it felt like setting isConnected to true is a hack, but it should be okay as it's immediately changed by the peerConnection handling for the other cases.

Signed-off-by: Marcel Hibbe <dev@mhibbe.de>
Signed-off-by: Marcel Hibbe <dev@mhibbe.de>
for small tiles with long participant names there is not enough space at the bottom..

revert 516b53f

Signed-off-by: Marcel Hibbe <dev@mhibbe.de>
Signed-off-by: Marcel Hibbe <dev@mhibbe.de>
This needs to be investigated but the PR needs to be merged to move forward.

Signed-off-by: Marcel Hibbe <dev@mhibbe.de>
@mahibi
mahibi force-pushed the noid/simplifyCallArchitcture branch from 328f2a9 to ee79d67 Compare November 27, 2025 13:33
@github-actions

Copy link
Copy Markdown
Contributor

APK file: https://www.kaminsky.me/nc-dev/android-artifacts/5402.apk

qrcode

To test this change/fix you can simply download above APK file and install and test it in parallel to your existing Nextcloud app.

@github-actions

Copy link
Copy Markdown
Contributor

Codacy

Lint

TypemasterPR
Warnings9999
Errors00

SpotBugs

CategoryBaseNew
Bad practice66
Correctness1010
Dodgy code5454
Internationalization33
Malicious code vulnerability33
Performance44
Security11
Total8181

@mahibi
mahibi merged commit a490edf into master Nov 27, 2025
17 of 18 checks passed
@mahibi
mahibi deleted the noid/simplifyCallArchitcture branch November 27, 2025 15:03
@mahibi mahibi mentioned this pull request Feb 13, 2026
6 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

3. to review Waiting for reviews

Projects

None yet

Development

Successfully merging this pull request may close these issues.

no fullscreen-screensharing with muliple participants

2 participants