Show screenshares as fullscreen #5402

mahibi · 2025-09-24T13:34:01Z

resolve no fullscreen-screensharing with muliple participants #1397

This PR will introduce to show screenshares as fullscreen

Whenever some shares the screen it will be shown as fullscreen
Screenshare can be closed by X button
Participants who share a screen have the screenshare icon which can be clicked
In screenshare view:
- zoom by pinching, move by dragging
- single tap to show the header with participant name and close button
- double tap to reset zoom level

This PR will also:

switch to software based rendering for screenshares so high resolution streams are supported for all devices.
- Other streams will continue to use hardware based rendering.
modify the call screen design, e.g.
- remove animated call controls so the screenshare buttons are not overlapped. Further improvements may come in the future.
migrate more code to Jetpack Compose
simplify classes around call architecture (e.g. by replacing classic observer patterns with kotlin flows)

Followup improvements should be:

for now, dragging of the self video view is removed. It will be re-implemented whenever the WebRtc lib is updated so we can make use of TextureViewRenderer.
- with TextureViewRenderer it should also be possible to have rounded corners for the self video view

🖼️ Video

ScreenshareFullscreenHandling.mp4

🚧 TODO

fix test testStateSentWithExponentialBackoffWhenAnotherParticipantAdded

🏁 Checklist

⛑️ Tests (unit and/or integration) are included or not needed
🔖 Capability is checked or not needed
🔙 Backport requests are created or not needed: /backport to stable-xx.x
📅 Milestone is set
🌸 PR title is meaningful (if it should be in the changelog: is it meaningful to users?)

was unused.. Signed-off-by: Marcel Hibbe <[email protected]>

Signed-off-by: Marcel Hibbe <[email protected]>

Classes related to calls are build with classic observer patterns. This is quite hard to debug and to keep an overview. This PR tries to simplify the architecture, e.g. by replacing observer patterns with kotlin flows. Signed-off-by: Marcel Hibbe <[email protected]>

Signed-off-by: Marcel Hibbe <[email protected]>

just as temporary solution? Signed-off-by: Marcel Hibbe <[email protected]>

Signed-off-by: Marcel Hibbe <[email protected]>

…rwrite if it was not offered because it's a voiceOnly call) Signed-off-by: Marcel Hibbe <[email protected]>

add log in getParticipant if not found Signed-off-by: Marcel Hibbe <[email protected]>

Signed-off-by: Marcel Hibbe <[email protected]>

keep to use hardware encoder/decoder for other streams. with hardware encoder/decoder, the stream might freeze for high res screenshares. Crash log snippet: 2025-11-07 10:37:44.750 9405-11072 MediaCodec com.nextcloud.talk2 I (0xb4000070580ed870) configure surface(0xb4000071f852e4f0) crypto(0x0) flags(0) 2025-11-07 10:37:44.750 9405-11072 MediaCodec com.nextcloud.talk2 D (0xb4000070580ed870) configure format: AMessage(what = 0x00000000) = { string mime = "video/avc" int32_t width = 4096 int32_t height = 2304 } 2025-11-07 10:37:44.750 9405-11072 CCodecConfig com.nextcloud.talk2 D no recognized params in: Dict { string vendor.dolby.filter_mimeType.value = "video/avc" } 2025-11-07 10:37:44.750 9405-11072 MediaCodec com.nextcloud.talk2 D pass vendor.dolby.filter_mimeType.value: video/avc 2025-11-07 10:37:44.751 9405-11860 SurfaceUtils com.nextcloud.talk2 D connecting to surface 0xb4000071f852e500, reason connectToSurface 2025-11-07 10:37:44.751 9405-11860 MediaCodec com.nextcloud.talk2 I [c2.qti.avc.decoder] setting surface generation to 9630833 2025-11-07 10:37:44.751 9405-11860 SurfaceUtils com.nextcloud.talk2 D disconnecting from surface 0xb4000071f852e500, reason connectToSurface(reconnect) 2025-11-07 10:37:44.751 9405-11860 Surface com.nextcloud.talk2 D Surface::disconnect 2025-11-07 10:37:44.751 9405-11860 BufferQueueProducer com.nextcloud.talk2 D [SurfaceTexture-1-9405-112](id:24bd00000080,api:3,p:9405,c:9405) disconnect: api 3 2025-11-07 10:37:44.751 9405-11860 SurfaceUtils com.nextcloud.talk2 D connecting to surface 0xb4000071f852e4f0, reason connectToSurface(reconnect-with-listener) 2025-11-07 10:37:44.751 9405-11861 CCodec com.nextcloud.talk2 D [c2.qti.avc.decoder] buffers are bound to CCodec for this session 2025-11-07 10:37:44.751 9405-11861 CCodecConfig com.nextcloud.talk2 D no c2 equivalents for native-window 2025-11-07 10:37:44.752 9405-11861 CCodecConfig com.nextcloud.talk2 D no c2 equivalents for native-window-generation 2025-11-07 10:37:44.752 9405-11861 CCodecConfig com.nextcloud.talk2 D no c2 equivalents for flags 2025-11-07 10:37:44.753 9405-11861 CCodecConfig com.nextcloud.talk2 D c2 config diff is c2::u32 raw.size.height = 2304 c2::u32 raw.size.width = 4096 2025-11-07 10:37:44.754 9405-11861 Codec2Client com.nextcloud.talk2 W query -- param skipped: index = 1107298332. 2025-11-07 10:37:44.754 9405-11861 CCodec com.nextcloud.talk2 D encoding statistics level = 0 2025-11-07 10:37:44.754 9405-11861 CCodec com.nextcloud.talk2 D setup formats input: AMessage(what = 0x00000000) = { int32_t feature-secure-playback = 0 int32_t frame-rate = 30 int32_t height = 2304 int32_t max-input-size = 13271040 string mime = "video/avc" int32_t priority = 1 int32_t profile = 8 int32_t width = 4096 Rect crop(0, 0, 4095, 2303) } 2025-11-07 10:37:44.754 9405-11861 CCodec com.nextcloud.talk2 D setup formats output: AMessage(what = 0x00000000) = { int32_t android._color-format = 0 int32_t android._video-scaling = 1 int32_t android._dataspace = 281411584 int32_t color-standard = 6 int32_t color-range = 2 int32_t color-transfer = 3 int32_t sar-height = 1 int32_t rotation-degrees = 0 Buffer hdr-static-info = { 00000000: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 00000010: 00 00 00 00 00 00 00 00 00 ......... } int32_t sar-width = 1 Rect crop(0, 0, 4095, 2303) int32_t width = 4096 int32_t feature-secure-playback = 0 int32_t frame-rate = 30 int32_t height = 2304 int32_t max-height = 2160 int32_t max-width = 4096 string mime = "video/raw" int32_t priority = 1 int32_t color-format = 2130708361 } 2025-11-07 10:37:44.754 9405-11861 CCodecConfig com.nextcloud.talk2 I query failed after returning 17 values (BAD_INDEX) 2025-11-07 10:37:44.755 9405-11861 nextcloud.talk2 com.nextcloud.talk2 E Failed to query component interface for required system resources: 6 2025-11-07 10:37:44.755 9405-11072 MediaCodec com.nextcloud.talk2 I (0xb4000070580ed870) start 2025-11-07 10:37:44.759 9405-11860 MediaCodec com.nextcloud.talk2 E Codec reported err 0xfffffff4/NO_MEMORY, actionCode 0, while in state 5/STARTING 2025-11-07 10:37:44.759 9405-11860 SurfaceUtils com.nextcloud.talk2 D disconnecting from surface 0xb4000071f852e500, reason disconnectFromSurface 2025-11-07 10:37:44.759 9405-11860 Surface com.nextcloud.talk2 D Surface::disconnect 2025-11-07 10:37:44.759 9405-11860 BufferQueueProducer com.nextcloud.talk2 D [SurfaceTexture-1-9405-112](id:24bd00000080,api:3,p:9405,c:9405) disconnect: api 3 2025-11-07 10:37:44.775 9405-11072 org.webrtc.Logging com.nextcloud.talk2 E AndroidVideoDecoder: initDecode failed 2025-11-07 10:37:44.775 9405-11072 org.webrtc.Logging com.nextcloud.talk2 E AndroidVideoDecoder: android.media.MediaCodec$CodecException: 2025-11-07 10:37:44.775 9405-11072 org.webrtc.Logging com.nextcloud.talk2 E AndroidVideoDecoder: android.media.MediaCodec$CodecException: (Ask Gemini) at android.media.MediaCodec.native_start(Native Method) at android.media.MediaCodec.start(MediaCodec.java:2798) at org.webrtc.MediaCodecWrapperFactoryImpl$MediaCodecWrapperImpl.start(MediaCodecWrapperFactoryImpl.java:42) at org.webrtc.AndroidVideoDecoder.initDecodeInternal(AndroidVideoDecoder.java:182) at org.webrtc.AndroidVideoDecoder.initDecode(AndroidVideoDecoder.java:146) 2025-11-07 10:37:44.775 9405-11072 org.webrtc.Logging com.nextcloud.talk2 I AndroidVideoDecoder: release 2025-11-07 10:37:44.775 9405-11072 org.webrtc.Logging com.nextcloud.talk2 I AndroidVideoDecoder: release: Decoder is not running. 2025-11-07 10:37:44.775 9405-11072 org.webrtc.Logging com.nextcloud.talk2 I SurfaceTextureHelper: stopListening() 2025-11-07 10:37:44.775 9405-11072 org.webrtc.Logging com.nextcloud.talk2 I SurfaceTextureHelper: dispose() 2025-11-07 10:37:44.778 9405-11864 org.webrtc.Logging com.nextcloud.talk2 I EglBase14Impl: Using OpenGL ES version 2 Signed-off-by: Marcel Hibbe <[email protected]>

Signed-off-by: Marcel Hibbe <[email protected]>

to examine why loading spinners keep showing for huge calls with many participants Signed-off-by: Marcel Hibbe <[email protected]>

Signed-off-by: Marcel Hibbe <[email protected]>

This happens e.g. when they don't have permissions to send audio and video. The fix is to set isConnected = true in the ParticipantUiState creation. At first it felt like setting isConnected to true is a hack, but it should be okay as it's immediately changed by the peerConnection handling for the other cases. Signed-off-by: Marcel Hibbe <[email protected]>

Signed-off-by: Marcel Hibbe <[email protected]>

for small tiles with long participant names there is not enough space at the bottom.. revert 516b53f Signed-off-by: Marcel Hibbe <[email protected]>

Signed-off-by: Marcel Hibbe <[email protected]>

github-actions · 2025-11-18T12:46:13Z

APK file: https://www.kaminsky.me/nc-dev/android-artifacts/5402.apk

To test this change/fix you can simply download above APK file and install and test it in parallel to your existing Nextcloud app.

github-actions · 2025-11-18T12:48:19Z

Codacy

Lint

Type	master	PR
Warnings	99	99
Errors	0	0

SpotBugs

Category	Base	New
Bad practice	6	6
Correctness	10	10
Dodgy code	54	54
Internationalization	3	3
Malicious code vulnerability	3	3
Performance	4	4
Security	1	1
Total	81	81

mahibi · 2025-11-18T15:21:42Z

The test testStateSentWithExponentialBackoffWhenAnotherParticipantAdded fails for now:

org.mockito.exceptions.verification.TooManyActualInvocations: 
messageSender.send(
    NCSignalingMessage(from=null, to=null, type=unmute, payload=NCMessagePayload(type=null, sdp=null, nick=null, iceCandidate=null, name=audio, state=null, timestamp=null, reaction=null), roomType=video, sid=null, prefix=null),
    "theSessionId"
);
Wanted 4 times:
-> at com.nextcloud.talk.call.MessageSender.send(MessageSender.java:63)
But was 5 times:
-> at com.nextcloud.talk.call.LocalStateBroadcasterMcu.sendState(LocalStateBroadcasterMcu.java:115)
-> at com.nextcloud.talk.call.LocalStateBroadcasterMcu.sendState(LocalStateBroadcasterMcu.java:115)
-> at com.nextcloud.talk.call.LocalStateBroadcasterMcu.sendState(LocalStateBroadcasterMcu.java:115)
-> at com.nextcloud.talk.call.LocalStateBroadcasterMcu.sendState(LocalStateBroadcasterMcu.java:115)
-> at com.nextcloud.talk.call.LocalStateBroadcasterMcu.sendState(LocalStateBroadcasterMcu.java:115)


	at app//com.nextcloud.talk.call.MessageSender.send(MessageSender.java:63)
	at app//com.nextcloud.talk.call.LocalStateBroadcasterMcuTest.testStateSentWithExponentialBackoffWhenAnotherParticipantAdded(LocalStateBroadcasterMcuTest.kt:370)

rapterjet2004

It works for me

mahibi · 2025-11-21T12:48:05Z

For now, @danxuliu also had no idea why the test fails & plans to have a closer look next week

mahibi added this to the 23.0.0 milestone Sep 24, 2025

mahibi self-assigned this Sep 24, 2025

mahibi added the 2. developing Work in progress label Sep 24, 2025

mahibi force-pushed the noid/simplifyCallArchitcture branch 2 times, most recently from 02d5796 to b3e4179 Compare October 17, 2025 13:34

mahibi changed the title ~~Noid/simplify call architcture~~ simplify call architecture + enlarge screenshares Oct 17, 2025

mahibi force-pushed the noid/simplifyCallArchitcture branch 2 times, most recently from 540b336 to 17424b4 Compare October 27, 2025 14:22

mahibi force-pushed the noid/simplifyCallArchitcture branch 4 times, most recently from 852e38d to 6bb9146 Compare November 4, 2025 13:43

mahibi force-pushed the noid/simplifyCallArchitcture branch 2 times, most recently from 9e811b9 to bb39632 Compare November 7, 2025 11:09

mahibi mentioned this pull request Nov 10, 2025

UI gets stuck in "ringing" state while media stream connects successfully #5451

Open

mahibi force-pushed the noid/simplifyCallArchitcture branch 2 times, most recently from 83679e8 to d8f7940 Compare November 12, 2025 15:26

This was referenced Nov 13, 2025

share screen from android #1530

Open

Voice call disconnects, when user reconnects he sees himself doubled. #5539

Open

mahibi force-pushed the noid/simplifyCallArchitcture branch 2 times, most recently from bd59919 to c10bda6 Compare November 17, 2025 21:50

mahibi added 9 commits November 18, 2025 11:40

remove ParticipantDisplayItemNotifier

cedab07

was unused.. Signed-off-by: Marcel Hibbe <[email protected]>

Rename .java to .kt

6425a74

Signed-off-by: Marcel Hibbe <[email protected]>

simplify architecture for call participant handling

4d5c799

Signed-off-by: Marcel Hibbe <[email protected]>

add share screen icon and logic

442a4b1

Signed-off-by: Marcel Hibbe <[email protected]>

remove animations for call controls + remove fullscreen grid

4c95fb9

Signed-off-by: Marcel Hibbe <[email protected]>

remove margin

8e74ffa

Signed-off-by: Marcel Hibbe <[email protected]>

fix self video zorder

9b036bc

Signed-off-by: Marcel Hibbe <[email protected]>

change top bar

575eadb

just as temporary solution? Signed-off-by: Marcel Hibbe <[email protected]>

mahibi added 16 commits November 18, 2025 11:40

enable to receive screenshare in audiocalls

d15d71d

Signed-off-by: Marcel Hibbe <[email protected]>

simplify ParticipantTile

9c72c6f

Signed-off-by: Marcel Hibbe <[email protected]>

request big size for avatars in call

feab800

Signed-off-by: Marcel Hibbe <[email protected]>

set OfferToReceiveVideo only for the peer that shares the screen (ove…

dd8d9b1

…rwrite if it was not offered because it's a voiceOnly call) Signed-off-by: Marcel Hibbe <[email protected]>

remove unnecessary checks for doesParticipantExist

0a49aab

add log in getParticipant if not found Signed-off-by: Marcel Hibbe <[email protected]>

adapt tests (1 still failing)

48f5a49

Signed-off-by: Marcel Hibbe <[email protected]>

adapt tests

34e90b0

Signed-off-by: Marcel Hibbe <[email protected]>

format code

2f6b3ea

Signed-off-by: Marcel Hibbe <[email protected]>

adapt tests

7352736

Signed-off-by: Marcel Hibbe <[email protected]>

add logging

ffb87cf

to examine why loading spinners keep showing for huge calls with many participants Signed-off-by: Marcel Hibbe <[email protected]>

resolve warnings

276436f

Signed-off-by: Marcel Hibbe <[email protected]>

fallback to "guest" if participant name not known

b2768ad

Signed-off-by: Marcel Hibbe <[email protected]>

set raise hand to upper right corner again

952d41e

for small tiles with long participant names there is not enough space at the bottom.. revert 516b53f Signed-off-by: Marcel Hibbe <[email protected]>

better fallback to "guest" if participant name not known

6c7f054

Signed-off-by: Marcel Hibbe <[email protected]>

mahibi force-pushed the noid/simplifyCallArchitcture branch from c10bda6 to 6c7f054 Compare November 18, 2025 12:34

mahibi requested a review from rapterjet2004 November 18, 2025 12:35

mahibi added 3. to review Waiting for reviews and removed 2. developing Work in progress labels Nov 18, 2025

mahibi marked this pull request as ready for review November 18, 2025 12:44

mahibi changed the title ~~simplify call architecture + enlarge screenshares~~ Show screenshares as fullscreen Nov 18, 2025

mahibi requested a review from danxuliu November 18, 2025 15:06

mahibi mentioned this pull request Nov 18, 2025

sort participants in call #5562

Draft

8 tasks

rapterjet2004 approved these changes Nov 19, 2025

View reviewed changes

mahibi mentioned this pull request Nov 20, 2025

Shared screen are only shown as large as all the other users #5561

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Show screenshares as fullscreen #5402

Show screenshares as fullscreen #5402

Uh oh!

mahibi commented Sep 24, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Nov 18, 2025

Uh oh!

github-actions bot commented Nov 18, 2025

Uh oh!

mahibi commented Nov 18, 2025

Uh oh!

rapterjet2004 left a comment

Uh oh!

mahibi commented Nov 21, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Show screenshares as fullscreen #5402

Are you sure you want to change the base?

Show screenshares as fullscreen #5402

Uh oh!

Conversation

mahibi commented Sep 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🖼️ Video

🚧 TODO

🏁 Checklist

Uh oh!

github-actions bot commented Nov 18, 2025

Uh oh!

github-actions bot commented Nov 18, 2025

Codacy

Lint

SpotBugs

Uh oh!

mahibi commented Nov 18, 2025

Uh oh!

rapterjet2004 left a comment

Choose a reason for hiding this comment

Uh oh!

mahibi commented Nov 21, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

mahibi commented Sep 24, 2025 •

edited

Loading