Repository navigation
Android: put each game screen and the menu on any display (gl and vulkan) - #19975
Open
Davey-Hughes wants to merge 25 commits into
Open
Davey-Hughes wants to merge 25 commits into
Davey-Hughes wants to merge 25 commits into
Conversation
A uint setting's list can leave values out: Mouse Index lists only the mice there are, and Keyboard Index lists only the keyboards there are and the one chosen now. The pick was worked out from the entry's position, as the minimum plus whole steps, so a value left out shifted every pick after it: with mice at #1 and #3, picking #3 set #2. Each entry now carries the value it lists, and the pick sets that one. The network settings harness passes each entry's entry_idx, which carries its value, as the menu does, and picks a Mouse Index past an index with no mouse and a Keyboard Index past an index with no keyboard.
gl2 published whether it can draw views only from gl2_frame(), so a new gl2 reported neither views nor screen windows until it had drawn, and the frame path took its first frame for windows turned off. gl2 now publishes it at init, as vulkan does.
From API 31 a Display obtained with the Activity's Resources reports RetroArch's own window, so the size published for a display could be RetroArch's window's, and it changed when that window went to another display. A Display with no Resources attached reports the display's own logical size.
Changing Android's Display size setting, or RetroArch's window going to a display of another density, relaunched the Activity and RetroArch exited. The Activity now handles a density change itself.
The second display's Presentation could take the focus, so a tap there gave its display the keys. When the content closed and the Presentation went, the keys stayed on that display and reached the app under it: on the AYN Thor the pad drove the launcher's bottom-screen activity instead of RetroArch's menu. Taking the focus back with moveTaskToFront did not help, since RetroArch's task was already in front on its own display. The Presentation is now never focusable. Touches still reach it; keys and sticks always go to RetroArch's Activity.
video_views_test_map_change names a map the core changes to once video_views_test_map_change_frame frames have run; none clears the map with an empty one. A harness can then check how the frontend places a game whose map changes early in a start or later on. It is off by default. The Vulkan modes keep their images at the first frame's size, so there a change that would resize the frame is refused and logged.
A screen will name its display by the platform's name for it, which says which port a monitor is on. video_windows_display_key() gives that name, with " 2" or " 3" added for a second or third display listed under the same name, so each one can be told apart and saved. A display also carries its product name, for the menu, and whether a Presentation failed on it.
A display whose Presentation failed is now published as refused instead of invalid. It stays a valid target, so Auto and a saved name still point at it and RetroArch can move onto it; only a Presentation is kept off it. A list that changes only in which display is RetroArch's own keeps it refused. Each display also carries its DeviceProductInfo name (Android 12 and later), which the menu shows next to the display's name.
Native code can now ask Java to move RetroArch's Activity onto a display of the published list. The request is posted to the UI thread and never waited on. Java starts the running Activity again with setLaunchDisplayId and a copy of its own intent, so the same content is seen and nothing restarts; Android then moves the Activity in place. When the move cannot start (before Android 8, a display that has gone, or a launch Android refuses), Java logs it once and raises a flag the main thread reads on its next frame.
When the focused Activity moves between displays, Android sends the new display's focus gain and the old display's loss in either order. If the loss comes last, the Activity is never told it has focus again, so RetroArch stays paused and stops drawing. After a move that leaves it unfocused, the window is made unfocusable until Android has taken the focus, then focusable again, which makes Android send the gain. Under load the window manager can apply a short toggle's set and clear back to back, and the focus never moves, so the flag is held until that focus event, or a second at most. A gain ends the move only once it has outlasted a traversal, so a loss that follows it still refocuses.
The second display's Presentation never has the focus now, so it never receives a key or a stick move and never reports a focus change. Its key and motion relay, the ring that carried them to native code, its focus commands and its part in the focus count go: RetroArch's focus is its Activity's, as before the second display existed.
Each published display now says whether it is Android's DEFAULT_DISPLAY. On a dual-screen handheld both panels report the same unknown connection, so this is the only way to tell the main panel from the second one when choosing a display automatically.
…ications On Android, Separate Screen Windows reads "Game Screens on Other Displays". While it is on, Stereo 3D & Screens lists a Game Screen N Display row for each screen, Menu Follows (Game Screen 1, Game Screen 2 and each further screen the core has, or Separate), Menu Display (only when the menu follows no screen) and Notifications (With the Game or With the Menu). A Menu Follows screen kept from a core with more screens stays until it is changed. A display row offers Auto, each display by name with its product and size, such as "Display: DP Screen (DELL U2720Q 2560x1440)", and Off. OK opens a dropdown of the displays listed now. "Game Screen" and "Display:" keep a game's second screen apart from a panel Android calls Screen-2. The name is what is saved, so the same port is found again whichever monitor is on it; a display no longer listed shows as "Display: <name>" and counts as Auto. video_screenN_display, video_menu_follows, video_menu_display and video_notifications_window are saved in the config and in core and content overrides. They replace video_screenN_window_mode: an old Hidden loads as Off, Attached and Separate as Auto, and the old key is no longer written. The mode setting is gone, and the monitor row is listed on no platform; video_screenN_window_monitor stays in the config for the desktop. Until placement reads them, screen 1 stays in RetroArch's window and the others go by Auto unless Off.
With Separate Screen Windows on, video_windows_place() decides each frame, from the display settings, the published display list, the content and the menu, where each screen, the menu and the notifications go, and which display RetroArch's own window is on. It replaces the route, which sent only screens 2 and up to a Presentation. Auto no longer depends on where RetroArch was started: Game Screen 1 takes an external monitor, else the default display; Game Screen 2 the built-in second panel, unless a monitor took Game Screen 1. A display is in use only while it shows a screen of the running game or the open menu. RetroArch's window goes to a display the game may use that refused a Presentation, else to the lowest shown screen's, else, with no game, to the menu's. Any other display in use gets the one Presentation. RetroArch moves only when content starts or closes, a setting or the display list changes, or a display refuses. A change of the core's map within two seconds of a start is still the start; a later one only re-routes the screens. Replacing content is a close and a start, decided once the new content is up. Opening or closing the menu never moves anything. Moves are made before they break: a display RetroArch leaves that still shows something gets its Presentation first, and one it moves onto keeps its Presentation, not drawn, until RetroArch has drawn there. A move that cannot start or does not arrive within two seconds gives that display up until the display names change. Placement runs whenever windows are on, with or without content or views. A UI layer whose Presentation is being made waits for it up to a second, the frame the menu opens included, then falls back to the main window. A main window that shows no screen reports touches off-screen, and the menu maps them over the whole window.
Under the threaded video wrapper the app thread runs ahead of the video thread, so for about a frame after a snapshot lets the Presentation go or moves it, the newest published layout still names the old display. A make decided then completed at once and RetroArch left before the new Presentation drew; a keep could be set on the window being retired. Each snapshot now carries a serial, and each published layout the serial of the snapshot it was drawn from. A window counts as shown only from a layout drawn at or after the last snapshot that asked for it differently, so make before break holds. A snapshot the wrapper takes back undrawn needs nothing special: the later one that is drawn counts. A keep only holds a window that is up. A driver with no window no longer makes one for it, which at content close covered RetroArch's display with a Presentation that was never drawn. The frame also reads shown, the clock and the menu state once, for placement and the UI targets alike.
A driver teardown let the second display's Presentation go. When content closed and the driver reinited, the hide could reach Android before RetroArch's Activity arrived on that display, which then showed neither. A context's window ops can now keep and take a window. At teardown, gl2 and vulkan keep each window their context can keep: its EGL or Vulkan surface goes, and the Presentation stays up with its last image. The next driver takes it at init and publishes it with no screens, so it still counts as shown on its display and the close's keep holds it. Placement lets it go as before. A window nobody took goes once the next driver is up, or at shutdown. gl2 no longer draws a kept window, so it keeps its last image as it does on vulkan.
The menu and the notification widgets each lay out at the size of the window they are drawn in, which can now differ from the main window's: the runloop asks for each layer's size, and under the threaded wrapper the widgets' worker takes the notifications' size from the frame it draws. With both in the main window nothing changes.
When placement puts the menu or the notifications in the Presentation, they are drawn into one offscreen image at that window's size and composited over its screens. A window that shows them is drawn on every frame it has UI to draw, with or without content and views, so the menu keeps answering under a paused core; with nothing to draw it keeps its last image. If that window cannot be drawn in a frame, the image goes over the main window in the same frame, unless the layer is still waiting for its window to appear. Without the views' programs the layers stay in the main window. With both layers in the main window the frame is drawn as before.
When placement puts the menu or the notifications in the Presentation, they are drawn into one offscreen image at that window's size, between two halves of the frame's pass, and composited over the window's screens. While the image is drawn, the display backend places its quads and meshes against the image's height, not the swapchain's. A window that shows the UI takes an image on every frame it has UI to draw, not only on a new frame with views: a paused hardware core repeats its frame under the menu, and a core without a map has no views. On a repeat its screens are drawn again from the last frame. A view that only this window draws, and that the last new frame did not copy, is left out rather than turning the main window's views off. A window with only UI publishes an empty layout, so touches there miss the screens. A window with nothing to draw this frame, or with no free image, keeps its last image. Only a window that is gone sends the UI to the main window, in the same pass, unless the layer is still waiting for its window. With HDR the layers stay in the main window. With both layers in the main window the frame is drawn as before; the menu's draw moves into a helper both targets share.
When the menu's display cannot take a window of its own and the game is shown elsewhere, RetroArch's own window has nothing to draw until the menu opens there. It now shows the RetroArch logo from the monochrome icon set, centred on black, drawn as one quad each frame. Without that file or without widgets it stays black.
Android's input driver is to publish its touches and mouse through the frontend's pointer publication, so that each touch carries the window it came from, without changing any answer a core or the menu gets from it. Five flags let a driver ask for that, and change nothing for the drivers that publish today: a touch pressed only inside what it is read against; touches in a screen's own window as a second list; the mouse answered by the driver, whose motion and wheel go with each read; touch indices as slots that keep their place, with no mouse standing in; and pointer ids the frontend does not hold left to the driver. The publication also holds 32 touches, and gives the menu a touch in its own window over that window's client area.
The touches on the main window and on the second display go to the frontend's pointer publication each poll, raw, each with the window it is in, and the publication answers the pointer, the screen pointer, the second display's pointer and a lightgun aimed by touch. The mouse stays the driver's: its motion and wheel are taken by each read, as before. Every answer is as it was; the pointer_frame test checks each one the publication gives against the driver's own answers before this change. Three things differ. Binds to the mouse's left, right and middle buttons now work: they never fired on Android. And a touch is placed at the first read after the poll instead of as its event came, so a finger held still while the viewport changes follows the change at once. And a tap on a main window that shows no screen is on nothing: it fires no lightgun button there, while a tap in the letterbox still does.
With the menu drawn in the Presentation, it reads that window's first touch, over the window at the menu layer's size, and nothing from the main window; RGUI maps the touch through its framebuffer as it is fitted to the layer. An input overlay, which stays in the main window, no longer turns the menu's touches off there. The press and release bounds, the drag and swipe thresholds, the wheel and the on-screen keyboard's hit test use the layer's size too. A press held while the menu changes window is dropped until it lifts: no tap, release or cancel comes of it in either window. With the menu in the main window nothing changes.
A touch or mouse button already down when the menu opened acted on the menu. A finger held on the game as the menu toggle fired landed on Resume once let go and shut the menu again, and a held right button cancelled on the menu's first frame. This happened in every window, a single window included. Such a press is now dropped until it lifts, as one held across a change of the menu's window is: no POINTER_DOWN, POINTER_UP, OK on release or cancel comes of it, and the next press works as any. The cancel cover takes a pointer's back, Android's Back key among them. A cancel already down in the window the menu moves to is held back the same way. Only a real open counts: retroarch_menu_running() with the menu already up, as after a failed load, changes nothing. A press the menu held as it closed no longer fires its release on the next open either.
A turn of an Android display reaches RetroArch as a new size on paths these harnesses already run on the host: the main window's content rectangle or EGL surface, the second display's surface, and the window layouts and display list the frontend publishes. Pin what each must do with it, so a regression fails here and not only on a device. - android_content_rect: a 1920x1080 to 1080x1920 swap asks one resize at the swapped size on the Vulkan context, and on the gl one, whose check_window() is now extracted too, with egl_get_video_size() stubbed. - android_second_surface: the same surface at a new size is one GEOMETRY at that size, the same generation, no CLOSE, still shown; a window kept across a driver reinit meanwhile is taken at the size last drawn, and the next driver's first poll brings the new size. - video_screens: a window drawn again at swapped dims lays the menu out at that size and maps touches through the new layout from the first read; a display whose size changes under the same name keeps its placement and does not ask again for a move that was given up; a menu waiting for a turned display's window lays out at the size listed now.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Guidelines
C89_BUILD=1Description
On Android, each game screen can go to any display, and the menu can have a display of its own, such as the AYN Thor's bottom panel. RetroArch uses a display only while it shows something there, and opening or closing the menu never moves anything.
Settings (Video → Stereo 3D & Screens). On Android the Separate Screen Windows toggle reads Game Screens on Other Displays (English until Crowdin has it). While it is on:
With the toggle off, everything stays in one window.
Behaviour
Also in this PR
#1and#3set#2for#3(Keyboard Index too). Menu Follows needs it; it can split out.Known gaps
Tested
Testing on your own dual-screen device
Only tested on the AYN Thor so far; reports from other devices are very welcome.
phoenix-normal-debug.apk(com.retroarch) orphoenix-aarch64-debug.apk(com.retroarch.aarch64). They are debug-signed and won't install over a Play Store or nightly build of the same package; uninstalling deletesAndroid/data/<package>/files, so back it up first.cores/melondsds_libretro_android.so, Load Core → Install or Restore a Core) withmelonds_frontend_layout = auto,melonds_screen_layout1 = top-bottom,melonds_number_of_screen_layouts = 1,melonds_screen_gap = 0.Check each screen lands on its own panel from either launch panel, Menu Follows Game Screen 2 with taps, the pad after a tap on the second display and after a game closes, gl and vulkan with threaded video on and off, and home, back, sleep, wake and the lid. Please include the device model, Android version,
adb shell dumpsys displayandadb logcat -d -s RetroArchwith Logging Verbosity on.Related Issues
[Any issues this pull request may be addressing]
Related Pull Requests
Reviewers
[If possible @mention all the people that should review your pull request]