Repository navigation
Scan from the command line: exit status, Manual Scan options, playlist refresh - #19977
Open
Davey-Hughes wants to merge 17 commits into
Open
Davey-Hughes wants to merge 17 commits into
Davey-Hughes wants to merge 17 commits into
Conversation
fill_pathname_expand_special() skipped two characters after a leading '~' or ':', so a bare "~" read past its NUL and "~x" lost the x. It now skips the prefix and a slash after it, if there is one.
task_push_dbscan() and task_push_manual_content_scan() take user data for their callback, and the callback's task_data is now a task_dbscan_result_t: how many entries the scan added, how many its playlists already held, and how many files matched nothing. A file is counted by what became of it, so an archive that is expanded is not counted as unmatched, and a loose scan's unmatched files count as added. Callers pass NULL for now.
A scan that ends before it runs (bad setup, no valid content, a DAT that will not load) or ends with a task error now sets the result's failed flag. A database folder that is missing or unset is treated as an empty one, so it reports the same "no database" error instead of finishing as if it had scanned.
A file whose scan fails (an unreadable file, a failed lookup) was in no count. The result now carries an errors count for them. They do not fail the scan.
A command-line scan now ends with a totals line on stdout, or with the reason it failed on stderr. A file it could not read is listed on stderr. The ++/==/?? marks are coloured only when their stream is a terminal, so pipes and redirected output get plain text; on Windows redirected output keeps its marks instead of losing them to WriteConsole.
A finished scan stays in the task queue while its callback runs, and the finder that refuses a second scan for the same playlist still matched it, so a scan pushed from that callback was refused.
Each --scan is collected while the command line is parsed and run in turn once it is done, so later options and further --scan flags are no longer dropped. A path is expanded and made absolute first; one that does not exist, or a scan that cannot start, is reported on stderr and the next path still runs. --scan with --cloudsync or --command is refused. Without --menu the exit status is 1 if any scan failed, else 0 if any added an entry, else 2, and a content path given with the scans is ignored, with a line on stderr. With --menu each finished scan starts the next, as a second scan is refused while one runs.
A playlist with no saved database name left the name buffer uninitialised, and the refresh compared it against every installed database, so whatever was on the stack could pick a database the playlist never named.
The option was meant to put the playlist's name in each entry's db_name, even for a database match (libretro#18641), for users who build their own thumbnail sets. When the scan picks the playlists, db_name also picks the playlist an entry goes to, so with the option on every match went to one playlist named after the content folder. It now applies only when the scan writes one playlist, and gives that playlist's name, a loose scan's unmatched entries included. With the target on <Automatic>, or Fully Automatic with the option left on from a Custom scan, matches go to their databases' playlists as with it off; the entries of a custom playlist look their thumbnails up under its name, not the folder's.
The "Scanning content" line, in the log and in --scan's output, has named the database folder since the manual scan and the database scan were merged (2fdf736); before, it named the path being scanned. It names the content folder again.
The command line's scan options need each scan to start from the defaults and to lay their values over them, or over a playlist's saved settings. Add a reset, a record of those options and a function that applies it through the existing setters: a core named by path, file name or display name, a database, a playlist, the database check or a DAT, extensions and the switches, stopping with a reason at the first that does not resolve. A core path is normalised before it is compared. A core's file name matches in the case given first, then ignoring case, on every platform; a display name matches exactly. Given another database, a playlist named after its database keeps that name as a custom one, so a refresh still writes the same playlist. The core setter's copy step becomes its own function, so a core found by path or file name can be set. Nothing calls the new functions yet.
The --scan paths ran through a loop in retroarch.c, and with --menu through a second chain of their own. The scan options and playlist refreshes to come need one runner for every item, from the command line and from the menu. task_push_dbscan_chain() runs a list of items one task at a time: a prep task lists the databases, then each step is pushed from the last task's callback on the main thread, so nothing waits and nothing locks. An item that fails is reported and the run goes on; a cancelled task ends it. retroarch.c builds the items in the first argument pass and hands them to the chain; without --menu it waits for the chain's end and exits as before. Running out of memory prints "out of memory" on stderr: for an item's path the item fails, and for the chain itself nothing runs, which without --menu exits 1. The chain also prints a scan's override errors, and the warnings for an option that needs one playlist or does nothing for a single file, which the scan options of the next commit can trigger.
--scan-playlist, --scan-database, --scan-match, --scan-dat, --scan-core, --scan-exts and the five switches set the Manual Scan settings of the --scan before them; a --scan without any stays Fully Automatic. Values are checked in the first pass, so an option before any --scan, an empty or bad value, or one too long for its setting ends the run with its reason on stderr and status 1, before the config loads and before anything is scanned. Cores, databases and DAT files are resolved when their scan runs, and one that does not resolve fails only that scan.
A refresh item rescans one playlist with its saved scan settings, as the menu's Refresh Playlist does, with the item's scan options laid over them and written back by the scan. A refresh-all item does the same for every playlist in the playlist folder that has saved settings, in the order Manage Playlists lists them, histories and favorites left out. A refresh item names its playlist by name or by a path in the playlist folder. A bare "name.lpl" is the playlist folder's file, and an absolute path is normalised before it is checked against the folder. The name is looked up in the folder's listing on every platform, the exact name first, then ignoring case, since macOS and FAT/exFAT cards ignore case; the refresh then takes the file's own name, so a database's playlist stays one. Two or more playlists that match only in other cases fail the item. The playlist folder walk and each playlist load run in prep tasks: on the worker with a threaded queue, under the shared I/O window without one, each pushed from the previous step's callback. A playlist whose saved content folder is gone is skipped rather than failed. A listing that runs out of memory fails the refresh-all; a playlist folder that is missing or cannot be read lists nothing, and the refresh-all ends with "no playlists with saved scan settings". A refresh's start line names its content folder; its totals and failure lines name the playlist.
--scan-refresh=NAME|PATH rescans a playlist with its saved scan settings, as the menu's Refresh Playlist does, and --scan-refresh-all rescans every playlist that has them. Both are items of the same run as --scan: the scan options after one are laid over the saved settings and written back, and the exit status follows the same rule. --scan-playlist after a refresh is refused, so a refresh always updates its own playlist, and an empty --scan-refresh is refused as the other empty values are. Like --scan, the refresh options cannot be combined with --cloudsync or --command; the error names whichever of the three comes first on the command line.
Manage Playlists gets Refresh All Playlists as its first entry when it lists any playlist, so an empty list still says there are no entries. It rescans every playlist that has saved scan settings, with those settings, through the same chain as --scan-refresh-all, and ends with one notification saying how many playlists were refreshed, skipped (content folder gone) and failed. While a content scan runs it is refused with a new message; a refused scan had none. Other languages come through Crowdin. While a scan chain runs (Refresh All, or a command-line scan with --menu), Start Scan, a playlist's Refresh, Scan Directory and Scan File are refused with the same message, as they would run with the settings the chain last loaded. task_push_dbscan() returns false then too, which covers the companion and the iOS caller.
--help lists the options but not what a script needs: the order the scans run in, the exit status and what the options set. The man page now has them, next to --cloudsync, and CHANGES.md has the entry.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Guidelines
C89_BUILD=1Description
retroarch --scan=PATHcould not be used from a script, could not choose any of the settings a Manual Scan can, and nothing rescanned the playlists that are already there.--scanfrom a scriptexit(0)while the command line was still being parsed, so every option after the first--scanwas dropped, including further--scanflags.log_verbosity = falsehid them, and nothing reported what the scan had done.~paths were not resolved, and a missing path was never reported.With this PR:
--scanis collected during parsing and run in turn afterwards. One that fails is reported, and the next still runs.PATH: not foundon stderr.!!. The++/==/??/!!marks are coloured only when their stream is a terminal.--menu, the exit status is 1 if any scan failed, 0 if any scan added an entry, and 2 if every scan ran but none added anything. Content given with the scans is then ignored (without --menu: ignoring "<content>"). With--menu, the scans run while the menu does.--scanand the refresh options below are refused with--cloudsyncor--command(exit status 1).Manual Scan options
--scan-playlist=NAME,--scan-database=NAME|first,--scan-match=strict|loose|none,--scan-dat=FILE,--scan-core=CORE,--scan-exts=EXTSand the switches--scan-no-recursive,--scan-no-archives,--scan-overwrite,--scan-validateand--scan-omit-db-refset the Manual Scan settings of the--scanbefore them. A--scanwithout any stays Fully Automatic, as before.--scan-coretakes a core's path (~,//and..resolve), its file name (with or without_libretroand the extension) or its display name. File names match exactly first, then ignoring case, on every platform; two or more matches fail the scan (mesen: 2 cores with this name, give the path).--scan(--scan-core must follow a --scan), or an empty, bad or too long value, ends the run with status 1 before anything is scanned. A core, database or DAT file that does not resolve fails only its own scan (mesen: no such core,~/x.dat: not a DAT file), and the next scan runs.--scan-overwrite,--scan-validateand--scan-omit-db-refact only when the scan has one playlist, and--scan-overwriteonly on a folder; otherwise they warn and are ignored.Playlist refresh
--scan-refresh=NAME|PATHrescans one playlist with the settings it saved, as Refresh Playlist does. A name matches exactly first, then ignoring case, on every platform; several that match only in another case fail the item. A path must be in the playlist folder.--scan-refresh-allrescans every playlist with saved scan settings, history and favorites aside.--scan-playlistis refused there, as a refresh keeps its playlist.Sega - Mega Drive: skipped: content folder not found (/mnt/usb/md)on stderr); a skipped playlist does not count toward the exit status. Any other problem fails that playlist, and the rest still run. With no playlist that has saved settings,--scan-refresh-allprintsno playlists with saved scan settingsand exits 2.Playlists refreshed: 10, skipped: 1, failed: 0, and is refused withA content scan is already running.while another scan runs. While it or a--menucommand-line scan runs, Start Scan, Refresh Playlist, Scan Directory and Scan File are refused too, as they would run with the settings the chain last loaded.tasks/task_database.c, each step pushed from the last one's callback and every listing and playlist load in a task, so nothing blocks the menu.Usage
The runs below use the default config,
~/.config/retroarch/retroarch.cfg, withlog_verbosity = "false". The[INFO] [Config]line is printed before the config is read whenever it is found by the default path; with--config=FILEit is not.A scan prints a progress line per file, then
++for each entry added,==for one already in the playlist (both naming the database and the game) and??for a file no database matched, and ends with its totals:The same scan again adds nothing, so the exit status is 2:
Several
--scanitems run in turn. A path that does not exist is reported on stderr, the next scan still runs, and the exit status is 1:Options apply to the
--scanbefore them. Here the SNES folder goes into a playlist named SNES, with Snes9x as its default core,.sfcand.smcfiles only and no subfolders (sotranslations/is left out), and the arcade folder is matched against a DAT that knows two of its three sets:A DAT scan prints no progress lines and no
??line; its totals still count the set the DAT does not list.A bad command line, such as an option before any
--scanor a value that is not allowed, stops with exit status 1 before anything is scanned:--scan-refreshrescans one playlist with the settings it saved. After a new game is copied into~/roms/snes:--scan-refresh-alldoes the same for every playlist with saved scan settings. Here there are three: SNES and arcade from above, with a new set in~/roms/arcade, and a Mega Drive playlist scanned from a USB stick (--scan=~/usb/megadrive --scan-database="Sega - Mega Drive - Genesis") that is now unplugged. That one is skipped, and a skipped playlist does not count toward the exit status:With
--menu, the scans run while the menu does, and RetroArch stays open.Without
--menu, the exit status is 1 if any scan failed, 0 if any scan added an entry, and 2 if every scan ran but none added anything, so a script can tell them apart:Also fixed in this PR
<Automatic>and Fully Automatic scans put every match into one<content folder>.lpl; now they fill the database playlists (Nintendo - NES.lpl, ...) with their thumbnails, icons and Explore entries. A<Custom> Xscan's entries carryX.lpl, so thumbnails are looked up under X, as the option's description says; with a specific database they carry<database>.lplinstead of an empty name.Not migrated: playlists the old behaviour made. Refreshing one still works, but a new automatic scan also fills the database playlists, so games show twice until the old
<folder>.lplis deleted; a refreshed<Custom>playlist mixes<folder>.lplandX.lplentries until rescanned with Overwrite. For one combined playlist, choose the content folder as the target, with the option on.Scanning content: "..."line, in the log and in--scan's output, names the folder being scanned again, not the database folder.fill_pathname_expand_special()read past the NUL on a bare~and dropped a character after~x.Code
task_push_dbscan()andtask_push_manual_content_scan()take user data for their callback, whosetask_datais atask_dbscan_result_twith the counts and a failed flag; existing callers pass NULL.task_push_dbscan()returns false while a chain runs (companion UI, iOS).task_push_dbscan_chain()runs a list of items; manual_content_scan gains a settings reset and an overrides record with its apply;core_info_get_file_id()is no longer static. Menu strings are inmsg_hash_us.honly, for Crowdin. The man page documents the options and the exit status, andCHANGES.mdhas an entry.Testing
samples/tasks/database(counts, failure flag, unreadable files, CLI output, overrides and core lookup, the chain, refresh and Refresh All on both task queues) and infill_pathname_test.retroarchand 326 against a--disable-threadsbuild.retroarch.exepass.Related Issues
Closes #15311.
Part of #14197 (Refresh All Playlists; Clean All is not part of this PR).
Manual Scan options on the command line: #12646.
Related Pull Requests
None.
Reviewers
@sonninnos