General plan to add DRM-free games from other stores

[boggydigital]

After validating getting metadata and downloading products for Steam and Epic Games, I'm ready to start generalizing this in vangogh.

Key challenges for any solution:

1. vangogh has been designed around GOG.com first and foremost - I'd like to use the opportunity to clean that up a bit. This is more true for backend, then frontend. Redux is already abstracting underlying store metadata and Steam/EGS will plug naturally into that.
2. Steam and EGS have predominantly DRM games with no clear way to distinguish between DRM'd and DRM-free. There are community sources, but it's still a trial an error - typically you'd want to install a game to confirm. It won't make sense to download all library products - most won't work without a client. Instead this work will focus on onboarding individual games (including bulk additions of course).
3. EGS integration is bespoke and should be able to support regular checks for updates and automatic downloads with no supervision or constant reauthentication
4. Steam integration uses SteamCMD, and it invalidates login information seemingly at random. Checking if application has updates should still work (doesn't require Steam login), but downloading updates might not be as automatic as GOG or EGS. At the moment I'm not planning to implement bespoke Steam integration like EGS.
5. Everything I have in mind for this will take some time to implement. I'm planning an iterative approach that would unblock user value as soon as possible without waiting for everything to converge - same approach I used so far. Browsing Steam, EGS games in vangogh - would take a while. Adding games through CLI, downloading and updating, then being able to install with theo - that should happen sooner.
6. Several open questions remain - that will be surfaced and explored as I get to them. E.g. would vangogh allow downloading those games without theo? Yes, unless there's a blocker I can't foresee.

Phase 1 - Preparation, Technical debt reduction, Clean up

• Rename GOG product types to indicate the origin (e.g. details -> gog-details)
• Migrate existing GOG product type directories to new ones
• Reset GetDataErrorDateProperty, GetDataErrorMessageProperty, GetDataLastUpdatedProperty since the format of the key has changed
• Rename GOG properties to indicate the origin (e.g. slug -> gog-slug)
• Migrate existing GOG properties to new ones
• Rename GOG key values to indicate the origin (e.g. changelog -> gog-changelog)
• Migrate existing GOG key values to new ones
• Change HLTB properties to be indexed by HLTB Id (not GOG Id) to allow usage with other stores data
• Change OpenCritic properties to be indexed by OpenCritic Id
• Migrate Steam properties to be prefixed by Steam and to be indexed by Steam AppId
• Migrate OperatingSystems, LanguageCode, DownloadType, NoPatches to Gog* properties
• Reduce GogOwned property during providing data types reduction
• Reduce TopPercent property during OpenCritic data reduction
• Deprecate SummaryRating property, use available ratings for display
• Deprecate DeveloperPublisher reduction (keep data as in from GOG, e.g. "TEST DEVELOPER")
• Reduce Demo property using providing data types reduction
• Reduce Credits during Wikipedia data reduction
• Deprecate ReduceMisc func and clear call sites
• Reduce gog-account-products order to allow sorting by it
• Change GOG search section to use gog-account-products order instead of GOG Order date
• Consider deprecating product-details type and instead request/use raw GOG data (through vangogh) for consistency with other Stores data use pattern
• Standardize pathways directories across vangogh and theo in southern_light
• Create vangogh migration for all the changes above
• Rename GOG Image types to make GOG part explicit
• Use gog-api-product for Steam shortcut vangogh assets
• Refactor sync to deprecate individual types options, focus on a coherent (non-configurable) flow of cmds
• Refactor updates as /search views, deprecate /updates as it it today
• Remove summary/summarize - it should all be automatic, based on search
• Remove atomus usage and archive the project
• Make purchases default option for sync, extra-data as optional
• Add VangoghSteamNewsUpdateDateProperty to be able to sort by recent news updates
• Add GogWishlistPageOrderProperty to be able to sort by date added to wishlist (won't work for local updates, but will sort itself out on sync)
• Complete /updates -> /gog migration by adding Sort options in each section (except for /gog/search).

Key outcome: Next phases are unblocked, this is pre-req.

Phase 2 - vangogh EGS onboarding - downloads and API

• Add connect cmd instead of import-cookies
• Reintroduce ability to connect to GOG with username/password and 2fa
• Migrate import GOG cookies under connect cmd
• Add helper to track if specific service has been connected
• Move over Origin type from theo to southern_light (as Distributor)
• Add ability to connect to EGS with connect cmd
• Move theo egs_integration code to southern_light to allow vangogh to use it
• Consider moving actual command payloads (not just *_integration) code to southern_light to allow sharing that code between theo and vangogh. This might not be ideal - for example, connecting to vangogh would not make sense to add to vangogh, so need to explore case by case.
• Add epic-games and steam to get-data, this would extend vangogh's get-data pipeline and establish data dependencies. For data - we want to get everything owned (not just DRM-free). Finally, steam parameter would be to establish EGS -> Steam connections for the next steps
• Reduce EGS data into redux
• Get steam data for EGS products for extra data, rationalize extra data in general
• Add epic-games parameter to get-downloads. This would require id (need to check if id already allows multiple values). This will download EGS chunks.
• Add epic-games parameter to validate. This would require id
• Validate that get-downloads update and validate flags work for EGS downloads
• Make sure download and validation status properties are set correctly for EGS downloads
• Add epic-games flag to cleanup to remove unnecessary files
• Integrate theo with EGS games on vangogh

Key outcome: vangogh can download EGS chunks, theo can install EGS games from vangogh.

Phase 3 - vangogh integration with theo - display

• Integrate EGS games into sync
• Add assemble-chunks command to assemble EGS chunks on the server
• Add assembly flag to validate command that would validate EGS assembled files
• Add epic-games flag to get-images and get whatever available images for EGS products
• Display EGS products in vangogh using redux and available images
• For downloads - display a link to assemble chunks on the server, compress and when ready - provide a link
• Add cleanup assembly to remove assembled files after some time (48 hours?)
• Add EGS data to source data

Key outcome: vangogh can display EGS products.

Phase 4 - vangogh Steam onboarding

• Setup SteamCMD for the current OS
• Add ability to connect to Steam with connect cmd
• Move theo steam_integration code to southern_light to allow vangogh to use it