Caching + New Search Algorithm + Update Liked Songs

[FlaShHolloway]

Description:

• Caching Video IDs: I’ve added caching functionality for video IDs in a JSON file. It's opt-in, so you’ll need to use the --use-cached flag to create or use the cache.
• Fix-Match Command: Introduced the fix-match command, which allows you to replace a bad match with a preferred video ID. This works only if you use the --use-cached flag to ensure consistency in fetching that ID.
• Cache Management: You can clear the cache by calling the cache-clear command, which removes the lookup file.
• Extended Search Algorithm: Implemented a new search algorithm with the --extended-search flag. You can adjust the confidence level (optional) (e.g. --confidence 0.6) for better match quality (range 0-1).
• Album Search: If you want to search for Spotify matches within albums, use the --search-albums flag. It’s slower as it searches through all albums but has a higher probability of yielding good matches.
• Fallback Option: The --enable-fallback optional flag can be used to fall back to the default search algorithm if no matches are found within the specified confidence level. It ensures that at least something is returned from YouTube Music, even if it’s not a perfect match.
• Searching: The search option can be used to search a Spotify's Youtube Music equivalent, all the same arguments as above can be used (i.e. --extended-search, --search-albums, etc.) to individually check matches or simply add a single song to lookup file using --use-cached flag.

Testing:

• Added test cases for the new functionality. Please review and let me know if there’s anything I can improve or change!

[FlaShHolloway]

In update argument, include liked argument instead of playlist_link will now allow you to update your YTM's liked playlist with Spotify's Liked Songs, include name of the YTM playlist as usual (If not changed, it would be Liked songs (Spotify)). This also means you can create a YTM playlist and use that playlist to sync your liked songs (or any other playlist). Note: there is a very likely chance of timeout if playlist is big, so --use-cached can come in handy. Also encountered an issue where, if fix-match id replacement used incorrectly and incorrect id is replaced, create or update will result in Error 400 Bad Request since ID(s) doesn`t exist(s).

[sigma67]

Hi man, I appreciate the effort, but this MR is way too big. The key in open source is to do incremental changes, not huge refactors doing everything at once

In addition it would be great if you could point out what problems with the existing lib you are trying to fix (esp. regarding song matching)

[FlaShHolloway]

Understood! I appreciate your response. Here is what I faced and trying to fix:

• Firstly use of caching. Since you had a made an attempt, I gave it a shot. I noticed the search results were not consistent at a time, a good matching track might not appear in the result every time, So I made it so that using caching, the good result can be retrieved consistently instead of it changing every time. It's opt-in so no need to force it use it if it works for you, plus for fixing incorrect matches directly in caching, will allow to maintain a lookup of correct tracks so it will help during updating of playlist, getting the track right every single time, while also enabling resuming if during search any error occurs.
• Secondly a new search algorithm, based off yours old. The reason I made this is because, duration match and title match works almost 70% of time if title is distinct and so is the duration. But problem arises when title is not distinct (i.e. Flowers by Miley Cyrus or GOATED By Armani White) I keep getting either totally wrong track or another version of track (i.e. I keep getting "Slowed" version of GOATED instead of original one). I also noticed that albums were not included in search for obvious reasons, I thought if I could include an opt-in option to include albums in search result at the cost of slow search, it could result in good matches, and that's what I got! Not only were albums search accurate but gave me the ATV versions that had really high quality audio (at least my ears agree). So the new algorithm searches for track with new criteria and handles many edge cases that I personally tested after a long observations of YT Music results and got a pretty good algorithm that is opt-in that can come in handy when needed. I added search argument to test matching of tracks so one can see which algorithm returns a track correctly.
• Lastly, I added the option to update liked songs, I saw an issue mentioning that wanted to update their liked songs so this was straight forward. But I also updated the way the update occurs. Your method was straight forward enough, and I had no problem, until I started updating the liked songs. I kept getting timed out at remove_songs. I guess removing 700+ songs at once is something YT doesn't appreciate. So I just made it so that only songs need to be removed (i.e. song in YT Music but not in Spotify) and songs need to be added are added/removed so the updating is fast and time out never happened. But this may also make the playlist order of tracks out of order and some folks wouldn't like it otherwise so I added an option (--reset) to completely wipe the playlist and re-add the tracks (ofc the traditional way). I really didn't plan to add this one but thought it would be helpful.

You're correct, this all should've been incremental changes and not huge refactors, I'll keep that in mind next time. If you want, I can remove/revert any changes that doesn't seem important. I've added tests for new features. Lessons learned!

[darshiz]

Hi, just checking if there is any plans on implementing the code here? Either in separate smaller commits, or as a large one commit? I am quite interested in having this option of updating playlists by appending instead of replacing the entire playlist.

[FlaShHolloway]

I ditched Spotify completely so I rarely use this now. Open to ideas.

[sigma67]

I don't mind merging as long as it's one feature at a time. The current size is impossible to review. I like that most of the features are opt in.

[FlaShHolloway]

Alright, I`ll try.

[sigma67]

Caching and search CLI have been merged.

[FlaShHolloway]

Next I'll go for updating existing --append mode for update command. I'll add the ability to update liked playlist and update logic of append (Case 0: If a track is removed from yt music playlist, then append mode would not add it back). New logic will ensure that yt music playlist is synced with spotify playlist. This and use of --use-cached in conjunction will allow quick update of all playlist (one can even create cron jobs for this too!).

[FlaShHolloway]

Should I create a separate command like --sync or update --append, since it is possible someone would need to add only new songs from spotify to yt music and some tracks were removed on purpose

[sigma67]

I'm a bit confused to be honest. Maybe you could list all the possible use cases that aren't supported right now and we decide from there?

[FlaShHolloway]

Sure. I'll mention few:

• Case 0: If a track is removed from yt music playlist, then append mode would not add it back (Sync mode will add/remove tracks that are required)
• Case 1: If user wants to keep updating the playlist while keeping it identical to spotify playlist, --sync will help achieve it, while keeping low footprint on requests.
• Case 2: Using --sync users can create cron jobs to daily sync all playlists.
• Case 3: --sync will help reduce timeouts that occurs when playlist is huge (since only few addition/removal is performed)
  While --append can be used by users who just prefer to transfer tracks over to YT Music playlist but has the target YT playlist customized and doesn't want any addition of previously removed tracks while also not want to remove tracks that are removed from spotify, while also adding new tracks from spotify to YT Music (Very specific use-case but possible one)

[sigma67]

@darshiz as you can see the option to append instead of replace already exists

@FlaShHolloway

• case 0: isn't current update alone the same as sync? update keeps the playlist in sync by replacing it with the latest spotify tracks
• case 1: same here.. i don't see the difference
• case 2: this is also possible using update?
• case 3: I still don't understand how --sync is different than update without --append

The only new request I can tell from your comment is the ability to "exclude" tracks from the update by respecting what a user has manually removed? For example --ignore-removed-tracks ?

[FlaShHolloway]

@sigma67 yes, update does the same thing, but once playlist is large enough, after a point, I could not update my playlist no matter how many times I tried, I kept getting timed out (The part were all tracks are removed and then immediately adding all the tracks back). I tried sleep timer (upto a minute) but didn't work either. And --append command, already does not add removed tracks back (idk if it's a bug), I tried on my playlist, removed 2 tracks from YT Music playlist then tried updating with --append, result was tracks were not added back that were removed before & present in Spotify Playlist.

[FlaShHolloway]

You're right, instead of making a whole new command, if append works as expected, only adding liked playlist support & --ignore-removed-tracks would be sufficient

[darshiz]

Thank you very much for the hard work guys! I really appreciate the effort to get this committed into the main branch!

Quick question on the update mode, does re-populate the playlist from scratch or just add the new songs? I thought it was re-populating the playlist every time.

[FlaShHolloway]

@darshiz by default if you do not add --append it will remove all tracks from the playlist and re-add them again. Append mode just skips the part were all tracks are removed. Hence the issue I faced when playlist was large enough to cause time-outs.

[darshiz]

Ah perfect, thank you for the clarification @FlaShHolloway

@sigma67 would you be able to release an updated compiled version with the newly merged commit please?

[FlaShHolloway]

@sigma67 did something changed with get_playlist() function of your ytmusicapi? Because it seems, the function only returns maximum of 100 results, no matter what I pass for limit.

[FlaShHolloway]

@sigma67 The function get_playlist() of ytmusicapi isn't returning more than 100 tracks despite setting limit to an appropriate value. This means the whole playlist will not be removed for playlists with more than 100 tracks, and the playlist will contain fewer tracks or not even half after update. I can raise the issue on ytmusicapi with the appropriate information.

Just realized it has already been discovered and fixed for v1.10.1.

[darshiz]

Hi guys,

I've tried using the --append APPEND option when updating playlists to ensure it does not remake the entire playlist every time, and it seems like it is not changing the playlist with this added.

If I remove the --append APPEND, then the playlist changes fine.

Can you please help me?

[FlaShHolloway]

@darshiz checkout #143 you might be facing the issue of duplicate track IDs that I mentioned and implemented a temporary fix for it.

[darshiz]

@darshiz checkout #143 you might be facing the issue of duplicate track IDs that I mentioned and implemented a temporary fix for it.

Cheers thank you @FlaShHolloway!