Loading metadata with p-limit - #103
Open
Borewit wants to merge 2 commits into
Open
Conversation
This was referenced Jul 20, 2025
Contributor
Author
|
Published this version on Netlify: https://audiomotion.netlify.app/ |
Borewit
force-pushed
the
loading-metadata-with-plimit
branch
2 times, most recently
from
July 20, 2025 15:03
ef64e07 to
c941b7c
Compare
Borewit
force-pushed
the
loading-metadata-with-plimit
branch
from
August 19, 2025 11:29
c941b7c to
1c1adcf
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR is build on top of:
music-metadata-browsertomusic-metadata#89I took a bunch of metadata files, spin op the audioMotion.js in server mode.
With the changes of #89 it metadata loaded slowly, massive parallel loading of the playlist, which took ~6 seconds to load all metadata.
First I disabled parallel fetching in the
addSongToPlayQueue,retrieveMetadata, function chain.The parallel loading is fact not done in
retrieveMetadata, which is kind of mind fuck, as this method is responsible for the entire queue. Yet if a bunch of files are added to the queue, this function called for each item again and again. Hence #89 makes the parallel loading worse, and therefor performance goes down the drain.After disabling all parallel processing, putting
awaits all over the place. The increased the performance to ~1100 ms!Then I re-enabled parallel loading, now using
p-limit, to avoid the number of parallel processing, and to slowly tear down the requirement to keep callingretrieveMetadataway to ofton. Resulting to loading the metadata in 600ms.Technical debt
audioMotion.js/src/index.js
Line 1258 in 471e947
I maybe I should say, I kept as is on
devbranch.There are several ways to resolve this, but it requires some refactoring.
One solution could be to push things which require metadata in a queue. Then we consume this queue by the the metadata
retrieveMetadata, which maybe could be a worker / parallel processing thingy.Alternative solution
Update: I created #104 as probably a better alternative them, as that one queues items