Import without updating the sequence

[Tunnelbliick]

Ive been looking over this for the past couple hours and think that there should be a better way of doing this.

{
  updateRecs: false
    // this will stop the update sequence for this rating
    // and greatly speed up the time to input all the data
    // however, there will not be any recommendations at the end.
    // if you fire a like/dislike with updateRecs on it will only update
    // recommendations for that user.
    // default === true
}

I currently already have a dataset of ~115000 which takes quite a while to import without updateRecs off.

Im currently confused on how i would go about updating each user, would i have to fire 1 like or dislike for that user to update his sequence or how would that work.

Is there an update function for individual users or for the whole class having these would make this process a bit more logical / streamlined.

[Tunnelbliick]

I tried importing it with updateRecs: false and then forcing and update for each user with a like and an unlike on value update.

Recomendations were not updated and remained empty if i did it this way

[jochemstoel]

Oof. It has been a really long time since I made this repo. If anybody has anything to say about this, please do. I'll have to really look into it again.

[Tunnelbliick]

@jochemstoel ive been reading into it.

updateSequence(userId, itemId, options = {}) {
    let updateWilson = true;
    if ('updateWilson' in options) {
      updateWilson = options.updateWilson ? true : false;
    }
  
    return new Promise((resolve, reject) => {
      this.#updateSimilarityFor(userId, () => {
        aSync.parallel([
          cb => {
            this.#updateWilsonScore(itemId, () => {
              cb(null);
            });
          },
          cb => {
            this.#updateRecommendationsFor(userId, () => {
              cb(null);
            });
          }
        ],
        err => {
          if (err){console.log('error', err);}
          resolve();
        });
      });
    });
  }

From what i can tell this option is called when items are added through like or dislike im currently testing if this works.

It takes quite a while tho to update all entries id say ~1h ish with 112327 entries.

I dont know if theres a way to speed it up further, which i think is the only issue this data is from ~2000 users and i was thinking of expanding the dataset to 10 or 100k which prob will be an issue with this update cycle time unfortunatly.

[Tunnelbliick]

@jochemstoel after alot of thinking on it and fooling around i think i have found the best way.

If we are batch importing stuff into proycon / redis its illogical for us to single process each like. Idealy we want to batch process everything thats already in the database.

Referenced from the updateSequence method we need to first updateSimilarityFor then updateWilsonScore then updateRecommendationsFor

Since i imported the sequence just before i still have it in memory and can use it aswell for this.

Doing it this way takes about 77 seconds!!! and yields recommendations. Keep in mind that i have done this with a batch processing count of 50 to limit I/O and cpu intensity.

Im unsure atm tho if the recommendations are similar / comparable to what full updateSequence or liked / disliked yields. I have not bothered importing those all since it takes quite a long time as stated above.

Im gonna check that and update this issue if the recommendations are the same then im gonna close it.

[Tunnelbliick]

Funny enough using updateSequence yielded 0 recommendations but took 162.64 minutes! at the same batch processing count of 50 as before.

My method with batch processing everything is ~12500% faster.

Gonna compare now against just using liked.

[Tunnelbliick]

liked took 57minutes and has recommendations

[Tunnelbliick]

The recommendations that come out arent the same but they have some similarities? My immediate guess would be that the similarity does not claculate well with this approach. Putting the wilson score first also yielded no changes in the recommendations.

Im a bit lost why else this could be the case maybee the depth it looks for neighbours since other neightbours maybee arent calculated yet?

using liked
[
  '983410_DT',  '979155_HDHR', '979155_HD',
  '957971_',    '954692_PF',   '953586_HR',
  '953586_',    '949029_',     '942561_',
  '933875_NF',  '932185_DT',   '909032_',
  '898575_',    '89670_DT',    '892780_DT',
  '891867_',    '880206_',     '876685_',
  '834354_DT',  '828940_',     '827660_DT',
  '822892_set', '820737_',     '819641_',
  '812010_DT',  '808849_HD',   '807302_',
  '795627_',    '786867_',     '774965_'
]
from 41108 keys

using my initial method
[
  '969681_HR',     '958819_',     '957971_',
  '954692_PF',     '933875_NF',   '928937_',
  '919772_',       '91215_',      '909302_set',
  '89670_DT',      '895822_',     '891867_',
  '89171_',        '888383_HR',   '855456_',
  '822892_set',    '812010_DT',   '808849_HD',
  '760488_HDHR',   '760466_',     '757823_DT',
  '746737_HDSDDT', '744444_HD',   '728871_HRHT',
  '713932_DT',     '705801_HDHR', '674128_',
  '655519_set',    '613967_DT',   '563370_DT'
]
from 41124 keys

[Tunnelbliick]

Im gonna close this issue since i have found a way that gives good enough results.

Im now updating in bulk like stated before. I had to expose updateWilsonScore updateSimilarityFor and updateRecommendationsFor.

Ive been running this with my database of 31k users with ~2.7million likes/dislikes altho it taking a while to import i still think the time gain over just regular liked is worth it. Since for the initial import im just looking for a good enouhg solution. I have noticed the same person as the most similar user tho eventho he has no similarity to my reference. I could dig a little deeper, but atm i just dont care and i dont think the time is well spend going deeper into this hole. Im gonna run the entier import with liked after the initial import which may speed things up a little and make it more accurate, just that for the initial state i think this approach is better.

[jochemstoel]

@Tunnelbliick
I have a feeling batch importing/updating can be optimized further to get better speed.
The processing times you mentioned so far all sound too long, although I haven't run any tests or even used this repository on a dataset as large as yours. (it is not that large though) I don't promise anything but if this turns out to be something more people have questions about, I might dive back into it if I have time. Right now I agree that the time is not well spent investigating this further.

Im gonna run the entier import with liked after the initial import which may speed things up a little and make it more accurate

Can you report your results here? Data size and time it took. For future reference.

I have noticed the same person as the most similar user tho eventho he has no similarity to my reference.

What do you mean by this?

If you believe your code fixes a problem with this repository and you have tested it sufficiently, feel free to do a pull request.

Thank you for sharing your process here, it might help others in the future. I will keep this thread open for now because we did not arrive at a definitive answer/solution.

[Tunnelbliick]

@jochemstoel it probly can. Theres somethings tho that i have noticed with this approach using redis and atm im not entierly sure if its a decent approach will supply data below.

Going through this entier thing and looking at about 20 different libs already i can atleast with confidence say that this lib has the easiest setup.

The Datasize deffinetly isnt "large" imo but who knows maybee i just did something entierly wrong afterall.

Can you report your results here? Data size and time it took. For future reference.

I cant say for sure since the logs of the import have since been deleted, i started it about 9pm ish and it finished somewhere at 3pm the next day this is using this approach.

Id say it took in the ball park of 18h ish.

Most of the time spend is updateSimilarityFor which is logical since it creates the similarityZSet and runs jaccardCoefficient for each user.

updateWilsonScore takes little to no time in comparison think it was done in around 20minutes tops

updateRecommendationsFor im not sure but it was a bit faster then updateSimilarityFor

This import stats are from my server also stats below

I havnt since run liked or disliked over the entier dataset since i figured the results to be "good enough" and this initial import consists of mostly implicite feedback whilst anything added afterwards is explicite feedback. Since it reevaluates the set on each like or dislike i didnt bother.

I have noticed the same person as the most similar user tho eventho he has no similarity to my reference.

Meaning the user A that was most equal to / similar to user B wasnt really similar / had no overlap to user B.

If you believe your code fixes a problem with this repository and you have tested it sufficiently, feel free to do a pull request.

There rlly isnt anything i changed i only removed the private declaration of updateWilsonScore updateSimilarityFor and updateRecommendationsFor so that i can call them myself.

I tried this with forEach aswell which yielded broken results and heavily stressed my cpu aswell as asyncBatch but in the end setteled for just a For of loop

       index = 0;
        for (let like of likeAggregation) {
            await new Promise(async (resolve) => {
                all.updateWilsonScore(like._id, () => {
                    if (index % 1000 == 0)
                        console.log(`updateWilsonScore ${index}/${likeAggregation.length}`)
                    return resolve(true);
                });
            });
        }

        index = 0;
        for (let user of userHash) {
            index++;
            await new Promise(async (resolve) => {
                all.updateSimilarityFor(user.osuid, () => {
                    if (index % 100 == 0)
                        console.log(`updateSimilarityFor ${index}/${userHash.length}`)
                    return resolve(true);
                });
            });
        }

        index = 0;
        for (let user of userHash) {
            index++;
            await new Promise(async (resolve) => {
                all.updateRecommendationsFor(user.osuid, () => {
                    if (index % 100 == 0)
                        console.log(`updateRecommendationsFor ${index}/${userHash.length}`)
                    return resolve(true);
                });
            });
        }

Redis Memory Info

Memory
used_memory:36987855080
used_memory_human:34.45G
used_memory_rss:37940776960
used_memory_rss_human:35.34G
used_memory_peak:36999439776
used_memory_peak_human:34.46G
used_memory_peak_perc:99.97%
used_memory_overhead:15690344
used_memory_startup:796232
used_memory_dataset:36972164736
used_memory_dataset_perc:99.96%
allocator_allocated:36988424336
allocator_active:37102899200
allocator_resident:37954207744
total_system_memory:67362332672
total_system_memory_human:62.74G
used_memory_lua:41984
used_memory_lua_human:41.00K
used_memory_scripts:0
used_memory_scripts_human:0B
number_of_cached_scripts:0
maxmemory:0
maxmemory_human:0B
maxmemory_policy:noeviction
allocator_frag_ratio:1.00
allocator_frag_bytes:114474864
allocator_rss_ratio:1.02
allocator_rss_bytes:851308544
rss_overhead_ratio:1.00
rss_overhead_bytes:-13430784
mem_fragmentation_ratio:1.03
mem_fragmentation_bytes:952962912
mem_not_counted_for_evict:0
mem_replication_backlog:0
mem_clients_slaves:0
mem_clients_normal:66616
mem_aof_buffer:0
mem_allocator:jemalloc-5.2.1
active_defrag_running:0
lazyfree_pending_objects:0

This is also an issue since it uses around 35GB of RAM which isnt ideal to say the least. I understand this is mostly todo since redis is an in memory database.

Data in memory (all recommendations from database) are around 3.5GB so 35GB is a bit of a stepup.

MongoDB size

3159914 -> likes/dislikes
31814 -> users
260127 -> items

  totalIndexSize: 231260160,
  totalSize: 324562944,
  indexSizes: {
    _id_: 57847808,
    osuid_1: 15720448,
    beatmapid_1: 21995520,
    origin_1: 15826944,
    value_1: 37490688,
    osuid_1_value_1_origin_1: 82378752
  },

Since my dataset is 100% public data i can also share it if you want it for further testing, note that it is in mongoDB.

I have looked at different libs for pretrained models or keeping the recs in postgresql i will probly look into a more pretrained solution or something similar for the future.

Hope this helps.