Set.fetch breaks if set was too large

[atitan]

Hi

We've been using forgetsy for 2 weeks, and is now receiving SystemStackError: stack level too deep when fetching results.

It turns out Forgetsy::Delta.fetch does not pass option[:n] to secondary_set.fetch, and fetch_raw returns large array (approx. 65K) to secondary_set.fetch for Hash[*result.flatten], which is too many arguments (approx. 131K after flatten) for Hash to handle.

[cavvia]

Thanks for this report and sorry for the delay in getting back to you. I cannot delegate the limit parameter :n to the secondary set fetch as it needs to be applied after normalizing the primary set by the secondary.

One potential fix is to declare a more aggressive hi_pass_filter constant - this is used when trimming the set at query time. I could turn that into an ENV var if that helps. You can then tune that to ensure your sets never hit a size over the limit.

Another approach I use is to trim sets back to around 50k periodically (after a descending score sort).

[cavvia]

Another option is slowing down the Hash assignment by using an iterator or other method... I can try to benchmark that to see if it will be feasible performance wise as this all runs at query time. That should allow scaling to very large number of items.

[dylancopeland]

@cavvia Thanks for this great gem! I just started playing around with it and unfortunately came across the same problem as @atitan.

Did slowing down the hash assignment prove to be successful? I am trying to determine the best way to move forward with forgetsy.