"gen_data_fin.py" out of memory for large data set

[lightsailpro]

Thanks for the tf2 version.
I have a large dataset on a server with 64GB RAM. "gen_data_fin.py" step caused out of memory error. Any suggestion for alternative ways to generate the .test.tfrecord, .train.tfrecord, .his, .vocab files? Do we have document of the these file formats? Can it be ported to nvtabular? Thanks

[asash]

Hi, I guess i it is a large number of items in your dataset that cause the problem. The original version of BERT4Rec is not very friendly to large datasets (particularly with a large number of items), however I am now working on my own implementation that can handle large datasets, and I hope to release it soon.
I may try to fix memory issues with this original implementation in the future, or if you can do that I would be happy to accept your patch :)

[lightsailpro]

Thank you for fast response. I am looking forward to try your new enhanced implementation. I am not quite sure on how to modify the existing code to handle large data set with python array or data frame. I was thinking generating the .train.tfrecord, .his, .vocab files on database side using SQL if I know the format. But I will wait for your code instead. Thanks.

[lightsailpro]

@asash Any update on the progress of the new version that can handle large dataset? I looked at the gen_data_fin.py. Right now, it needs to load the whole raw dataset into python array which leads to OOM for large data set. If we split the raw data into multiple chunks, it will generate multiple .train.tfrecord, .test.tfrecord, .vocab, .his files. Not sure if these files can be merged back for training. Thanks for your efforts.