Question: RL Algorithms to train LLMs

[VigneshHexo]

Quick question: The repo README mentions that the goal is to implement RL algorithms to train LLMs in a research environment. Upon going through the repository I couldn't find any specific code for RL training LLMs. Requesting the organizers to help with the following questions:

1. Could you confirm that at this moment, no specific code for RL training is released
2. Could you let us know if this is in the future roadmap and if yes, when can we expect it to be released?

Thank you very much for the excellent work. Loved it!

[VigneshHexo]

Hi @deepakn97, Bumping this question up please. Thank you very much

[deepakn97]

Hi @VigneshHexo
I'm sorry for not getting back to you sooner.

1. Correct, there is no RL training in this codebase at the moment.
2. We have plans to implement an RL training loop. This is currently under development, and the expected release date is by the end of April.

To give some context, for starters, we are integrating MLGym with TorchRL and creating a step-by-step blog post on how to use the MLGym package with the TorchRL package.
The decision to use an existing RL package is to avoid reinventing the wheel. An added benefit of this will be that you will be able to use different RL algorithms out of the box.

[VigneshHexo]

Thanks Deepak! Looking forward to it! Is there a tentative ETA on the blog post?

[deepakn97]

Just updated the previous comment. I hit enter before the comment was finished, so you saw an incomplete reply 😅

[VigneshHexo]

Thanks Deepak! Got it! Looking forward to it! Please let us know if there is any way by which we could contribute to it. Will be happy to do so

[VigneshHexo]

Hi @deepakn97,
If we have any update on this? How does the ETA look like at this moment?

[deepakn97]

Hi @VigneshHexo
There have been some significant delays and we are trying to stabilize the changes for RL and integrate them in the repo.
Current ETA is 3 weeks but we will try to push it asap.

[VigneshHexo]

Thanks @deepakn97 . Looking forward