Hi authors,
I am a first-year PhD student at UMD. Thank you for the excellent paper. I’m working on secure code generation as well, and your reward design has been very useful for thinking through my own setup.
In my own runs, I seem to observe a training dynamic where the model first improves code-quality/style signals, then security, and only later recovers functionality/correctness. Did you observe something similar, and is that one reason you train for a relatively large number of epochs (eg. the 7B model was trained for 140 epochs?!)
I was also curious whether you tried functionality-gated rewards - for example, only allowing some security-oriented reward components to matter when the candidate already satisfies a minimum level of correctness. I’m wondering whether that might help preserve capability during training, or whether in your experience it makes optimization too sparse.
Thanks again for the great paper.
Best,
Abhinav
Hi authors,
I am a first-year PhD student at UMD. Thank you for the excellent paper. I’m working on secure code generation as well, and your reward design has been very useful for thinking through my own setup.
In my own runs, I seem to observe a training dynamic where the model first improves code-quality/style signals, then security, and only later recovers functionality/correctness. Did you observe something similar, and is that one reason you train for a relatively large number of epochs (eg. the 7B model was trained for 140 epochs?!)
I was also curious whether you tried functionality-gated rewards - for example, only allowing some security-oriented reward components to matter when the candidate already satisfies a minimum level of correctness. I’m wondering whether that might help preserve capability during training, or whether in your experience it makes optimization too sparse.
Thanks again for the great paper.
Best,
Abhinav