Support for Gefen Optimizer

[iwr-redmond]

The newly released Gefen optimizer is a drop-in replacement for AdamW that claims an 8x memory reduction with comparable quality. It can be easily added as it is already compatible with Transformers, and has the potential to democratize training to a significant degree if it works as well as is being claimed. It may be preferable to trial the Gefen-X fork by @thad0ctor as the original author is deleting issues and appears disinterested in fixing up the code for modern architectures.

[thad0ctor]

The newly released Gefen optimizer is a drop-in replacement for AdamW that claims an 8x memory reduction with comparable quality. It can be easily added as it is already compatible with Transformers, and has the potential to democratize training to a significant degree if it works as well as is being claimed. It may be preferable to trial the Gefen-X fork by @thad0ctor as the original author is deleting issues and appears disinterested in fixing up the code for modern architectures.

Haha I am glad I am not the only irked by them deleting issues, I have no reason to believe they are acting in good faith any more.

[iwr-redmond]

More likely they have just fallen off the tater truck. Either way, there is not going to be any of the reassurance needed for adding Gefen to Transformers.

[thad0ctor]

Yea I wrote up something to reply to all the open issues/PRs across megatron, transformers to warn them not to waste their time but it turns out the repo owner banned me after deleting the issue so I can't post on any of his PRs/issues - I understand cherry picking benches but at this point I am left to think they are just making false claims... and for what, I have no idea

[iwr-redmond]

The user is a PhD student, so perhaps the paper is part of their doctoral thesis work. Sometimes PoC code needs to be set-and-forget once the paper is finished. As there are two co-authors I reckon that the claims may well be valid, albeit poorly implemented in the code!, for the tested architectures while being defective in terms of untested ones.

[thad0ctor]

That's legit and I'd be inclined to agree if they weren't opening PRs across all major platforms. If it is a proof of concept, peace, but it clearly reads like a full capability and the PRs and everything public-facing implies that too.

[ndvbd]

Thank you @iwr-redmond for the respectful discussion. Feel free to contact me directly for any support, collaboration, or ideas. More engineering features and support for various distributed training setups and other specific configurations are definitely welcome, and will be added.
The current Gefen code is primarily targeted at model pre-training. So far, we have received case studies up to 32B parameters, with some reporting 6x shorter training time due to reduced CPU offloading.

As for Gefen-Muon, the Muon replacement, as it does not require second moment compression, so we also tested it on a fine-tuning task. Using Axolotl with the Qwen3 1.7B architecture on a single GPU, fine-tuned on Alpaca, we see almost identical training loss curves. Although more Muon to Gefen-Muon comparisons should be made, this suggests that Muon fine-tuning can potentially be replaced with Gefen-Muon. We will definitely update the repo when more case studies will come.

The optimizer peak memory consumption of Gefen-Muon was 27% less than Muon (3.7GiB vs 5.1Gib), and training curves looks almost the same. Note that Muon is applied on certain tensors (non embedding tensors, etc., as recommended by Muon authors and in Axolotl). And as for AdamW->Gefen, the main focus at the moment is pre-training, and fine-tuning later.

Bear in mind that this is relatively new, and support for additional frameworks will be added over time.

[thad0ctor]

@iwr-redmond I re-opened issues on my fork, I can't reply to your other comment because ndvbd banned me

[iwr-redmond]

ndvbd banned me

The closest I have come ever is being labelled a bot. You must be double-backboned 😆

[thad0ctor]

ndvbd banned me

The closest I have come ever is being labelled a bot. You must be double-backboned 😆

I will admit I am quite stubborn lol... open source is all about collaboration & open dissemination of information and there is really no profit incentive to puff up claims. So after burning hours on this getting it to work as advertised all to get blocked, my PR merged without authorship and the author deleting discussion about legitimate issues that would stonewall other users it struck a nerve.