Suggestion on LLM models config in config.yaml

[maisyhoang]

I try using this format in config.yaml to use multiple models from multiple providers. It works without any implementation changes, and LLM sampling weights are already normalized to 1. Should this be a better way to select the models compared to the current primary & secondary model way in examples?

Also it would be great if PR #130 got reviewed and merged since that feature would help not having to hard code multiple API key inside the config file.

[codelion]

This config format is supported, the primary and secondary model was the original format based on what was described in the alphaevolve paper. I am yet to see any example where having more than 2 LLMs actually helps with convergence or improvement over the best program. Did you test it on something where it is critical to have 4 LLMs in an ensemble?

[maisyhoang]

I don't know if using more LLMs is better than 2. Google may say they use 2 because they have only 2 models category commercially (Flash and Pro). You also use different models in different stages, that would somewhat count as more than 2 models, just not all at once.

For the circle packing I remember getting around 98.7% of SOTA with only Deepseek-R1-0528 (both primary and secondary models are the same), but I got impatient and stop early since it is eating my cloud budget, and the idea is kinda demonstrated. I tested with Deepseek because it is the cheapest (near) frontier models. Anyway, this means that using one model works also, but we lose the what other models can offer.

The only thing I like about this format more is that it is more convenient to have multiple frontier models from different providers in one stage (e.g, Gemini Pro, Claude Opus, O3, Grok Heavy), if one of them just happen to be smarter than others regarding the problem then it could make the breakthrough. Kinda like the more experts the better. But in terms of convergence, I think it is quite difficult to quantify should x LLMs better than y LLMs, or what is the optimal numbers of LLMs to use.

Since there is nothing to be changed, you can close this issue if you want.