New optimizer class

[MatteoRobbiati]

I was thinking about the current status of the optimizer module.

Current optimization status

We use the optimize function, in which we can set the method variable. This function calls different tools according to the selected method. The function's structure is the following:

1. if method $\in$ scipy.optimize.minimize, set method directly inside the minimize function and use it.
2. if not, calls other methods, like the cmaes algorithm.

I think this procedure can lead to some misunderstanding, especially as a consequence of the fact that the default options for the optimizer are the same of scipy.optimize.minimize. It is clear the utility now, because minimize covers the 99% of the choices, but I think we are going to have a larger spectrum of possibilities.

New purposed structure

I think it can be useful to think about a general Optimizer class, with set_options and optimize methods, in which we can customize the argument setting and the calling function. I try to explain in the following:

from scipy.optimize import basinhopping

class Optimizer:
    """Parent optimizer"""
 
    _method = None
    
    def __init__(self, loss, ...):
        # set stuff
    
    def set_options(self, kwargs**):
        """Cast the options passed into the format expected by the optimizer"""
        
    def optimize(self, initial_parameters):
        """Performs the optimizations and returns f_best, x_best."""
    
class BasinHopping(Optimizer):
    """Basin Hopping algorithm for optimization."""
    
    _method = "basinhopping"

    def set_options(self, **kwargs):
        # set custom options here

    def optimize(self, initial_p):
        # call the basinhopping here
        res = basinhopping(...)


def launch_optimization(default_options, optimizer):
    """Performs the optimization using the chosen `Optimizer` class"""

Any suggestions/complains?

Roadmap

I purpose the following drawing as guideline for the future implementations.

[scarrazza]

Thanks @MatteoRobbiati, related to #235 and potentially #657, here some comments:

• The proposed abstraction layout is acceptable and should not break current applications, however, how are you planning to standardize the options format?
• Could you please complete the right part of the plot with the status (implemented, ongoing, not implemented) for the optimizers?
• I believe the algorithms for gradient might require another level of abstraction dedicated to Gradients, given that we do not provide SGD algorithms outside tensorflow.

[MatteoRobbiati]

• The proposed abstraction layout is acceptable and should not break current applications, however, how are you planning to standardize the options format?

I was thinking to define a default set of options for the Optimizer, like niterations and loss_target. Then, the options dictionary can be updated in each child class by adding new elements, e.g. initial_temperature if dealing with an annealer.
I am not sure if this can work well, but it seems more general than the current one.

• Could you please complete the right part of the plot with the status (implemented, ongoing, not implemented) for the optimizers?

Sure, I just added three symbols:

• the green V means implemented;
• the red X means not implemented;
• the code symbol means ongoing.

Is it clear?

• I believe the algorithms for gradient might require another level of abstraction dedicated to Gradients, given that we do not provide SGD algorithms outside tensorflow.

I think you are right.
In my opinion it should be amazing to have one super high level Optimizer call, but maybe the most comfortable solution is do divide the optimizers into three classes:

1. analytical gradient-based (executable both on hardware and simulation);
2. meta-heuristic: genetic algorithms, annealers, etc. We can have a look to this very huge collection, or at the pymoo repo, which is more in general optimization-oriented.
3. newtonian based strategies, relying on the already used scipy.optimize.minimize module.