Parallel Backtesting

[SW4T400]

Hi,

I was wondering how to run a number of backtests simultaneously. For a quick survey during research I think I might want to explore a list of parameters (lookbacks).

I cant just add an execution for each parameter as that gives an error saying I already have an execution for that symbol.
How much of bad practice or how dangerous would be using joblib? I think pybroker uses this internally and I think that the following could be a very bad idea?

Do you guys have similar requirements? How do you do it - any ideas?

# This will be too slow:
# ----------------------

my_bts=[]

for i in [5,10,15,20,25,30,35,40,45,50]:
    my_bts.append(do_a_backtest(lookback=i))

# so I currently do this but am unsure wether this is save:
# ------------------------------------------------------------

my_bts = Parallel(n_jobs=-1,backend="loky")( 
    delayed(do_a_backtest)(i) 
    for i in [5,10,15,20,25,30,35,40,45,50])

the function i use:

def do_a_backtest(lookback):

    pybroker.disable_indicator_cache()
    pullback_20_indi_obj = pybroker.indicator('pullback_20_indi_name', pullback_fcn, lookback=lookback)

    strategy = Strategy(df, START_DATE, END_DATE, config) 

    strategy.clear_executions()

    #my1st:
    strategy.add_execution(buy_pullback_rule, 
                           [
                                f"ADAUSDT_{freq}",
                                f"VETUSDT_{freq}"
                                              ]
                            ,indicators=[
                                pullback_20_indi_obj,
                                ]
                           )

    bt = strategy.walkforward( #adf,
        timeframe=freq,
        warmup=30,
        windows=1,
        train_size=0,
        lookahead=1,
        calc_bootstrap=False,
        
    )
    
    return bt```

[edtechre]

Hi @SW4T400,

Yes, PyBroker uses joblib to compute indicator values in parallel. I have not tried running a backtest within joblib but it should work fine. Are you seeing any issues?

I originally planned to support something like this. This may be worth revisiting by integrating with Optuna.

[SW4T400]

My simple code did actually work with joblib and a "do_backtest" function in parallel to test various parameters of a rule. I just wanted to check wheter you feel this could be an issue as your are aware of the inner workings of your library.

Since you mention Optuna there´s one guy who uses pybroker in the objective function of Optuna .
But now as I am copying it iI see n_jobs is set to 1 (single job - not parallel) and not -1 (use all CPUs in parallel).

 direction = "maximize"
    study = optuna.create_study(direction=direction)
    study.optimize(objective, n_trials=n_trials, timeout=11000, n_jobs=1)